Stability regression - Latency regressions

The following two pairs of 1-year recordings of 5-min worst-case timer and wakeup latency and kernel version have been obtained on two different single-core and one multi-core computer. The single-core computers are an AMD Geode LX-800 processor running at 500 MHz and an Intel Celeron M processor running at 1500 MHz; the multi-core computer is an Intel 2-way with hyperthreading i3-2350M processor running at 2300 MHz. Although the market introduction of the first two processors dates back several years ago, they still matter, since they are shipped and used in industrial products up to today.

Upgrading from a 3.0 to a 3.2 series kernel apparently was followed by a clear-cut increase in the continuously registered 5-min worst-case timer and wakeup latencies - an obvious case of a regression that must be analyzed and fixed before a PREEMPT_RT kernel version can be labeled "Latest Stable".

This is not a recording problem or artifact, since sporadic latency outliers have also been registered in the plot histograms since upgrading (image in full resolution):

This is another case of an obvious regression after upgrading from a 2.6.x to a 3.0 PREEMPT_RT kernel. The problem was fixed initially, but during subsequent upgrade from 3.0 to 3.2 the latencies again increased and still are higher than with the initial 2.6.x kernel.

Patch unreverted:

Patch reverted:

After a Linux kernel patch was forwarded from the current development tree to the 3.8, 3.6 and 3.2 stable trees, high latencies of up to 2 ms were observed. The regression was also reported by Christoph Mathys who even found the culprit to be the patch drm/i915: Workaround incoherence between fences and LLC across multiple CPUs which he reported in another posting. Unfortunately, he was not immediately able to convince the RT maintainers to revert the patch that obviously is unacceptable in an RT kernel. We can only hope that the additional proof obtained in this OSADL QA Farm system will provide sufficient evidence to finally revert this patch and ask Chris Wilson, the author of the patch, for a better solution of the problem he wanted to fix.