Mainline Realtime Linux (PREEMPT_RT) 2.6.29 declared "Latest Stable"

By: Carsten Emde

Upgrade to 2.6.29.4-rt17 and enjoy better performance and responsiveness of the Linux kernel than ever before

On the vanilla side of life

Others have already made extensive benchmarks of the Linux vanilla kernels from 2.6.24 to 2.6.29 and found impressive performance gains that occurred first in 2.6.29. Some of these improvements reverted earlier degradations, but some of them were new - and unexpected. One of these unexpected performance gains was the system time required to scale an image using the ImageMagick library. We tried to reproduce these findings

2.6.26:

# convert test-3264x2448.ppm -size 1024x768 test-1024x768.ppm
    User time (seconds): 0.56
    System time (seconds): 0.54

2.6.29:

# convert test-3264x2448.ppm -size 1024x768 test-1024x768.ppm
    User time (seconds): 0.55
    System time (seconds): 0.23

and were, obviously, successful. The above graphic displays the result of 10 consecutive conversions on each of the two kernel versions. Amazing, isn't it? But do you think that this is certainly the maximum increase in performance that can ever be reached by upgrading to a new kernel version? Read on!

On the RT side of life

We have received a number of complaints with respect to the timing of sending signals between programs and from driver to program. This has led to the creation of a simple test program and a related driver. The backfire driver reads input from the /dev/backfire device; when a number between 1 and 31 is written to the driver through this communication channel, the driver immediately sends a signal of that number back to the calling program. A subsequent read request from /dev/backfire returns the time of the day when the signal was sent. This makes it possible for the user program sendme to measure the time required for the signal to travel from driver level to user space. Some more details of sendme and backfire are given in the related manpages sendme.8 and backfire.4. Download instructions and a quick HOWTO are available here. These tests are now part of the benchmark suite that must have been passed successfully before a kernel is declared "Latest Stable". As part of these benchmarks, we have executed the sendme/backfire test on an Intel Core 2 Quad CPU @2.4 GHz with the following simultaneous stress conditions:

# while true
> do
> hackbench 25
> done

# while true
> do
> ls -Ral /
> done

Running a total of 10,000,000 loops of sendme under the above stress conditions on 2.6.26.8-rt16 revealed:

# sendme -i200 -a -p99 -l10000000
Samples: 10000000
To:   Min    1, Cur    2, Avg    2, Max   14
From: Min    5, Cur    6, Avg    7, Max  290

Doing the same on our "Latest Stable" 2.6.29.4-rt17 gives:

# sendme -i200 -a -p99 -l10000000
Samples: 10000000
To:   Min    0, Cur    1, Avg    1, Max   12
From: Min    2, Cur    2, Avg    2, Max   17

The average duration dropped from 7 to 2 µs, and the worst-case duration even dropped from 290 to 17 µs! Wow, we have discovered another miracle of our "Latest Stable" 2.6.29.4-rt17. What happened here?

Instrumentation and tracing of the critical code in kernel 2.6.26 revealed a number of missing lock breaks in the memory allocation code. Fixing these omissions already reduced the worst-case duration of signal traveling considerably. Kudos to Peter Zijlstra for doing such a great job! In addition, Thomas Gleixner applied special optimizations to further improve the timing in cases when only a single signal is used. Thus, programmers who make heavy use of signals in parallel with frequent memory allocations will probably see a considerably better kernel responsiveness after upgrading to 2.6.29.4-rt17. In addition and as always, the new PREEMPT_RT kernel fixes many bugs and incorporates many improvements.

So, the message is: There are a number of good reasons for everybody to upgrade to our "Latest Stable". Everybody? Well, sorry, not everybody - the MIPS architecture patches didn't make it into the new kernel yet. But the MIPS people are working heavily on it, and we will make it available as soon as possible. The 2.6.26 -rt tree will, therefore, be maintained for some time, and fixes will be backported should it become necessary. If everything works well, MIPS support will be back in the -rt tree of 2.6.30.

BTW: In terms of release date, real-time Linux is now pretty close to vanilla Linux - the -rt tree of 2.6.29 became "Latest Stable" on the same day as vanilla 2.6.30 was released - we are only one release number behind! And the distance between vanilla and -rt will become even closer in the next future, until both are released the same day - the day when the CONFIG_PREEMPT_RT configuration item will appear in an unpatched vanilla kernel ...