During the development of a PREEMPT_RT patch series of a Linux kernel version, OSADL runs a number of tests to ensure such kernels are ready for production purposes. This QA activity includes monitored stress tests to detect

• kernel crashes
• memory leaks
• driver misbehavior
• unsatisfying worst-case latency
• etc.

on a large number of different UP/SMP and 32/64-bit processors. A related PREEMPT_RT version is not released as "Latest Stable", unless all systems under test pass these quality checks. The systems are hosted in a QA farm on OSADL Test Racks in several OSADL testing labs. A part of this rack space is reserved for customized CPU boards that are tested continuously against every newly built kernel. This is a special service for OSADL member companies and also for non-members who may then "freeze" a particular kernel version in shipped systems but still follow the kernel development. Whenever a newly built kernel fails to run on such test systems or shows unsatisfying real-time behavior, a notification is sent out. Because of the close temporal relationship between the introduction of the bug and the notification, the underlying problem can be located easily. Should it become necessary to upgrade the "frozen" kernel of production systems, the required kernel version is already available and can be deployed without major upgrade hassles.

Test Racks #1 to #10 run, at least partly, popular systems the results of which may be interesting for a larger audience. They are, therefore, presented at the following pages. Some more systems have been added to create a wide variablility of different architectures (x86, ARM, PowerPC, MIPS), processor topologies (UP, single-socket SMP, multi-socket SMP), bit length (32, 64), memory size (64 MByte to 12 GByte), storage media (hard disk, SSD), storage arrays (raw, software RAID, hardware RAID) and distros (Arago, CentOS, ElinOS, Fedora, PTXdist, RAYS, Ubuntu). The data are obtained via a direct link to one of the OSADL testing labs and are updated continuously.

The OSADL QA Farm uses the OpenNMS Networking Monitoring System. It was extended with a number of LLDP-based and other discovery scripts to allow for monitoring embedded systems that OpenNMS normally cannot handle. The main purpose of using OpenNMS is to provide an overall display of the topology of all systems. It allows for monitoring system availability in real-time as an important prerequisite to quickly react to any irregularity and ensure, whenever possible, that long-term stability measurements are not interrupted.