Raphaël Beamonte, Ecole Polytechnique de Montréal, Québec, Canada
Michel Dagenais, Ecole Polytechnique de Montréal, Québec, Canada

Real-time systems have always been more difficult to monitor and debug because of the real-time constraints which rule out any tool significantly impacting the system latency and performance. Tracing is often the most reliable tool available for studying real-time systems. In recent years, the real-time behavior of Linux systems has greatly improved and, with proper CPU shielding on multicore systems, it is now possible to have latencies in the low microsecond range. In that context, tracers must insure that their overhead is within that range, predictable and scales well to multiple cores.

The recently released LTTng 2.0 toolchain has been optimized for multicore performance, scalability and flexibility. We have studied its impact on the maximum latency for serving hard real-time applications in a multicore environment using CPU shielding. In order to achieve this, we have used and extended the real time verification tools cyclictest (from the rt-tests suite), and the hwlat_detector module. These tools were first used to establish the baseline of real-time system performance and then to measure the impact added by tracing with both LTTng kernel tracing and LTTng user-space tracing (UST). This identified modifications required to the buffer switch protocol in LTTng UST, and special care required to isolate the shielded real-time cores from the RCU interprocess synchronization routines.

This work resulted in extended tools to measure the real-time properties of multicore Linux systems, a precise characterization of the real-time impact of LTTng kernel and UST tracing tools, and improvements to LTTng, and its use of RCU, for tracing real-time systems.

It will thus be easier to assess the real-time performance of multicore Linux systems. Moreover, LTTng will become a tool of choice to study the performance and behavior of such hard real-time multicore systems, given its small and deterministic impact on the maximum latency.