Steven Rostedt, Red Hat, Inc

One of the difficult tasks analyzing Real-Time systems is finding a source/cause of an unexpected latency. Is the latency caused by the application or the kernel? Is it a wake up scheduling latency or a latency caused by interrupts being disabled, or is it a latency caused by preemption being disabled, or a combination of disabled interrupts and preemption.
 
Ftrace has its origins from the -rt patch latency tracer, and still  carries the capabilities to track down latencies. It can catch the maximum wake up latency for the highest priority task. This wake up latency can also be tuned to only trace Real-Time processes. There is a latency tracer to find the latency of how long interrupts and/or preemption are disabled. The max is captured and you can even see the functions that were called in the mean time. Ftrace also has a rich array of tracing features that can help determine if latencies are caused by the kernel, or simply are a bi-product of an application.

This paper will describe in detail how to use the latency tracers of  ftrace, specifically to find the origins of latencies seen by Real-Time tasks.