You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-11 - 22:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Fri Apr 11, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350440,0irq/25-eth10-21swapper/306:45:103
743223526,0sleep20-21swapper/206:43:462
770223422,0sleep00-21swapper/006:46:560
750223324,0sleep10-21swapper/106:44:331
28415070,0irq/27-eth0-tx-0-21swapper/006:55:310
78609966,0cyclictest0-21swapper/210:17:152
78609965,0cyclictest6072-21diskmemload09:54:332
78609960,0cyclictest0-21swapper/211:28:502
78589966,0cyclictest0-21swapper/011:23:510
339960,0migration/330897-21sh11:38:043
339960,0migration/323807-21sh09:25:223
339960,0migration/32274-21sh07:28:213
339960,0migration/318496-21idleruntime08:08:143
339960,0migration/31063-21sh12:04:383
28415060,0irq/27-eth0-tx-24425-21ssh11:57:372
28415060,0irq/27-eth0-tx-0-21swapper/212:14:282
28415060,0irq/27-eth0-tx-0-21swapper/212:06:452
28415060,0irq/27-eth0-tx-0-21swapper/211:40:142
28415060,0irq/27-eth0-tx-0-21swapper/211:36:032
28415060,0irq/27-eth0-tx-0-21swapper/210:58:082
28415060,0irq/27-eth0-tx-0-21swapper/210:55:302
28415060,0irq/27-eth0-tx-0-21swapper/210:43:122
28415060,0irq/27-eth0-tx-0-21swapper/210:33:202
28415060,0irq/27-eth0-tx-0-21swapper/210:20:422
28415060,0irq/27-eth0-tx-0-21swapper/210:01:422
28415060,0irq/27-eth0-tx-0-21swapper/209:51:562
28415060,0irq/27-eth0-tx-0-21swapper/209:47:272
28415060,0irq/27-eth0-tx-0-21swapper/209:40:302
28415060,0irq/27-eth0-tx-0-21swapper/209:34:072
28415060,0irq/27-eth0-tx-0-21swapper/209:24:062
28415060,0irq/27-eth0-tx-0-21swapper/208:59:132
28415060,0irq/27-eth0-tx-0-21swapper/008:08:110
28415060,0irq/27-eth0-tx-0-21swapper/007:31:350
28405060,0irq/26-eth0-rx-18055-21irqstats11:53:123
28405060,0irq/26-eth0-rx-0-21swapper/311:08:153
249960,0watchdog/27403-21ssh11:45:112
249960,0watchdog/20-21swapper/208:53:072
149960,0migration/026005-21kthreadcore07:48:170
149960,0migration/02287-21aten2.4-expect07:08:110
78609954,0cyclictest10943-21sed08:02:562
78609950,0cyclictest0-21swapper/211:13:042
78609950,0cyclictest0-21swapper/210:23:242
78599954,0cyclictest19534-21kworker/1:011:48:041
78599954,0cyclictest19534-21kworker/1:011:47:481
78599954,0cyclictest19534-21kworker/1:010:56:541
78599954,0cyclictest19534-21kworker/1:010:28:581
78599954,0cyclictest19534-21kworker/1:010:27:361
78599954,0cyclictest19534-21kworker/1:009:39:541
78599954,0cyclictest19534-21kworker/1:008:18:101
78599954,0cyclictest19534-21kworker/1:008:13:181
78599953,0cyclictest19534-21kworker/1:011:13:021
78589955,0cyclictest0-21swapper/011:52:320
339950,0migration/39992-21kthreadcore11:18:143
339950,0migration/38895-21idleruntime06:48:133
339950,0migration/38432-21taskset07:09:053
339950,0migration/38037-21kthreadcore07:58:143
339950,0migration/3618-21latency_hist09:57:563
339950,0migration/332380-21taskset10:18:293
339950,0migration/331789-21idleruntime08:18:123
339950,0migration/328080-21taskset10:47:173
339950,0migration/322379-21munin-node06:58:093
339950,0migration/321073-21kthreadcore08:33:123
339950,0migration/318005-21cpuspeed_turbos09:48:103
339950,0migration/31382-21fschecks_count09:58:093
339950,0migration/313504-21kthreadcore10:33:153
339950,0migration/312304-21kthreadcore10:08:153
339950,0migration/310146-21cpu07:38:073
28415050,0irq/27-eth0-tx-9005-21munin-node07:13:190
28415050,0irq/27-eth0-tx-24847-21munin-node11:58:132
28415050,0irq/27-eth0-tx-20980-21ssh11:08:032
28415050,0irq/27-eth0-tx-19215-21ssh11:02:562
28415050,0irq/27-eth0-tx-16668-21munin-node07:43:110
28415050,0irq/27-eth0-tx-15351-21grep10:08:162
28415050,0irq/27-eth0-tx-0-21swapper/212:08:092
28415050,0irq/27-eth0-tx-0-21swapper/211:49:322
28415050,0irq/27-eth0-tx-0-21swapper/211:24:522
28415050,0irq/27-eth0-tx-0-21swapper/211:19:322
28415050,0irq/27-eth0-tx-0-21swapper/210:04:482
28415050,0irq/27-eth0-tx-0-21swapper/209:20:332
28415050,0irq/27-eth0-tx-0-21swapper/209:13:232
28415050,0irq/27-eth0-tx-0-21swapper/209:08:192
28415050,0irq/27-eth0-tx-0-21swapper/008:37:310
28415050,0irq/27-eth0-tx-0-21swapper/008:03:080
28415050,0irq/27-eth0-tx-0-21swapper/008:01:310
28415050,0irq/27-eth0-tx-0-21swapper/007:03:160
28405050,0irq/26-eth0-rx-9972-21ssh10:29:533
28405050,0irq/26-eth0-rx-9968-21cut11:48:133
28405050,0irq/26-eth0-rx-864-21grep10:48:173
28405050,0irq/26-eth0-rx-3865-21kernelversion12:08:083
28405050,0irq/26-eth0-rx-31248-21taskset09:29:043
28405050,0irq/26-eth0-rx-28658-21smartctl11:33:203
28405050,0irq/26-eth0-rx-21155-21cut07:43:133
28405050,0irq/26-eth0-rx-17550-21ssh12:15:263
28405050,0irq/26-eth0-rx-0-21swapper/311:30:043
28405050,0irq/26-eth0-rx-0-21swapper/311:25:153
28405050,0irq/26-eth0-rx-0-21swapper/311:15:083
28405050,0irq/26-eth0-rx-0-21swapper/310:13:263
28405050,0irq/26-eth0-rx-0-21swapper/309:38:093
28405050,0irq/26-eth0-rx-0-21swapper/309:37:313
28405050,0irq/26-eth0-rx-0-21swapper/309:18:203
28405050,0irq/26-eth0-rx-0-21swapper/308:58:053
259950,0migration/230915-21latency_hist08:17:562
259950,0migration/229315-21munin-node10:48:132
249950,0watchdog/20-21swapper/210:38:352
249950,0watchdog/20-21swapper/210:32:192
249950,0watchdog/20-21swapper/209:32:192
249950,0watchdog/20-21swapper/209:05:312
249950,0watchdog/20-21swapper/208:48:592
249950,0watchdog/20-21swapper/208:43:072
249950,0watchdog/20-21swapper/208:31:112
249950,0watchdog/20-21swapper/208:25:352
249950,0watchdog/20-21swapper/208:08:272
249950,0watchdog/20-21swapper/207:53:192
249950,0watchdog/20-21swapper/207:48:152
249950,0watchdog/20-21swapper/207:38:232
249950,0watchdog/20-21swapper/207:28:112
249950,0watchdog/20-21swapper/207:14:032
249950,0watchdog/20-21swapper/207:08:112
249950,0watchdog/20-21swapper/207:06:472
179950,0migration/18594-21irqcore09:13:161
179950,0migration/14354-21irqcore07:33:131
179950,0migration/132709-21gltestperf08:43:091
179950,0migration/128788-21kthreadcore09:28:101
179950,0migration/126737-21munin-node11:08:161
179950,0migration/119155-21latency_hist11:02:561
179950,0migration/115807-21kthreadcore08:53:121
179950,0migration/114741-21aten_r1power_cu06:53:081
179950,0migration/11027-21munin-node07:03:151
149950,0migration/09566-21kthreadcore11:18:140
149950,0migration/07671-21sh11:45:560
149950,0migration/0584-21sh12:03:260
149950,0migration/05212-21kthreadcore10:28:100
149950,0migration/030916-21latency_hist08:17:560
149950,0migration/030365-21df_abs07:53:090
149950,0migration/029276-21sh11:34:060
149950,0migration/028856-21kthreadcore11:58:160
149950,0migration/025045-21kthreadcore06:58:110
149950,0migration/020798-21kthreadcore10:38:140
149950,0migration/019249-21kthreadcore09:48:140
149950,0migration/018470-21aten_r1power_fr10:13:110
149950,0migration/018434-21taskset11:00:280
149950,0migration/016248-21kthreadcore08:53:120
149950,0migration/015304-21kthreadcore08:28:160
149950,0migration/010256-21taskset10:53:590
8225040,0irq/24-0000:00:0-21swapper/208:13:012
78609943,0cyclictest3110-21irqbalance08:38:112
78609943,0cyclictest3110-21irqbalance08:35:212
78609943,0cyclictest3110-21irqbalance07:58:012
78609943,0cyclictest3110-21irqbalance07:43:112
78609943,0cyclictest3110-21irqbalance07:33:312
78599944,0cyclictest19534-21kworker/1:012:13:221
78599944,0cyclictest19534-21kworker/1:012:08:081
78599944,0cyclictest19534-21kworker/1:012:03:041
78599944,0cyclictest19534-21kworker/1:011:58:041
78599944,0cyclictest19534-21kworker/1:011:54:041
78599944,0cyclictest19534-21kworker/1:011:38:021
78599944,0cyclictest19534-21kworker/1:011:33:021
78599944,0cyclictest19534-21kworker/1:011:28:081
78599944,0cyclictest19534-21kworker/1:011:23:181
78599944,0cyclictest19534-21kworker/1:011:18:221
78599944,0cyclictest19534-21kworker/1:010:58:001
78599944,0cyclictest19534-21kworker/1:010:49:201
78599944,0cyclictest19534-21kworker/1:010:43:061
78599944,0cyclictest19534-21kworker/1:010:38:141
78599944,0cyclictest19534-21kworker/1:010:18:001
78599944,0cyclictest19534-21kworker/1:010:07:581
78599944,0cyclictest19534-21kworker/1:010:04:221
78599944,0cyclictest19534-21kworker/1:009:59:061
78599944,0cyclictest19534-21kworker/1:009:54:161
78599944,0cyclictest19534-21kworker/1:009:48:021
78599944,0cyclictest19534-21kworker/1:009:43:221
78599944,0cyclictest19534-21kworker/1:009:22:581
78599944,0cyclictest19534-21kworker/1:009:18:141
78599944,0cyclictest19534-21kworker/1:009:08:001
78599944,0cyclictest19534-21kworker/1:009:03:021
78599944,0cyclictest19534-21kworker/1:008:58:001
78599944,0cyclictest19534-21kworker/1:008:48:001
78599944,0cyclictest19534-21kworker/1:008:37:581
78599944,0cyclictest19534-21kworker/1:008:27:581
78599944,0cyclictest19534-21kworker/1:008:22:561
78599944,0cyclictest19534-21kworker/1:008:08:261
78599944,0cyclictest19534-21kworker/1:007:58:021
78599944,0cyclictest19534-21kworker/1:007:38:121
78599944,0cyclictest19534-21kworker/1:007:23:101
78599944,0cyclictest19534-21kworker/1:007:17:581
78599944,0cyclictest19534-21kworker/1:007:13:021
78599944,0cyclictest19534-21kworker/1:007:08:241
78599944,0cyclictest19534-21kworker/1:006:58:061
78599944,0cyclictest19534-21kworker/1:006:48:021
78599943,0cyclictest3425-21lldpd08:03:061
78599943,0cyclictest3425-21lldpd07:48:051
78599943,0cyclictest19534-21kworker/1:009:33:021
78599943,0cyclictest19340-21grep08:33:091
78589944,0cyclictest0-21swapper/011:05:240
78589944,0cyclictest0-21swapper/009:06:130
78589943,0cyclictest8376-21awk09:13:150
78589943,0cyclictest6445-21snmpd11:08:360
78589943,0cyclictest6445-21snmpd10:20:510
78589943,0cyclictest6445-21snmpd10:11:060
78589943,0cyclictest6445-21snmpd07:24:500
78589943,0cyclictest3110-21irqbalance12:14:300
78589943,0cyclictest3110-21irqbalance10:44:100
78589943,0cyclictest3110-21irqbalance10:25:000
78589943,0cyclictest3110-21irqbalance09:44:010
78589943,0cyclictest30742-21sadc08:17:560
78589943,0cyclictest18619-21sadc10:37:560
339940,0migration/37041-21sh11:44:093
339940,0migration/35020-21kthreadcore07:33:143
339940,0migration/329951-21latency_hist07:52:563
339940,0migration/327307-21kthreadcore08:13:143
339940,0migration/324021-21fschecks_time07:48:143
339940,0migration/323353-21irqcore07:23:123
339940,0migration/319575-21kthreadcore07:18:143
339940,0migration/316795-21kthreadcore06:53:143
339940,0migration/316753-21switchtime08:03:193
28415040,0irq/27-eth0-tx-0-21swapper/008:48:160
28415040,0irq/27-eth0-tx-0-21swapper/008:25:220
28415040,0irq/27-eth0-tx-0-21swapper/006:48:090
28405040,0irq/26-eth0-rx-10065-21sh10:53:223
28405040,0irq/26-eth0-rx-0-21swapper/311:58:103
28405040,0irq/26-eth0-rx-0-21swapper/311:03:043
28405040,0irq/26-eth0-rx-0-21swapper/310:58:133
28405040,0irq/26-eth0-rx-0-21swapper/310:38:083
28405040,0irq/26-eth0-rx-0-21swapper/310:23:523
28405040,0irq/26-eth0-rx-0-21swapper/309:44:103
28405040,0irq/26-eth0-rx-0-21swapper/309:14:423
28405040,0irq/26-eth0-rx-0-21swapper/309:10:213
28405040,0irq/26-eth0-rx-0-21swapper/309:03:333
28405040,0irq/26-eth0-rx-0-21swapper/308:53:123
28405040,0irq/26-eth0-rx-0-21swapper/308:50:173
28405040,0irq/26-eth0-rx-0-21swapper/308:43:133
259940,0migration/222763-21kthreadcore06:58:102
249940,0watchdog/20-21swapper/207:23:592
249940,0watchdog/20-21swapper/207:18:512
249940,0watchdog/20-21swapper/206:53:112
249940,0watchdog/20-21swapper/206:47:592
179940,0migration/1538-21kthreadcore07:28:171
179940,0migration/132150-21kthreadcore07:53:131
169940,0watchdog/10-21swapper/110:33:151
169940,0watchdog/10-21swapper/110:13:031
149940,0migration/09223-21irqcore09:38:110
149940,0migration/07382-21runrttasks09:59:040
149940,0migration/031578-21kthreadcore11:13:130
149940,0migration/031469-21fschecks_count11:38:110
149940,0migration/030438-21ntp_offset09:28:120
149940,0migration/027431-21kthreadcore09:53:150
149940,0migration/017646-21kthreadcore07:18:140
149940,0migration/01743-21kthreadcore09:08:140
78599933,0cyclictest19534-21kworker/1:007:42:581
78589933,0cyclictest0-21swapper/012:08:370
78589932,0cyclictest6445-21snmpd09:34:360
78589932,0cyclictest6445-21snmpd09:18:510
78589932,0cyclictest5011-21strings08:43:140
78589932,0cyclictest3425-21lldpd09:24:360
78589932,0cyclictest30950-21sensors08:38:180
78589932,0cyclictest29429-21awk10:48:130
78589932,0cyclictest20582-21sensors11:28:130
78589932,0cyclictest17610-21awk11:53:100
78589932,0cyclictest10890-21awk07:38:110
28405030,0irq/26-eth0-rx-0-21swapper/310:02:573
28405030,0irq/26-eth0-rx-0-21swapper/308:41:003
28405030,0irq/26-eth0-rx-0-21swapper/307:13:093
28405030,0irq/26-eth0-rx-0-21swapper/307:03:073
149930,0migration/029329-21runrttasks09:02:000
78589921,0cyclictest0-21swapper/010:02:560
78589920,0cyclictest0-21swapper/007:33:080
339920,0migration/317656-21python08:28:213
28405020,0irq/26-eth0-rx-0-21swapper/308:23:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional