You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 04:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Wed Apr 02, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1134528277,0sleep00-21swapper/018:45:260
291350450,0irq/25-eth10-21swapper/318:47:513
1154723627,0sleep20-21swapper/218:47:482
1157223026,0sleep10-21swapper/118:48:031
116709977,0cyclictest0-21swapper/220:48:422
339960,0migration/38865-21sh21:41:333
339960,0migration/326710-21timerwakeupswit23:58:403
339960,0migration/324224-21munin-node21:28:373
339960,0migration/321481-21sh22:40:483
28415060,0irq/27-eth0-tx-26538-21ssh21:56:092
28415060,0irq/27-eth0-tx-24576-21ssh21:29:392
28415060,0irq/27-eth0-tx-0-21swapper/223:59:482
28415060,0irq/27-eth0-tx-0-21swapper/223:50:102
28415060,0irq/27-eth0-tx-0-21swapper/223:20:352
28415060,0irq/27-eth0-tx-0-21swapper/223:13:092
28415060,0irq/27-eth0-tx-0-21swapper/222:54:022
28415060,0irq/27-eth0-tx-0-21swapper/222:50:342
28415060,0irq/27-eth0-tx-0-21swapper/222:31:492
28415060,0irq/27-eth0-tx-0-21swapper/222:26:012
28415060,0irq/27-eth0-tx-0-21swapper/222:06:412
28415060,0irq/27-eth0-tx-0-21swapper/221:49:552
28415060,0irq/27-eth0-tx-0-21swapper/221:44:542
28415060,0irq/27-eth0-tx-0-21swapper/221:39:242
28415060,0irq/27-eth0-tx-0-21swapper/221:23:232
28415060,0irq/27-eth0-tx-0-21swapper/221:15:102
28415060,0irq/27-eth0-tx-0-21swapper/221:04:512
28405060,0irq/26-eth0-rx-5757-21proc_pri21:13:383
28405060,0irq/26-eth0-rx-4268-21sh00:08:513
28405060,0irq/26-eth0-rx-19513-21cut23:08:293
28405060,0irq/26-eth0-rx-10579-21kthreadcore20:53:353
249960,0watchdog/231399-21ssh21:11:112
249960,0watchdog/20-21swapper/221:21:592
249960,0watchdog/20-21swapper/219:13:352
116709960,0cyclictest0-21swapper/223:43:592
116619960,0cyclictest0-21swapper/122:35:001
35150,0ktimersoftd/33067-17auditd19:13:163
35150,0ktimersoftd/30-21swapper/321:12:203
339950,0migration/39172-21munin-node21:18:333
339950,0migration/37125-21sh22:56:003
339950,0migration/35962-21kthreadcore19:58:353
339950,0migration/35690-21kthreadcore20:28:403
339950,0migration/33317-21taskset20:28:093
339950,0migration/32879-21kthreadcore19:03:353
339950,0migration/32669-21runrttasks22:02:503
339950,0migration/326609-21munin-node22:18:373
339950,0migration/325382-21switchtime21:53:393
339950,0migration/324377-21latency_hist22:48:173
339950,0migration/323397-21runrttasks22:45:503
339950,0migration/32137-21kthreadcore19:28:403
339950,0migration/319060-21sh23:53:463
339950,0migration/316509-21runrttasks21:24:293
339950,0migration/316412-21taskset20:34:563
339950,0migration/314839-21munin-node20:08:333
339950,0migration/314728-21kthreadcore21:48:303
339950,0migration/311899-21cpuspeed_turbos23:28:253
339950,0migration/311379-21latency_hist23:28:173
339950,0migration/310993-21munin-node23:48:363
28415050,0irq/27-eth0-tx-26476-21df_abs21:33:312
28415050,0irq/27-eth0-tx-0-21swapper/223:54:142
28415050,0irq/27-eth0-tx-0-21swapper/223:39:122
28415050,0irq/27-eth0-tx-0-21swapper/223:28:242
28415050,0irq/27-eth0-tx-0-21swapper/223:23:202
28415050,0irq/27-eth0-tx-0-21swapper/223:04:032
28415050,0irq/27-eth0-tx-0-21swapper/222:45:022
28415050,0irq/27-eth0-tx-0-21swapper/222:38:202
28415050,0irq/27-eth0-tx-0-21swapper/222:33:442
28415050,0irq/27-eth0-tx-0-21swapper/221:59:352
28415050,0irq/27-eth0-tx-0-21swapper/221:01:052
28415050,0irq/27-eth0-tx-0-21swapper/220:53:252
28415050,0irq/27-eth0-tx-0-21swapper/200:09:352
28415050,0irq/27-eth0-tx-0-21swapper/200:03:282
28415050,0irq/27-eth0-tx-0-21swapper/020:48:340
28405050,0irq/26-eth0-rx-32449-21ssh23:14:393
28405050,0irq/26-eth0-rx-23676-21irqcore20:18:343
28405050,0irq/26-eth0-rx-23074-21chrt20:39:523
28405050,0irq/26-eth0-rx-0-21swapper/322:35:503
28405050,0irq/26-eth0-rx-0-21swapper/321:44:053
28405050,0irq/26-eth0-rx-0-21swapper/321:33:393
28405050,0irq/26-eth0-rx-0-21swapper/321:03:403
28405050,0irq/26-eth0-rx-0-21swapper/320:59:463
28405050,0irq/26-eth0-rx-0-21swapper/320:51:083
259950,0migration/28053-21latency_hist22:58:172
259950,0migration/221719-21munin-node20:13:392
249950,0watchdog/228739-21kthreadcore20:43:352
249950,0watchdog/224026-21kthreadcore23:33:352
249950,0watchdog/219398-21ssh22:14:152
249950,0watchdog/212745-21python19:33:392
249950,0watchdog/20-21swapper/223:13:312
249950,0watchdog/20-21swapper/222:18:432
249950,0watchdog/20-21swapper/222:08:432
249950,0watchdog/20-21swapper/220:42:512
249950,0watchdog/20-21swapper/220:30:192
249950,0watchdog/20-21swapper/219:20:552
249950,0watchdog/20-21swapper/219:01:112
179950,0migration/19434-21munin-node20:28:451
179950,0migration/18948-21gltestperf22:58:331
179950,0migration/17475-21munin-node19:58:361
179950,0migration/131700-21kthreadcore19:53:361
179950,0migration/125161-21sh21:31:231
169950,0watchdog/115265-21ssh21:21:311
149950,0migration/031041-21kthreadcore20:23:340
149950,0migration/024898-21fschecks_time21:08:310
116709950,0cyclictest0-21swapper/220:18:432
116709950,0cyclictest0-21swapper/200:18:032
116619955,0cyclictest0-21swapper/123:29:521
116619954,0cyclictest8134-21kworker/1:221:48:501
116619954,0cyclictest8134-21kworker/1:221:25:361
116619954,0cyclictest8134-21kworker/1:221:06:441
116619954,0cyclictest8134-21kworker/1:220:37:041
116619954,0cyclictest8134-21kworker/1:219:28:451
116619954,0cyclictest15403-21kworker/1:023:45:381
116619954,0cyclictest15403-21kworker/1:023:10:261
116619954,0cyclictest15403-21kworker/1:000:17:241
116619950,0cyclictest0-21swapper/123:04:391
116619950,0cyclictest0-21swapper/121:13:491
116609955,0cyclictest0-21swapper/023:46:190
116609955,0cyclictest0-21swapper/023:11:280
116609955,0cyclictest0-21swapper/022:22:080
116609955,0cyclictest0-21swapper/022:07:560
116609955,0cyclictest0-21swapper/022:00:320
116609955,0cyclictest0-21swapper/021:35:090
116609955,0cyclictest0-21swapper/000:11:230
116609954,0cyclictest5947-21awk23:48:290
116609954,0cyclictest30632-21cut23:38:350
116609954,0cyclictest20409-21awk19:18:330
116609953,0cyclictest0-21swapper/019:03:410
116609952,0cyclictest9451-21latency_hist21:43:160
116609952,0cyclictest23416-21sed20:18:310
116609952,0cyclictest14344-21munin-plugin-st22:38:170
116609951,0cyclictest6478-21cut19:08:330
116609951,0cyclictest13868-21munin-run20:58:160
116609950,0cyclictest41ktimersoftd/019:23:340
116609950,0cyclictest0-21swapper/022:43:060
116609950,0cyclictest0-21swapper/000:02:240
8225040,0irq/24-0000:00:0-21swapper/220:03:452
339940,0migration/34944-21diskmemload22:29:133
339940,0migration/320537-21kthreadcore18:53:343
339940,0migration/319931-21fschecks_count19:18:313
339940,0migration/314824-21kthreadcore19:13:343
339940,0migration/31468-21taskset19:55:353
339940,0migration/312136-21kthreadcore22:58:373
339940,0migration/312001-21kthreadcore19:33:353
28405040,0irq/26-eth0-rx-0-21swapper/323:43:423
28405040,0irq/26-eth0-rx-0-21swapper/323:40:173
28405040,0irq/26-eth0-rx-0-21swapper/323:18:293
28405040,0irq/26-eth0-rx-0-21swapper/323:03:473
28405040,0irq/26-eth0-rx-0-21swapper/322:23:303
28405040,0irq/26-eth0-rx-0-21swapper/322:13:583
28405040,0irq/26-eth0-rx-0-21swapper/322:08:533
28405040,0irq/26-eth0-rx-0-21swapper/322:03:333
28405040,0irq/26-eth0-rx-0-21swapper/318:48:353
28405040,0irq/26-eth0-rx-0-21swapper/300:13:213
28405040,0irq/26-eth0-rx-0-21swapper/300:04:013
259940,0migration/29785-21kthreadcore19:08:382
259940,0migration/223583-21kthreadcore19:48:322
249940,0watchdog/215197-21grep18:48:352
249940,0watchdog/20-21swapper/220:33:552
249940,0watchdog/20-21swapper/220:23:312
249940,0watchdog/20-21swapper/219:45:552
249940,0watchdog/20-21swapper/219:38:392
249940,0watchdog/20-21swapper/219:23:192
249940,0watchdog/20-21swapper/219:06:072
249940,0watchdog/20-21swapper/218:53:392
179940,0migration/19259-21kthreadcore19:08:371
179940,0migration/130264-21sensors_fan22:48:341
179940,0migration/129790-21kthreadcore21:58:341
179940,0migration/124995-21kthreadcore19:48:321
179940,0migration/122066-21kernelversion23:58:341
179940,0migration/121613-21kthreadcore23:33:341
179940,0migration/120891-21latency_hist22:18:171
179940,0migration/117747-21kthreadcore19:13:351
179940,0migration/116948-21munin-node20:38:281
179940,0migration/116607-21kthreadcore22:38:341
179940,0migration/113874-21aten_r1power_vo19:38:301
179940,0migration/111782-21latency_hist18:48:161
169940,0watchdog/10-21swapper/122:23:351
169940,0watchdog/10-21swapper/122:08:271
169940,0watchdog/10-21swapper/100:08:351
116709943,0cyclictest3425-21lldpd20:08:332
116619944,0cyclictest8134-21kworker/1:222:29:321
116619944,0cyclictest8134-21kworker/1:221:53:261
116619944,0cyclictest8134-21kworker/1:221:09:481
116619944,0cyclictest8134-21kworker/1:220:53:201
116619944,0cyclictest8134-21kworker/1:220:48:241
116619944,0cyclictest8134-21kworker/1:220:43:581
116619944,0cyclictest8134-21kworker/1:220:23:201
116619944,0cyclictest8134-21kworker/1:220:18:201
116619944,0cyclictest8134-21kworker/1:220:13:221
116619944,0cyclictest8134-21kworker/1:220:03:231
116619944,0cyclictest8134-21kworker/1:219:43:531
116619944,0cyclictest8134-21kworker/1:219:23:211
116619944,0cyclictest8134-21kworker/1:219:18:251
116619944,0cyclictest8134-21kworker/1:219:03:211
116619944,0cyclictest8134-21kworker/1:218:58:191
116619944,0cyclictest8134-21kworker/1:218:53:511
116619944,0cyclictest15403-21kworker/1:023:53:261
116619944,0cyclictest15403-21kworker/1:023:48:201
116619944,0cyclictest15403-21kworker/1:023:38:281
116619944,0cyclictest15403-21kworker/1:023:23:441
116619944,0cyclictest15403-21kworker/1:023:18:201
116619944,0cyclictest15403-21kworker/1:023:13:261
116619944,0cyclictest15403-21kworker/1:022:53:321
116619944,0cyclictest15403-21kworker/1:022:43:281
116619944,0cyclictest15403-21kworker/1:000:03:301
116619944,0cyclictest0-21swapper/121:43:271
116619943,0cyclictest8134-21kworker/1:222:13:281
116619943,0cyclictest8134-21kworker/1:222:04:241
116619943,0cyclictest8134-21kworker/1:221:33:181
116619943,0cyclictest7981-21awk19:33:321
116619943,0cyclictest2542-21awk21:38:301
116619943,0cyclictest15003-21awk20:58:321
116619943,0cyclictest10227-21awk20:08:281
116609944,0cyclictest238602sleep021:07:560
116609944,0cyclictest0-21swapper/021:44:460
116609943,0cyclictest9085-21awk22:58:350
116609943,0cyclictest6445-21snmpd19:48:260
116609943,0cyclictest20557-21awk23:33:330
116609943,0cyclictest0-21swapper/019:38:350
116609942,0cyclictest24995-21fschecks_count22:48:260
116609942,0cyclictest12168-21grep18:48:300
116609942,0cyclictest0-21swapper/023:28:250
116609942,0cyclictest0-21swapper/019:53:340
116609941,0cyclictest88282sleep020:04:580
116609941,0cyclictest30986-21aten_r1power_fr22:28:240
116609941,0cyclictest28970-21rm22:23:180
116609941,0cyclictest28661-21rm00:03:350
116609941,0cyclictest23037-21rm22:44:460
116609941,0cyclictest1858-21cut23:18:280
116609941,0cyclictest17743-21crond23:08:170
116609941,0cyclictest14103-21cut19:13:340
116609941,0cyclictest0-21swapper/023:53:300
116609941,0cyclictest0-21swapper/023:24:360
116609941,0cyclictest0-21swapper/023:13:320
116609941,0cyclictest0-21swapper/022:53:290
116609941,0cyclictest0-21swapper/022:13:330
116609941,0cyclictest0-21swapper/022:08:170
116609941,0cyclictest0-21swapper/021:53:290
116609941,0cyclictest0-21swapper/021:48:250
116609941,0cyclictest0-21swapper/021:28:290
116609941,0cyclictest0-21swapper/021:18:290
116609941,0cyclictest0-21swapper/021:13:310
116609941,0cyclictest0-21swapper/020:58:300
116609941,0cyclictest0-21swapper/020:43:310
116609941,0cyclictest0-21swapper/020:38:300
116609941,0cyclictest0-21swapper/020:28:160
116609941,0cyclictest0-21swapper/020:13:280
116609941,0cyclictest0-21swapper/020:08:300
116609941,0cyclictest0-21swapper/019:58:300
116609941,0cyclictest0-21swapper/019:33:280
116609941,0cyclictest0-21swapper/019:28:390
116609941,0cyclictest0-21swapper/018:58:300
116609941,0cyclictest0-21swapper/018:53:300
116609941,0cyclictest0-21swapper/000:13:330
116609940,0cyclictest10553-21tr20:33:290
28405030,0irq/26-eth0-rx-0-21swapper/323:34:053
28405030,0irq/26-eth0-rx-0-21swapper/320:43:453
28405030,0irq/26-eth0-rx-0-21swapper/320:13:353
28405030,0irq/26-eth0-rx-0-21swapper/320:04:043
28405030,0irq/26-eth0-rx-0-21swapper/319:52:233
28405030,0irq/26-eth0-rx-0-21swapper/319:43:473
249930,0watchdog/20-21swapper/219:58:272
249930,0watchdog/20-21swapper/219:53:312
249930,0watchdog/20-21swapper/219:28:352
116609931,0cyclictest41ktimersoftd/019:43:160
116609931,0cyclictest0-21swapper/021:23:170
35120,0ktimersoftd/318325-21kthreadcore19:38:363
339920,0migration/325575-21diskstats18:58:303
28405020,0irq/26-eth0-rx-0-21swapper/319:23:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional