You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-11 - 22:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Fri Apr 11, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
232999250,24cyclictest28938-21apt-get09:12:470
233599240,23cyclictest4523-21apt-get08:22:522
233599240,23cyclictest4523-21apt-get08:22:522
233199240,1cyclictest654-21cron10:40:001
2329992422,1cyclictest7113-21sendmail-msp07:20:000
232999240,1cyclictest0-21swapper/007:15:000
233999230,22cyclictest598-21rs:main3
233599230,22cyclictest11194-21apt-get12:25:022
2331992323,0cyclictest14455-21apt-get12:32:531
2331992321,1cyclictest896-21turbostat09:20:001
233199230,22cyclictest0-21swapper/107:32:441
233199230,22cyclictest0-21swapper/107:32:441
2329992320,2cyclictest17448-21awk12:40:000
232999230,2cyclictest24381-21perf07:55:000
232999230,22cyclictest0-21swapper/009:25:010
2339992220,2cyclictest8561-21sendmail11:00:013
233999220,21cyclictest27929-21apt-get11:47:503
233999220,21cyclictest23682-21turbostat.cron09:00:003
233999220,21cyclictest0-21swapper/308:42:533
2335992220,1cyclictest29179-21basename08:05:002
2335992220,1cyclictest271rcuc/211:30:002
233599220,21cyclictest25716-21apt-get10:22:442
233599220,21cyclictest11890-21latency_hist07:30:002
233599220,21cyclictest0-21swapper/210:52:492
233599220,1cyclictest612-21gdbus09:50:002
233599220,1cyclictest1534-21apt-get09:20:012
2331992221,1cyclictest12432-21apt-get09:47:511
2331992221,1cyclictest10965-21sh11:05:001
2331992221,1cyclictest0-21swapper/111:25:021
2331992220,1cyclictest593-21dbus-daemon11:40:011
2331992220,1cyclictest30846-21cron09:15:001
233199220,21cyclictest0-21swapper/112:25:021
233199220,21cyclictest0-21swapper/112:22:561
233199220,21cyclictest0-21swapper/109:59:591
233199220,21cyclictest0-21swapper/107:47:501
233199220,1cyclictest31571-21sh08:09:591
233199220,1cyclictest26545-21apt-get09:07:481
233199220,0cyclictest0-21swapper/112:07:511
2329992222,0cyclictest0-21swapper/010:22:450
2329992222,0cyclictest0-21swapper/007:22:500
2329992220,1cyclictest9550-21ls09:40:000
232999220,21cyclictest0-21swapper/012:07:520
232999220,21cyclictest0-21swapper/010:30:000
2339992121,0cyclictest0-21swapper/309:12:463
2339992120,1cyclictest1190-21apt-get10:42:453
2339992119,1cyclictest31563-21idleruntime-cro08:10:003
233999210,21cyclictest0-21swapper/312:00:033
233999210,21cyclictest0-21swapper/309:20:023
233999210,20cyclictest2097-21apt-get08:17:493
233999210,20cyclictest14830-21apt-get09:52:563
233999210,20cyclictest0-21swapper/307:52:483
233999210,1cyclictest8909-21cat09:34:593
2335992120,0cyclictest20272-21apt-get10:07:472
233599210,21cyclictest0-21swapper/212:00:032
233599210,20cyclictest24858-21apt-get07:57:482
233599210,0cyclictest0-21swapper/207:22:512
2331992121,0cyclictest30321-21apt-get11:52:491
2331992121,0cyclictest0-21swapper/111:22:531
2331992121,0cyclictest0-21swapper/109:32:501
2331992121,0cyclictest0-21swapper/108:45:011
2331992121,0cyclictest0-21swapper/107:17:481
2331992119,2cyclictest32053-21apt-get08:12:461
233199210,0cyclictest21744-21apt-get08:57:501
233199210,0cyclictest0-21swapper/110:27:511
233199210,0cyclictest0-21swapper/107:12:461
2329992120,0cyclictest0-21swapper/008:47:500
2329992118,3cyclictest0-21swapper/011:12:510
232999210,21cyclictest0-21swapper/009:07:490
232999210,21cyclictest0-21swapper/008:42:510
232999210,20cyclictest0-21swapper/011:27:490
232999210,20cyclictest0-21swapper/010:42:440
232999210,1cyclictest8142-21perf12:15:000
232999210,0cyclictest0-21swapper/012:02:460
232999210,0cyclictest0-21swapper/011:47:490
232999210,0cyclictest0-21swapper/010:47:520
232999210,0cyclictest0-21swapper/009:32:500
232999210,0cyclictest0-21swapper/008:27:490
2339992020,0cyclictest20076-21apt-get07:47:493
2339992020,0cyclictest0-21swapper/312:32:533
2339992020,0cyclictest0-21swapper/311:55:003
2339992020,0cyclictest0-21swapper/309:47:513
2339992020,0cyclictest0-21swapper/309:42:523
2339992020,0cyclictest0-21swapper/308:25:013
2339992020,0cyclictest0-21swapper/308:25:013
2339992019,1cyclictest2766-21apt-get07:12:453
2339992019,1cyclictest12362-21apt-get07:32:453
2339992019,1cyclictest12362-21apt-get07:32:453
2339992019,0cyclictest19837-21apt-get11:25:013
2339992019,0cyclictest0-21swapper/310:34:593
233999200,20cyclictest9510-21perf07:25:003
233999200,20cyclictest0-21swapper/310:12:493
233999200,20cyclictest0-21swapper/308:45:013
233999200,20cyclictest0-21swapper/307:37:493
233999200,0cyclictest10981-21apt-get11:05:023
233999200,0cyclictest0-21swapper/312:22:543
2335992020,0cyclictest0-21swapper/212:32:522
2335992020,0cyclictest0-21swapper/210:32:492
2335992020,0cyclictest0-21swapper/210:27:512
2335992020,0cyclictest0-21swapper/209:17:482
2335992018,1cyclictest6911-21apt-get08:27:482
2335992018,1cyclictest16023-21apt-get08:45:012
233599200,0cyclictest17298-21apt-get11:22:522
233599200,0cyclictest0-21swapper/209:02:522
233599200,0cyclictest0-21swapper/208:12:472
2331992020,0cyclictest0-21swapper/110:52:491
2331992020,0cyclictest0-21swapper/110:42:461
2331992020,0cyclictest0-21swapper/110:22:451
2331992020,0cyclictest0-21swapper/107:52:491
2331992020,0cyclictest0-21swapper/107:22:501
2331992019,1cyclictest0-21swapper/108:42:521
233199200,0cyclictest0-21swapper/109:42:531
233199200,0cyclictest0-21swapper/108:52:501
233199200,0cyclictest0-21swapper/108:32:501
2329992020,0cyclictest0-21swapper/011:32:490
2329992020,0cyclictest0-21swapper/010:12:490
2329992020,0cyclictest0-21swapper/009:52:550
2329992020,0cyclictest0-21swapper/007:42:520
2329992019,1cyclictest25536-21apt-get11:42:500
2329992019,1cyclictest24858-21apt-get07:57:470
232999200,0cyclictest0-21swapper/008:22:530
232999200,0cyclictest0-21swapper/008:22:530
232999200,0cyclictest0-21swapper/008:04:590
232999200,0cyclictest0-21swapper/007:47:490
2339991919,0cyclictest27276-21apt-get08:02:523
2339991919,0cyclictest17875-21apt-get10:02:463
2339991919,0cyclictest0-21swapper/312:07:523
2339991919,0cyclictest0-21swapper/311:22:523
2339991919,0cyclictest0-21swapper/311:02:463
2339991919,0cyclictest0-21swapper/309:07:493
2339991919,0cyclictest0-21swapper/307:57:483
2339991918,1cyclictest28115-21apt-get10:27:503
2339991918,1cyclictest17391-21turbostat10:00:003
2339991916,2cyclictest8766-21cron12:19:593
2335991919,0cyclictest19349-21apt-get08:52:512
2335991919,0cyclictest0-21swapper/211:42:512
2335991919,0cyclictest0-21swapper/211:07:492
2335991919,0cyclictest0-21swapper/209:32:502
2335991919,0cyclictest0-21swapper/207:32:452
2335991919,0cyclictest0-21swapper/207:32:442
2335991918,1cyclictest14297-21apt-get07:35:012
2335991918,0cyclictest22481-21apt-get11:32:502
2331991919,0cyclictest16752-21perf09:54:591
2331991919,0cyclictest11846-21apt-get11:07:501
2331991919,0cyclictest11698-21apt-get08:37:501
2331991919,0cyclictest0-21swapper/112:35:021
2331991919,0cyclictest0-21swapper/112:02:451
2331991919,0cyclictest0-21swapper/111:47:501
2331991919,0cyclictest0-21swapper/109:22:541
2331991919,0cyclictest0-21swapper/107:42:521
2331991918,1cyclictest27472-21munin-run11:45:011
2331991918,0cyclictest0-21swapper/112:12:511
2331991917,2cyclictest32436-21ls10:35:001
2329991919,0cyclictest10042-21apt-get09:42:510
2329991919,0cyclictest0-21swapper/008:17:490
2329991917,1cyclictest25227-21systemctl10:20:000
2339991818,0cyclictest0-21swapper/312:35:033
2339991818,0cyclictest0-21swapper/311:42:513
2339991818,0cyclictest0-21swapper/308:27:493
2339991818,0cyclictest0-21swapper/308:12:473
2339991817,1cyclictest31335-21apt-get09:17:493
233999180,18cyclictest0-21swapper/310:52:503
2335991818,0cyclictest0-21swapper/210:42:452
2335991817,0cyclictest0-21swapper/209:12:462
2331991818,0cyclictest0-21swapper/110:47:521
2331991818,0cyclictest0-21swapper/109:27:491
2331991818,0cyclictest0-21swapper/108:22:531
2331991818,0cyclictest0-21swapper/108:22:521
2331991818,0cyclictest0-21swapper/107:57:481
2331991817,1cyclictest15158-21apt-get07:37:491
2329991818,0cyclictest0-21swapper/012:32:520
2329991818,0cyclictest0-21swapper/011:52:490
2329991818,0cyclictest0-21swapper/011:07:500
2329991818,0cyclictest0-21swapper/010:07:480
2329991818,0cyclictest0-21swapper/009:47:500
2329991818,0cyclictest0-21swapper/008:37:500
2339991717,0cyclictest0-21swapper/312:25:023
2339991717,0cyclictest0-21swapper/308:52:513
2339991717,0cyclictest0-21swapper/308:32:493
2339991716,1cyclictest5164-21apt-get07:17:483
2339991715,2cyclictest25031-21cron11:40:013
2339991711,5cyclictest781-21lldpd11:59:503
2335991717,0cyclictest0-21swapper/210:47:522
2335991717,0cyclictest0-21swapper/208:57:522
2335991717,0cyclictest0-21swapper/208:17:492
2335991716,0cyclictest0-21swapper/209:27:482
2335991715,1cyclictest593-21dbus-daemon10:40:002
2335991714,2cyclictest8799-21awk12:20:002
233599170,1cyclictest8585-21grep11:00:012
2331991717,0cyclictest0-21swapper/108:02:531
2331991716,1cyclictest14238-21apt-get11:12:511
2329991717,0cyclictest0-21swapper/010:32:490
2329991717,0cyclictest0-21swapper/007:32:460
2329991717,0cyclictest0-21swapper/007:32:460
2329991715,1cyclictest31586-21perf08:09:590
2329991715,1cyclictest17390-21cron09:59:590
2339991616,0cyclictest0-21swapper/311:12:513
2339991616,0cyclictest0-21swapper/309:02:523
2339991615,0cyclictest0-21swapper/310:22:453
2339991614,1cyclictest0-21swapper/309:39:593
2335991616,0cyclictest0-21swapper/211:52:502
2335991616,0cyclictest0-21swapper/209:52:562
2335991616,0cyclictest0-21swapper/209:42:532
2335991616,0cyclictest0-21swapper/208:42:532
2335991616,0cyclictest0-21swapper/207:42:512
2335991614,0cyclictest0-21swapper/207:17:482
233599160,1cyclictest25090-21cat11:40:012
233599160,15cyclictest16801-21latency_hist11:19:592
2331991616,0cyclictest0-21swapper/110:12:491
2331991616,0cyclictest0-21swapper/110:07:471
2331991616,0cyclictest0-21swapper/108:27:491
2329991616,0cyclictest6000-21apt-get10:52:480
2329991616,0cyclictest0-21swapper/012:27:480
2329991616,0cyclictest0-21swapper/008:57:520
232999160,15cyclictest0-21swapper/007:37:490
2339991515,0cyclictest0-21swapper/310:46:153
2339991514,0cyclictest0-21swapper/308:37:493
2335991515,0cyclictest0-21swapper/210:12:472
2335991515,0cyclictest0-21swapper/208:32:492
2335991515,0cyclictest0-21swapper/208:07:492
2335991513,1cyclictest9200-21chrt09:35:572
233599151,13cyclictest313-21systemd-journal10:00:002
233599150,1cyclictest0-21swapper/207:52:502
233599150,15cyclictest0-21swapper/209:07:482
2331991515,0cyclictest0-21swapper/111:59:591
2331991513,1cyclictest0-21swapper/110:05:001
233199150,14cyclictest0-21swapper/109:37:011
2329991515,0cyclictest0-21swapper/010:02:450
2329991515,0cyclictest0-21swapper/008:52:490
2329991513,1cyclictest718-21munin-run10:39:590
232999150,14cyclictest9437-21munin-node08:32:420
2339991414,0cyclictest0-21swapper/311:30:543
2339991414,0cyclictest0-21swapper/311:19:533
2339991414,0cyclictest0-21swapper/310:36:223
2339991414,0cyclictest0-21swapper/310:15:503
233999140,14cyclictest0-21swapper/310:05:163
233999140,14cyclictest0-21swapper/307:42:343
233999140,14cyclictest0-21swapper/307:25:433
233999140,0cyclictest0-21swapper/309:26:523
2335991414,0cyclictest0-21swapper/208:37:022
233599140,14cyclictest0-21swapper/212:36:442
233599140,14cyclictest0-21swapper/212:12:352
233599140,14cyclictest0-21swapper/212:05:232
233599140,14cyclictest0-21swapper/211:45:132
233599140,14cyclictest0-21swapper/211:10:322
233599140,14cyclictest0-21swapper/211:02:032
233599140,14cyclictest0-21swapper/210:15:342
233599140,14cyclictest0-21swapper/210:00:372
233599140,14cyclictest0-21swapper/207:46:322
233599140,0cyclictest0-21swapper/212:20:022
2331991414,0cyclictest0-21swapper/112:15:131
2331991414,0cyclictest0-21swapper/111:19:361
2331991414,0cyclictest0-21swapper/110:57:061
2331991414,0cyclictest0-21swapper/108:15:051
233199140,14cyclictest0-21swapper/109:02:581
233199140,14cyclictest0-21swapper/107:25:191
2329991414,0cyclictest0-21swapper/010:55:430
232999140,14cyclictest0-21swapper/012:17:370
232999140,14cyclictest0-21swapper/011:56:300
232999140,14cyclictest0-21swapper/011:35:400
232999140,14cyclictest0-21swapper/011:20:540
232999140,14cyclictest0-21swapper/011:17:220
232999140,14cyclictest0-21swapper/009:26:490
232999140,14cyclictest0-21swapper/007:25:180
232999140,0cyclictest0-21swapper/011:00:240
232999140,0cyclictest0-21swapper/009:00:250
2335991313,0cyclictest0-21swapper/207:10:402
233599130,0cyclictest0-21swapper/211:56:162
2329991313,0cyclictest0-21swapper/008:12:470
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional