You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 04:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Wed Apr 02, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
366899280,27cyclictest25604-21sh19:55:002
366899280,22cyclictest1377-21apt-get21:27:002
366399250,1cyclictest6167-21munin-run00:20:011
367099240,23cyclictest5922-21awk23:00:003
366899240,23cyclictest334-21cron20:10:002
365999240,1cyclictest593-21dbus-daemon19:20:000
3670992319,3cyclictest23780-21awk21:05:013
367099230,1cyclictest30236-21mandb00:00:003
3668992320,3cyclictest0-21swapper/222:27:002
3668992320,3cyclictest0-21swapper/221:42:012
3668992319,3cyclictest14372-21apt-get23:20:022
3668992317,5cyclictest14210-21munin-plugin-st20:45:002
3663992321,1cyclictest11138-21dump-pmu-power20:35:001
366399230,2cyclictest10951-21munin-plugin-st00:30:001
366399230,1cyclictest19646-21irqstats19:42:071
3659992323,0cyclictest0-21swapper/020:02:060
3659992320,2cyclictest3504-21apt00:12:020
365999230,22cyclictest27771-21apt-get22:32:000
365999230,22cyclictest18432-21perf19:40:000
365999230,22cyclictest0-21swapper/021:27:000
365999230,0cyclictest1197-21apt-get20:12:010
3670992220,1cyclictest6564-21apt-get00:21:563
3670992220,1cyclictest6358-21awk20:25:013
367099220,21cyclictest8430-21perf19:20:003
367099220,21cyclictest30411-21basename22:40:013
367099220,21cyclictest30410-21cron20:05:003
367099220,21cyclictest29566-21dump-pmu-power23:55:003
367099220,21cyclictest0-21swapper/321:26:593
367099220,21cyclictest0-21swapper/320:47:003
367099220,21cyclictest0-21swapper/320:27:003
3668992220,1cyclictest13758-21munin-run23:20:002
366899220,21cyclictest0-21swapper/223:47:032
366899220,21cyclictest0-21swapper/220:23:002
3663992220,1cyclictest8712-21apt-get23:07:011
3663992220,1cyclictest0-21swapper/100:10:001
366399220,21cyclictest8894-21dump-pmu-power21:45:001
366399220,21cyclictest25195-21apt-get23:47:031
366399220,21cyclictest21338-21grep23:35:011
366399220,21cyclictest0-21swapper/121:47:001
366399220,21cyclictest0-21swapper/120:21:581
366399220,1cyclictest8422-21apt-get19:20:021
366399220,1cyclictest14610-21apt-get20:47:001
366399220,1cyclictest13078-21idleruntime-cro23:15:001
3659992221,1cyclictest22395-21apt-get23:41:550
3659992220,1cyclictest674-21perf00:05:000
3659992220,1cyclictest18935-21awk23:30:010
3659992220,1cyclictest0-21swapper/023:40:000
365999220,21cyclictest22577-21perf22:20:000
365999220,1cyclictest13679-21grep21:55:000
3670992120,0cyclictest19527-21apt-get22:11:593
3670992120,0cyclictest0-21swapper/320:41:563
3670992119,1cyclictest593-21dbus-daemon23:30:003
367099210,21cyclictest0-21swapper/300:27:023
367099210,20cyclictest0-21swapper/323:47:033
367099210,20cyclictest0-21swapper/323:01:573
367099210,20cyclictest0-21swapper/322:47:003
367099210,20cyclictest0-21swapper/321:32:013
367099210,20cyclictest0-21swapper/321:17:013
367099210,20cyclictest0-21swapper/319:32:003
367099210,20cyclictest0-21swapper/300:12:023
367099210,20cyclictest0-21swapper/300:07:013
367099210,0cyclictest0-21swapper/319:47:003
367099210,0cyclictest0-21swapper/319:36:593
367099210,0cyclictest0-21swapper/300:01:563
3668992121,0cyclictest0-21swapper/223:32:042
3668992121,0cyclictest0-21swapper/220:57:012
3668992121,0cyclictest0-21swapper/200:12:012
3668992119,1cyclictest14342-21munin-run22:00:012
366899210,21cyclictest0-21swapper/219:22:002
366899210,20cyclictest28385-21grep22:32:032
366899210,20cyclictest21236-21apt-get19:47:002
366899210,20cyclictest18443-21basename19:40:012
366899210,20cyclictest13963-21sh00:40:002
366899210,20cyclictest0-21swapper/223:01:562
366899210,20cyclictest0-21swapper/222:15:012
366899210,20cyclictest0-21swapper/200:07:002
366899210,0cyclictest0-21swapper/221:47:012
366899210,0cyclictest0-21swapper/220:10:022
3663992121,0cyclictest0-21swapper/120:15:001
3663992120,1cyclictest0-21swapper/123:01:561
3663992119,1cyclictest21414-21munin-run21:00:001
3663992119,1cyclictest0-21swapper/121:05:001
366399210,20cyclictest25373-21apt-get22:27:001
366399210,20cyclictest0-21swapper/119:47:001
366399210,1cyclictest16427-21apt-get19:37:001
366399210,0cyclictest0-21swapper/121:51:591
366399210,0cyclictest0-21swapper/120:06:591
3659992120,1cyclictest790-21apt-get22:46:590
3659992120,1cyclictest17140-21apt-get22:07:020
3659992120,1cyclictest0-21swapper/023:45:000
3659992120,0cyclictest18845-21apt-get19:42:020
3659992119,1cyclictest5917-21munin-run23:00:000
3659992119,1cyclictest30246-21munin-plugin-st23:59:590
3659992118,2cyclictest13-21ksoftirqd/023:20:010
365999210,21cyclictest6031-21perf19:15:000
365999210,1cyclictest0-21swapper/000:39:590
365999210,0cyclictest1099-21apt-get00:07:010
365999210,0cyclictest0-21swapper/000:27:030
3670992020,0cyclictest0-21swapper/322:41:573
3670992020,0cyclictest0-21swapper/320:07:003
3670992019,1cyclictest4042-21apt-get19:12:013
3670992018,1cyclictest11296-21awk21:50:013
367099200,0cyclictest0-21swapper/321:22:023
367099200,0cyclictest0-21swapper/321:07:003
3668992020,0cyclictest30820-21apt-get22:41:572
3668992020,0cyclictest11689-21apt-get21:52:002
3668992020,0cyclictest0-21swapper/223:27:012
3668992019,0cyclictest11104-21apt-get23:12:002
366899200,19cyclictest25220-21ntpq21:07:042
366899200,19cyclictest20946-21cat19:45:002
366899200,19cyclictest0-21swapper/222:07:012
366899200,0cyclictest0-21swapper/221:17:002
366899200,0cyclictest0-21swapper/220:31:592
366899200,0cyclictest0-21swapper/219:57:002
3663992020,0cyclictest0-21swapper/123:41:561
3663992020,0cyclictest0-21swapper/123:20:021
3663992020,0cyclictest0-21swapper/122:41:571
3663992020,0cyclictest0-21swapper/121:06:591
3663992019,1cyclictest21516-21perf22:15:001
3663992018,1cyclictest612-21gdbus00:35:001
3663992018,1cyclictest28444-21apt-get20:02:051
3663992018,1cyclictest0-21swapper/122:09:591
3663992018,1cyclictest0-21swapper/120:20:001
366399200,19cyclictest0-21swapper/119:57:001
366399200,0cyclictest0-21swapper/122:01:541
3659992020,0cyclictest30829-21apt-get20:07:000
3659992020,0cyclictest19397-21apt-get20:57:000
3659992020,0cyclictest0-21swapper/023:07:010
3659992020,0cyclictest0-21swapper/022:41:560
3659992020,0cyclictest0-21swapper/022:21:560
3659992020,0cyclictest0-21swapper/022:11:590
3659992020,0cyclictest0-21swapper/019:46:590
3659992020,0cyclictest0-21swapper/000:32:010
3659992019,0cyclictest3849-21apt-get21:32:000
365999200,19cyclictest23628-21apt-get19:52:000
365999200,0cyclictest26021-21apt-get19:56:590
3670991919,0cyclictest0-21swapper/323:41:553
3670991919,0cyclictest0-21swapper/323:35:003
3670991919,0cyclictest0-21swapper/321:42:013
3670991919,0cyclictest0-21swapper/321:12:013
3670991919,0cyclictest0-21swapper/320:52:013
3670991919,0cyclictest0-21swapper/319:52:013
367099190,0cyclictest0-21swapper/319:22:003
3668991919,0cyclictest0-21swapper/221:32:012
3668991919,0cyclictest0-21swapper/219:17:002
3668991919,0cyclictest0-21swapper/200:01:562
3668991918,1cyclictest11353-21apt-get00:32:012
3668991917,1cyclictest6151-21apt00:20:002
366899190,18cyclictest0-21swapper/223:41:562
366899190,0cyclictest3288-21cpu22:52:012
366899190,0cyclictest0-21swapper/222:47:052
366899190,0cyclictest0-21swapper/222:01:552
366899190,0cyclictest0-21swapper/220:27:042
3663991919,0cyclictest0-21swapper/121:27:001
3663991919,0cyclictest0-21swapper/120:53:171
3663991919,0cyclictest0-21swapper/119:52:011
3663991918,1cyclictest16955-21apt-get23:27:021
3663991918,0cyclictest31122-21apt-get00:01:551
3663991918,0cyclictest11639-21apt-get19:27:001
365999192,16cyclictest6315-21apt-get23:01:570
3659991919,0cyclictest21818-21apt-get21:02:070
3659991919,0cyclictest0-21swapper/020:41:550
3659991918,1cyclictest9245-21apt-get19:22:010
3659991917,1cyclictest0-21swapper/022:40:000
3670991818,0cyclictest0-21swapper/322:52:013
3670991818,0cyclictest0-21swapper/322:15:013
3670991818,0cyclictest0-21swapper/320:12:013
367099180,17cyclictest0-21swapper/300:32:003
3668991818,0cyclictest0-21swapper/220:35:472
3668991818,0cyclictest0-21swapper/200:21:562
3668991816,1cyclictest10709-21munin-plugin-st23:09:592
366899180,17cyclictest0-21swapper/200:27:092
366899180,0cyclictest0-21swapper/223:51:592
3663991818,0cyclictest6564-21apt-get00:21:571
3663991818,0cyclictest0-21swapper/122:46:591
3663991818,0cyclictest0-21swapper/121:32:011
3663991818,0cyclictest0-21swapper/121:25:001
3663991818,0cyclictest0-21swapper/121:17:011
3663991817,1cyclictest3505-21apt-get00:12:011
3659991818,0cyclictest9164-21apt-get20:32:000
3659991818,0cyclictest31410-21apt-get21:22:030
3659991818,0cyclictest0-21swapper/023:12:010
3659991818,0cyclictest0-21swapper/019:27:000
3670991717,0cyclictest0-21swapper/323:07:013
3670991716,0cyclictest11639-21apt-get19:26:593
367099170,14cyclictest0-21swapper/320:32:043
3668991717,0cyclictest0-21swapper/222:11:582
3663991717,0cyclictest0-21swapper/119:12:001
3663991715,1cyclictest6771-21apt-get20:27:011
3663991715,1cyclictest30210-21dump-pmu-power23:59:591
3663991715,1cyclictest13743-21apt-get23:20:001
3659991717,0cyclictest28994-21apt-get21:17:010
3659991717,0cyclictest24203-21apt-get21:07:000
3659991717,0cyclictest0-21swapper/022:27:000
3659991717,0cyclictest0-21swapper/020:27:010
3659991715,1cyclictest0-21swapper/020:25:010
365999170,16cyclictest14032-21apt-get19:32:010
3670991616,0cyclictest0-21swapper/323:12:003
3670991616,0cyclictest0-21swapper/322:31:593
3670991616,0cyclictest0-21swapper/322:01:563
3670991616,0cyclictest0-21swapper/321:52:003
3670991614,1cyclictest3920-21munin-run20:20:013
3670991614,1cyclictest14348-21sendmail-msp22:00:013
367099160,15cyclictest14012-21awk00:39:593
3668991616,0cyclictest0-21swapper/222:21:562
3668991614,1cyclictest21944-21cron23:40:002
3668991611,4cyclictest0-21swapper/219:12:002
366899160,9cyclictest3903-21sendmail-msp20:20:002
366899160,15cyclictest30248-21/usr/sbin/munin00:00:002
366899160,0cyclictest23288-21ssh21:02:152
3663991616,0cyclictest0-21swapper/122:21:561
3659991616,0cyclictest0-21swapper/020:46:590
3659991614,1cyclictest0-21swapper/023:32:120
365999160,15cyclictest7949-21ntp_states21:42:050
365999160,15cyclictest14975-21diskstats23:21:580
3670991515,0cyclictest14553-21apt-get23:21:563
3670991515,0cyclictest0-21swapper/322:07:023
367099150,14cyclictest0-21swapper/319:42:043
3668991515,0cyclictest0-21swapper/219:27:002
3668991514,1cyclictest0-21swapper/219:34:292
3668991513,1cyclictest5882-21cron22:59:592
3663991513,1cyclictest29739-21cron22:35:001
3663991513,1cyclictest11804-21idleruntime-cro20:40:001
3659991513,1cyclictest0-21swapper/021:50:000
3670991414,0cyclictest0-21swapper/323:18:003
3670991414,0cyclictest0-21swapper/322:21:083
3670991414,0cyclictest0-21swapper/320:57:013
367099140,14cyclictest0-21swapper/323:39:323
367099140,14cyclictest0-21swapper/321:36:573
367099140,14cyclictest0-21swapper/319:57:053
367099140,0cyclictest0-21swapper/322:25:083
367099140,0cyclictest0-21swapper/320:35:493
367099140,0cyclictest0-21swapper/300:19:323
366899147,6cyclictest26566-21/usr/sbin/munin21:12:082
3668991412,1cyclictest0-21swapper/221:22:032
366899140,14cyclictest0-21swapper/221:36:402
366899140,0cyclictest0-21swapper/220:53:172
3663991414,0cyclictest26598-21apt-get21:12:001
3663991414,0cyclictest0-21swapper/122:55:271
3663991414,0cyclictest0-21swapper/122:15:421
3663991414,0cyclictest0-21swapper/121:55:551
366399140,14cyclictest0-21swapper/123:52:121
366399140,14cyclictest0-21swapper/123:36:281
366399140,14cyclictest0-21swapper/121:36:331
366399140,0cyclictest0-21swapper/120:42:001
366399140,0cyclictest0-21swapper/100:35:281
3659991414,0cyclictest0-21swapper/021:12:100
3659991414,0cyclictest0-21swapper/020:52:000
3659991414,0cyclictest0-21swapper/020:15:440
3659991414,0cyclictest0-21swapper/000:20:530
365999140,14cyclictest0-21swapper/020:36:580
365999140,0cyclictest0-21swapper/022:02:520
365999140,0cyclictest0-21swapper/021:35:330
365999140,0cyclictest0-21swapper/000:16:070
3668991313,0cyclictest0-21swapper/222:35:202
3668991313,0cyclictest0-21swapper/220:02:062
3663991313,0cyclictest0-21swapper/122:38:201
3659991313,0cyclictest0-21swapper/021:55:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional