You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-12 - 05:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Mon Mar 10, 2025 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4149039993732,3cyclictest4160809-40gst-plugin-scan07:39:142
4149039993732,3cyclictest4160809-40gst-plugin-scan07:39:142
4149037993626,4cyclictest4193463-21dump-pmu-power09:05:011
4149039993121,7cyclictest99554-21grep11:20:232
4149043993025,3cyclictest4160802-40gmain07:39:143
4149043993025,3cyclictest4160802-40gmain07:39:133
4149037993019,7cyclictest1-21systemd10:44:591
4149039992925,2cyclictest0-21swapper/212:15:012
4149037992925,3cyclictest105619-21cron11:29:591
4149043992824,2cyclictest0-21swapper/309:25:003
4149043992819,6cyclictest16121-21sh09:30:003
4149037992823,3cyclictest4160772-40tracker-extract07:39:131
4149037992823,3cyclictest4160772-40tracker-extract07:39:121
4149034992824,2cyclictest97597-21sendmail11:20:000
414903999276,17cyclictest75522-21grep10:50:192
414903499278,6cyclictest4186277-21uname08:45:180
4149034992622,2cyclictest0-21swapper/012:10:010
4149034992621,3cyclictest106972-21mailstats11:30:190
414903499255,17cyclictest42733-21sh10:06:260
4149034992521,3cyclictest4160772-40tracker-extract07:39:130
4149034992521,3cyclictest4160772-40tracker-extract07:39:120
4149034992513,8cyclictest123278-21basename11:50:090
4149043992414,7cyclictest124207-21grep11:50:203
414904399241,20cyclictest4161945-21ntpq07:40:183
4149039992420,2cyclictest153730-21apt-get12:20:012
4149037992419,3cyclictest88060-21sh11:07:421
4149037992418,4cyclictest126270-21ssh11:53:321
414903499244,2cyclictest1455-21diskmemload09:44:270
414903499244,2cyclictest1455-21diskmemload09:44:270
4149034992420,2cyclictest712-21tpm2-abrmd10:15:010
4149034992419,3cyclictest90382-21needreboot11:10:130
4149043992319,3cyclictest153689-21/usr/sbin/munin12:20:013
4149043992317,5cyclictest0-21swapper/310:20:013
4149043992313,7cyclictest699-21dbus-daemon12:33:583
414904399231,19cyclictest111509-21ntpq11:35:313
4149039992318,3cyclictest1492-21sh09:10:012
4149039992314,6cyclictest4163770-21grep07:45:122
4149039992312,9cyclictest0-21swapper/212:24:592
414903799235,15cyclictest43907-21sh10:08:421
4149037992319,2cyclictest37788-21cron09:59:591
4149037992318,3cyclictest712-21tpm2-abrmd10:15:011
414903499234,5cyclictest158909-21sh12:24:130
4149034992319,2cyclictest52583-21cron10:20:010
4149034992319,2cyclictest114752-21munin-run11:40:000
4149034992318,3cyclictest127740-21cpuspeed_turbos11:55:130
414903499231,19cyclictest60431-21ntpq10:30:150
4149034992311,8cyclictest133162-21awk12:00:100
4149043992219,2cyclictest4165181-21systemd-run07:50:013
4149043992217,3cyclictest0-21swapper/311:00:103
4149039992218,2cyclictest0-21swapper/210:50:012
4149039992217,3cyclictest0-21swapper/207:15:012
4149039992212,7cyclictest107135-21wc11:30:212
414903999221,18cyclictest5969-21cut09:15:172
4149039992211,8cyclictest119684-21ssh11:45:162
414903999220,19cyclictest13630-21cron09:25:002
414903799224,4cyclictest4164316-21grep07:45:191
414903499222,17cyclictest8240-21sh09:17:450
4149034992218,2cyclictest1455-21diskmemload09:47:150
414903499221,7cyclictest150226-21runrttasks12:15:380
414903499221,17cyclictest104260-21ssh11:27:430
4149043992117,2cyclictest715-21in:imuxsock11:09:193
4149043992117,2cyclictest4189522-21idleruntime-cro08:54:593
4149043992117,2cyclictest34586-21fschecks_time09:55:133
4149043992117,2cyclictest0-21swapper/307:29:593
4149043992113,5cyclictest4175968-21cut08:15:473
4149043992112,6cyclictest262-21systemd-journal12:36:193
414904399211,17cyclictest4179628-21cron08:29:593
414904399211,16cyclictest138547-21latency_hist12:05:013
414904399210,19cyclictest43881-21sh10:08:393
414904399210,18cyclictest16750-21ssh09:30:103
414903999211,7cyclictest940-21runrttasks10:16:082
4149039992114,5cyclictest30218-21kworker/u16:0+rpciod10:38:542
4149039992113,5cyclictest63375-21sort10:35:002
4149039992112,8cyclictest1589427-21kworker/u16:1+rpciod09:28:392
4149039992112,6cyclictest4157732-21memory07:30:592
414903999211,17cyclictest39591-21taskset10:00:292
4149039992111,7cyclictest25799-21taskset09:41:472
4149039992111,7cyclictest25799-21taskset09:41:462
414903999211,17cyclictest161937-21ssh12:27:382
414903999210,18cyclictest940-21runrttasks11:37:242
414903999210,18cyclictest42410-21ssh10:05:482
414903799219,8cyclictest262-21systemd-journal10:21:191
414903799215,3cyclictest331ktimers/109:40:001
4149037992116,3cyclictest4169159-21idleruntime-cro08:00:001
4149037992116,3cyclictest148897-21gltestperf12:15:131
4149037992113,5cyclictest117673-21sh11:42:261
4149037992112,6cyclictest167340-21sh12:32:211
4149037992112,6cyclictest155265-21ssh12:20:201
414903799211,17cyclictest38612-21awk10:00:151
414903499216,13cyclictest1455-21diskmemload12:39:160
4149034992118,2cyclictest2293-21kernelversion09:10:190
4149034992117,2cyclictest4177002-21sh08:20:000
4149034992117,2cyclictest1-21systemd07:31:030
4149034992116,3cyclictest703-21polkitd07:40:210
4149034992116,3cyclictest53943-21processes10:20:220
4149034992114,2cyclictest191rcu_preempt09:35:000
4149034992112,6cyclictest167965-21sh12:33:070
414903499210,17cyclictest41835-21latency_hist10:05:010
414904399208,8cyclictest942-21snmpd11:29:193
4149043992017,2cyclictest20256-21gltestperf09:35:143
4149043992016,3cyclictest4183569-21sendmail08:40:003
4149043992015,3cyclictest940-21runrttasks09:05:003
414904399201,17cyclictest4133787-21munin-node10:59:403
4149043992011,6cyclictest940-21runrttasks09:15:523
4149043992011,6cyclictest726-21gdbus07:55:013
4149043992010,8cyclictest0-21swapper/312:20:193
4149043992010,7cyclictest121282-21sshd11:47:593
414903999209,7cyclictest262-21systemd-journal07:55:182
414903999201,6cyclictest136141-21ssh12:02:142
4149039992015,3cyclictest0-21swapper/211:55:142
4149039992014,4cyclictest4161387-21fschecks_count07:40:132
414903999201,16cyclictest32844-21ssh09:52:512
4149039992010,7cyclictest4182623-21collect208:35:322
414903999200,5cyclictest2327150ssh09:39:572
414903999200,17cyclictest97081-21ssh11:19:162
414903799201,6cyclictest92268-21ssh11:12:421
414903799201,6cyclictest4186138-21head08:45:171
414903799201,17cyclictest28483-21ssh09:46:421
4149037992010,6cyclictest262-21systemd-journal12:26:001
4149037992010,6cyclictest262-21systemd-journal07:10:191
4149037992010,6cyclictest170040-21df_inode12:35:151
414903799200,17cyclictest81109-21ssh10:58:111
414903799200,17cyclictest27321-21sh09:45:001
414903799200,17cyclictest27321-21sh09:45:001
414903799200,17cyclictest18806-21sh09:33:091
414903499209,7cyclictest722-21gdbus08:05:010
414903499204,5cyclictest118562-21sh11:43:490
414903499201,6cyclictest32761-21sh09:52:430
414903499201,6cyclictest147475-21ssh12:14:090
4149034992012,6cyclictest4184369-21/usr/sbin/munin08:40:160
414903499201,16cyclictest83307-21ntpq11:00:170
414904399194,4cyclictest69914-21sh10:42:173
414904399191,8cyclictest940-21runrttasks10:02:063
414904399191,7cyclictest24528-21ssh09:40:203
414904399191,7cyclictest24528-21ssh09:40:193
4149043991917,1cyclictest1455-21diskmemload10:35:373
414904399191,6cyclictest4173912-21/usr/sbin/munin08:10:203
4149043991915,3cyclictest90088-21gltestperf11:10:093
4149043991914,3cyclictest4168040-21needreboot07:55:183
4149043991913,4cyclictest0-21swapper/311:30:223
4149043991912,5cyclictest1455-21diskmemload10:24:073
4149043991912,4cyclictest29831-21sh09:49:193
4149043991911,6cyclictest75870-21runrttasks10:50:463
414904399191,16cyclictest58184-21sh10:27:383
414904399191,16cyclictest46875-21sh10:11:503
414904399191,16cyclictest147728-21ssh12:14:293
414904399191,15cyclictest131427-21ssh11:58:223
4149043991910,6cyclictest262-21systemd-journal09:11:363
414903999199,7cyclictest4175885-21memory08:15:482
414903999199,7cyclictest117248-21ssh11:41:432
4149039991915,2cyclictest37807-21sendmail10:00:002
4149039991915,2cyclictest1206-21sendmail-mta08:50:002
4149039991915,2cyclictest0-21swapper/212:30:092
4149039991915,2cyclictest0-21swapper/209:10:152
4149039991910,7cyclictest0-21swapper/210:20:262
4149039991910,6cyclictest29427-21taskset09:48:322
414903999190,6cyclictest82444-21taskset11:00:002
414903999190,16cyclictest80256-21ssh10:56:022
414903999190,16cyclictest4156022-21ntpq07:26:212
414903799199,7cyclictest4173622-21perl08:10:181
414903799199,6cyclictest4189768-21apt-get08:55:131
414903799191,7cyclictest9922-21runrttasks09:20:051
4149037991915,2cyclictest78091-21latency_hist10:55:001
4149037991915,2cyclictest67224-21apt10:40:001
4149037991914,3cyclictest99134-21pmu-power11:20:191
4149037991914,3cyclictest4170677-21unixbench_multi08:00:241
4149037991914,3cyclictest0-21swapper/109:29:591
414903799191,16cyclictest139124-21sh12:05:391
4149037991911,5cyclictest49313-21load10:15:171
4149037991910,6cyclictest4122-21ssh09:12:271
4149037991910,6cyclictest107354-21grep11:30:221
414903799190,16cyclictest110647-21ssh11:35:031
414903499199,7cyclictest20924-21cut09:35:230
414903499198,8cyclictest4182571-21ntpq08:35:310
414903499198,7cyclictest34882-21memory09:55:140
414903499191,7cyclictest159955-21ssh12:25:110
4149034991915,2cyclictest0-21swapper/010:40:140
4149034991914,3cyclictest4178060-21cpuspeed_turbos08:25:450
4149034991914,3cyclictest4155174-21cpu07:25:130
4149034991912,5cyclictest120756-21sh11:47:100
414903499191,16cyclictest59149-21ssh10:29:390
414903499191,16cyclictest4150226-21systemctl07:10:240
414903499190,6cyclictest4173514-21/usr/sbin/munin08:10:160
4149043991814,2cyclictest0-21swapper/309:10:013
4149043991811,5cyclictest4179913-21apt-get08:30:153
4149043991811,4cyclictest100690-21sh11:22:093
4149043991810,5cyclictest4154894-21turbostat.cron07:25:003
4149043991810,5cyclictest162097-21ssh12:27:503
414904399180,7cyclictest61441-21runrttasks10:31:123
414903999181,5cyclictest1455-21diskmemload12:07:522
4149039991812,4cyclictest1455-21diskmemload11:53:212
4149039991812,4cyclictest1455-21diskmemload11:14:522
414903999181,15cyclictest56965-21ssh10:25:052
4149039991811,4cyclictest46258-21sh10:10:312
4149039991810,5cyclictest4185160-21grep08:40:232
414903999180,16cyclictest4170095-21ld08:00:172
414903999180,14cyclictest4178741-21ntpq08:25:512
414903799189,7cyclictest0-21swapper/110:25:471
414903799187,7cyclictest4171656-21perl08:05:151
414903799187,7cyclictest262-21systemd-journal10:46:471
4149037991814,2cyclictest82899-21gltestperf11:00:121
4149037991811,5cyclictest0-21swapper/111:15:151
4149037991810,6cyclictest4188358-21ls08:50:201
4149037991810,6cyclictest146359-21ssh12:12:461
4149037991810,6cyclictest130937-21ssh11:57:451
4149037991810,6cyclictest120240-21ssh11:46:181
414903799180,6cyclictest4155614-21/usr/sbin/munin07:25:171
414903799180,6cyclictest136763-21ssh12:03:001
414903499189,6cyclictest699-21dbus-daemon11:20:020
414903499189,6cyclictest4153198-21apt07:20:140
414903499186,8cyclictest9994-21apt-get09:20:130
414903499181,15cyclictest14338-21sh09:25:590
4149034991810,6cyclictest80702-21ssh10:57:080
414903499180,16cyclictest65704-21ssh10:36:540
414903499180,15cyclictest75693-21ssh10:50:300
4149043991713,2cyclictest0-21swapper/308:25:003
4149043991711,4cyclictest1455-21diskmemload12:06:433
4149043991710,5cyclictest97011-21sh11:19:003
4149043991710,4cyclictest4149386-21tr07:10:143
414903999178,7cyclictest170604-21ntpq12:35:202
414903999178,6cyclictest103339-21ssh11:26:102
4149039991714,2cyclictest488-21hwrng11:06:532
4149039991712,3cyclictest4188087-21gltestperf08:50:182
414903999171,14cyclictest262-21systemd-journal08:55:012
414903799177,6cyclictest699-21dbus-daemon08:25:001
414903799176,7cyclictest699-21dbus-daemon08:30:241
414903799171,7cyclictest4194072-21tail09:05:141
414903799171,6cyclictest4184640-21ldconfig08:40:181
4149037991713,2cyclictest4175408-21cpuspeed_turbos08:15:441
414903799171,14cyclictest6488-21ld09:15:201
414903799171,13cyclictest4161370-21diskstats07:40:131
414903799170,6cyclictest940-21runrttasks10:30:261
414903799170,15cyclictest4155061-21latency_hist07:25:011
414903499178,6cyclictest4190174-21cat08:55:160
414903499176,7cyclictest4167719-21grep07:55:150
4149034991713,3cyclictest4180539-21needreboot08:30:190
414903499171,13cyclictest4194156-21/usr/sbin/munin09:05:140
414903499170,6cyclictest88722-21sshd11:08:470
414903499170,15cyclictest4152239-21/usr/sbin/munin07:15:240
414904399167,6cyclictest4185834-21apt-get08:45:143
414904399161,3cyclictest4183847-21/usr/sbin/munin08:40:163
4149043991611,3cyclictest4171442-21cpuspeed_turbos08:05:123
4149043991610,4cyclictest4169754-21gltestperf08:00:143
414904399160,12cyclictest4151951-21ntp_kernel_pll_07:15:183
414903999167,6cyclictest4173550-21cut08:10:182
414903999165,7cyclictest726-21gdbus07:15:202
4149039991612,2cyclictest4153620-21/usr/sbin/munin07:20:182
4149039991610,3cyclictest163882sleep209:30:042
414903799168,5cyclictest34-21ksoftirqd/107:15:151
414903799167,6cyclictest699-21dbus-daemon07:31:061
4149037991612,3cyclictest4179201-21unixbench_multi08:25:561
414903799160,14cyclictest4165727-21grep07:50:101
414903799160,14cyclictest4133787-21munin-node08:35:111
414903499167,6cyclictest4163585-21cut07:45:100
414903499166,7cyclictest4172109-21ntpq08:05:190
414903499166,6cyclictest4193513-21sed09:05:010
414903499166,6cyclictest4188045-21tune2fs08:50:180
414904399156,6cyclictest4157993-21open_files07:31:033
414904399151,12cyclictest4189879-21grep08:55:133
4149043991511,2cyclictest1455-21diskmemload10:45:483
414903999157,5cyclictest4166253-21cut07:50:162
414903999151,11cyclictest4192531-21ntpq09:00:222
4149034991511,3cyclictest4166130-21ntpq07:50:140
4149034991511,2cyclictest262-21systemd-journal08:24:180
414903999144,7cyclictest4180241-21grep08:30:182
414903999140,11cyclictest4177707-21systemd-run08:25:002
414903999139,2cyclictest0-21swapper/208:05:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional