You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-13 - 20:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Mar 13, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
71402237163,26sleep20-21swapper/207:09:342
69362236160,25sleep30-21swapper/307:06:573
68472236163,59sleep10-21swapper/107:05:471
69382202165,24sleep00-21swapper/007:06:580
2862821710,4sleep1726899cyclictest09:11:001
1297921690,5sleep1726899cyclictest11:30:091
70521460,3sleep1726899cyclictest10:55:131
7272997875,2cyclictest0-21swapper/211:21:082
16452660,3sleep3727899cyclictest10:55:223
7272994946,2cyclictest0-21swapper/209:03:442
7272994542,2cyclictest0-21swapper/207:37:152
726899351,31cyclictest24225-21/usr/sbin/munin10:30:141
726899350,8cyclictest0-21swapper/110:50:301
726899341,16cyclictest0-21swapper/109:20:201
726899341,14cyclictest0-21swapper/108:05:221
726899321,21cyclictest0-21swapper/111:24:121
726299328,2cyclictest10876-21kworker/0:009:56:250
726299321,9cyclictest19555-21kworker/0:311:50:210
726299321,29cyclictest12413-21kworker/0:011:33:050
726299321,12cyclictest28763-21kworker/0:112:18:350
726899311,11cyclictest17234-21/usr/sbin/munin08:45:241
726899311,11cyclictest0-21swapper/110:44:021
7262993113,3cyclictest0-21swapper/010:40:210
726899301,12cyclictest0-21swapper/109:51:041
7262993016,6cyclictest5823-21kworker/0:012:39:210
726899296,12cyclictest0-21swapper/108:50:201
7268992911,5cyclictest20428-21perf10:20:021
726899291,13cyclictest13160-21grep08:35:281
726899291,11cyclictest0-21swapper/110:12:241
726899290,20cyclictest0-21swapper/109:40:021
726299299,3cyclictest1715-21kworker/0:312:30:210
726299290,15cyclictest19555-21kworker/0:312:04:230
47152290,3sleep011-21rcuc/009:35:190
47152290,3sleep011-21rcuc/009:35:190
7268992813,2cyclictest0-21swapper/107:35:241
726899281,18cyclictest0-21swapper/111:26:421
726899281,12cyclictest0-21swapper/108:20:241
726899281,11cyclictest0-21swapper/111:06:041
726299288,3cyclictest17865-21kworker/0:310:15:190
51872280,3sleep0726299cyclictest08:15:180
726899275,10cyclictest30161-21interrupts12:15:181
726899271,4cyclictest32474-21memory12:20:201
7268992712,3cyclictest31119-21needreboot08:00:201
726899271,15cyclictest0-21swapper/109:05:221
726899271,12cyclictest0-21swapper/111:10:101
726299278,16cyclictest13555-21munin-run08:40:000
726299274,10cyclictest0-21swapper/008:00:020
7262992715,2cyclictest30257-21kworker/0:311:09:450
7262992713,3cyclictest30257-21kworker/0:310:55:110
7262992713,3cyclictest1045-21kworker/0:212:28:090
7262992711,3cyclictest26502-21kworker/0:309:10:270
7278992610,13cyclictest20697-21grep07:35:233
726899266,3cyclictest0-21swapper/111:48:021
726899266,3cyclictest0-21swapper/111:48:021
726899266,3cyclictest0-21swapper/110:46:001
726899265,6cyclictest0-21swapper/112:34:361
726899261,15cyclictest0-21swapper/111:38:341
726899261,10cyclictest0-21swapper/109:15:081
7268992610,5cyclictest0-21swapper/112:39:221
7268992610,2cyclictest0-21swapper/111:50:161
726299267,3cyclictest31743-21kworker/0:008:05:480
7262992612,3cyclictest6004-21kworker/0:009:45:060
726299261,12cyclictest21886-21kworker/0:107:14:130
726299260,13cyclictest27488-21kworker/0:110:51:590
726899255,4cyclictest2825-21tr11:00:141
726899255,2cyclictest0-21swapper/110:10:021
726899253,7cyclictest0-21swapper/112:12:441
7268992516,3cyclictest0-21swapper/110:35:201
726899251,12cyclictest0-21swapper/108:14:421
726899251,10cyclictest0-21swapper/107:53:221
726299259,3cyclictest3641-21kworker/0:109:42:120
726299257,3cyclictest17865-21kworker/0:310:22:190
726299257,3cyclictest13458-21kworker/0:108:58:050
726299252,11cyclictest8814-21kworker/0:308:32:530
726299252,10cyclictest19555-21kworker/0:311:59:090
7262992512,3cyclictest27929-21kworker/0:108:04:160
7262992512,3cyclictest17983-21kworker/0:007:53:240
7262992511,2cyclictest3641-21kworker/0:109:34:010
7262992511,2cyclictest23030-21kworker/0:210:38:190
7262992511,2cyclictest0-21swapper/011:35:110
726299251,11cyclictest30907-21kworker/0:009:29:210
7262992510,14cyclictest0-21swapper/009:25:000
102842251,20sleep39927-21/usr/sbin/munin11:20:163
727899245,11cyclictest41-21ksoftirqd/310:50:123
727899241,7cyclictest0-21swapper/311:10:023
727899241,10cyclictest19733-21grep08:50:293
727299241,9cyclictest0-21swapper/209:34:262
727299241,6cyclictest0-21swapper/209:10:312
726899245,3cyclictest0-21swapper/112:01:541
726899245,3cyclictest0-21swapper/110:23:041
726899241,10cyclictest0-21swapper/111:58:421
726899241,10cyclictest0-21swapper/109:56:181
726299249,3cyclictest12545-21/usr/sbin/munin07:20:170
726299248,3cyclictest30257-21kworker/0:311:10:210
726299247,3cyclictest9519-21kworker/0:011:17:550
726299247,3cyclictest30907-21kworker/0:009:19:010
726299246,2cyclictest17865-21kworker/0:310:26:130
726299242,9cyclictest18835-21kworker/0:307:35:420
726299242,12cyclictest15945-21kworker/0:010:10:170
726299241,10cyclictest19336-21kworker/0:011:48:230
726299241,10cyclictest19336-21kworker/0:011:48:230
7262992410,5cyclictest30257-21kworker/0:311:04:150
726299240,3cyclictest13458-21kworker/0:108:40:230
727899231,15cyclictest0-21swapper/311:35:143
727899231,11cyclictest0-21swapper/310:05:123
727299232,8cyclictest27631-21munin-run10:40:012
727299231,7cyclictest0-21swapper/210:11:142
727299231,6cyclictest0-21swapper/211:57:252
727299231,6cyclictest0-21swapper/211:39:362
727299231,11cyclictest0-21swapper/208:42:512
726899239,3cyclictest0-21swapper/109:45:441
726899238,2cyclictest0-21swapper/108:40:201
726899236,3cyclictest0-21swapper/107:13:111
726899232,3cyclictest0-21swapper/110:02:101
726899231,6cyclictest2493-21hddtemp_smartct09:30:131
726899231,13cyclictest21799-21sed08:55:281
726299239,3cyclictest9532-21kworker/0:111:27:590
726299239,3cyclictest8410-21kworker/0:210:01:210
726299239,3cyclictest28763-21kworker/0:112:22:390
726299238,3cyclictest28763-21kworker/0:112:10:310
726299232,11cyclictest26533-21kworker/0:012:05:410
726299231,9cyclictest5998-21kworker/0:108:22:530
726299231,10cyclictest10796-21sshd09:52:300
727899229,5cyclictest26751-21diskmemload09:54:333
727899227,6cyclictest0-21swapper/311:03:453
727899223,11cyclictest4372-21/usr/sbin/munin08:15:253
727899221,6cyclictest0-21swapper/312:12:583
727899221,5cyclictest0-21swapper/312:27:353
727899221,4cyclictest0-21swapper/312:05:573
727899221,18cyclictest11583-21diskstats09:55:173
727299229,4cyclictest0-21swapper/212:20:012
7272992217,2cyclictest10999-21idleruntime-cro09:55:012
726899226,3cyclictest23636-21ntp_states09:00:241
7268992215,4cyclictest25050-21sed07:45:271
726299228,3cyclictest17983-21kworker/0:007:48:160
726299227,3cyclictest15315-21kworker/0:110:05:210
726299227,3cyclictest13458-21kworker/0:108:45:490
726299226,3cyclictest14642-21kworker/0:111:40:570
726299225,2cyclictest9532-21kworker/0:111:20:190
726299223,2cyclictest17983-21kworker/0:007:31:560
726299222,11cyclictest17723-21kworker/0:308:50:210
726299222,11cyclictest14507-21kworker/0:107:27:020
726299221,8cyclictest26502-21kworker/0:309:09:490
293842221,17sleep229521-21users07:55:262
727899216,6cyclictest0-21swapper/309:36:243
727899216,6cyclictest0-21swapper/309:36:243
727899214,9cyclictest26751-21diskmemload11:47:253
727899214,9cyclictest26751-21diskmemload11:47:253
727899212,11cyclictest31560-21/usr/sbin/munin12:20:143
727899211,5cyclictest0-21swapper/311:26:233
727899211,4cyclictest0-21swapper/311:43:413
727899211,4cyclictest0-21swapper/310:22:143
727899211,10cyclictest26756-21munin-run09:10:013
727899211,10cyclictest23555-21cpu07:45:123
727299212,8cyclictest32641-21munin-run10:55:022
727299211,6cyclictest0-21swapper/210:24:452
7272992116,3cyclictest25870-21grep12:00:282
7272992115,3cyclictest17794-21sed11:40:152
7272992115,3cyclictest14078-21gltestperf10:00:162
727299211,3cyclictest0-21swapper/212:21:482
726899215,4cyclictest0-21swapper/111:16:021
726899214,3cyclictest10455-21ntp_states08:30:201
726899210,2cyclictest0-21swapper/109:28:181
7262992114,4cyclictest9159-21munin-run08:30:000
727899209,3cyclictest21342-21/usr/sbin/munin07:40:173
727899206,3cyclictest0-21swapper/311:06:333
727899204,2cyclictest15487-21kworker/u8:009:10:283
727899203,9cyclictest39-21rcuc/311:55:013
727899202,15cyclictest13578-21tr10:00:123
727899201,9cyclictest13722-21needreboot11:30:153
727899201,9cyclictest10420-21kworker/3:109:30:223
727899201,8cyclictest86550irq/126-enp1s0-11:56:063
727899201,4cyclictest0-21swapper/312:16:133
727899201,3cyclictest26835-21ps10:35:243
727899201,3cyclictest0-21swapper/310:13:223
727899201,11cyclictest848-21sssd_nss08:49:303
727899201,11cyclictest26751-21diskmemload12:32:183
727899201,11cyclictest13781-21/usr/sbin/munin08:40:233
727899201,11cyclictest11582-21/usr/sbin/munin08:35:143
727899201,10cyclictest29091-21gunzip10:40:293
727899201,10cyclictest0-21swapper/309:25:163
7278992010,3cyclictest967-21awk09:25:003
727299202,6cyclictest13555-21munin-run08:40:012
727299201,8cyclictest0-21swapper/212:36:132
727299201,3cyclictest14634-21snmpd12:33:442
7272992013,4cyclictest23237-21munin-run10:30:012
726899207,5cyclictest86850irq/129-enp1s0-07:55:141
726899205,4cyclictest18029-21sh07:30:281
726899204,3cyclictest0-21swapper/107:24:111
726899202,2cyclictest86850irq/129-enp1s0-08:15:041
726899201,9cyclictest0-21swapper/107:42:421
7268992016,3cyclictest0-21swapper/109:35:541
7268992016,3cyclictest0-21swapper/109:35:541
726899201,2cyclictest0-21swapper/108:27:211
726899201,1cyclictest0-21swapper/112:09:081
726899200,3cyclictest0-21swapper/110:27:001
726899200,11cyclictest0-21swapper/112:27:401
726299207,3cyclictest29705-21kworker/0:010:46:050
726299206,3cyclictest86750irq/128-enp1s0-07:43:400
726299201,4cyclictest24718-21smart_sda10:30:200
241952201,2sleep0810-21abrt-dump-journ09:00:430
76232190,5sleep2311ktimersoftd/209:40:262
727899198,3cyclictest9382-21/usr/sbin/munin08:30:113
727899198,2cyclictest39-21rcuc/312:35:273
727899197,6cyclictest0-21swapper/311:19:253
727899196,4cyclictest0-21swapper/309:40:293
727899195,4cyclictest0-21swapper/309:17:023
727899192,10cyclictest27935-21/usr/sbin/munin07:55:173
727899191,8cyclictest86550irq/126-enp1s0-10:30:543
727899191,7cyclictest0-21swapper/310:27:013
727899191,11cyclictest0-21swapper/310:19:203
727899191,11cyclictest0-21swapper/307:15:263
727899191,10cyclictest26751-21diskmemload12:00:093
727899191,10cyclictest22573-21/usr/sbin/munin09:00:143
727899191,10cyclictest0-21swapper/310:46:163
727899191,10cyclictest0-21swapper/309:46:293
727299199,2cyclictest29839-21awk10:45:012
727299194,5cyclictest26765-21perf09:10:012
727299194,2cyclictest86550irq/126-enp1s0-11:50:242
727299191,8cyclictest581-21systemd-journal07:45:012
727299191,7cyclictest11349-21sh08:35:012
727299191,6cyclictest0-21swapper/211:02:232
727299191,6cyclictest0-21swapper/209:16:032
727299191,4cyclictest15650-21kworker/u8:110:05:242
727299190,7cyclictest0-21swapper/207:20:202
726899191,3cyclictest0-21swapper/111:45:011
726899191,17cyclictest0-21swapper/107:15:371
726299192,2cyclictest86850irq/129-enp1s0-08:10:030
726299191,10cyclictest9248-21kworker/0:207:17:590
217962190,3sleep3401ktimersoftd/308:55:273
727899181,7cyclictest27134-21cat07:50:293
727899181,7cyclictest2109-21cat08:10:013
727899181,6cyclictest0-21swapper/307:20:113
727899181,4cyclictest14634-21snmpd07:12:543
727899181,10cyclictest14634-21snmpd08:23:073
727299185,2cyclictest24095-21kworker/2:109:48:412
727299184,3cyclictest0-21swapper/211:32:102
727299182,5cyclictest9467-21munin-node07:15:012
727299182,5cyclictest32115-21munin-run08:05:012
727299182,13cyclictest19938-21unixbench_multi10:15:272
727299181,7cyclictest0-21swapper/211:19:262
727299181,7cyclictest0-21swapper/209:21:262
727299181,5cyclictest0-21swapper/211:05:352
727299181,5cyclictest0-21swapper/209:26:332
7272991814,3cyclictest32-21ksoftirqd/209:40:002
7272991814,3cyclictest32-21ksoftirqd/209:40:002
7272991813,3cyclictest19765-21users08:50:302
7272991812,3cyclictest26993-21sendmail_mailtr07:50:282
726899181,16cyclictest0-21swapper/107:29:301
318362180,2sleep30-21swapper/308:01:373
727299178,2cyclictest86750irq/128-enp1s0-12:29:252
727299176,3cyclictest32504-21/usr/sbin/munin08:05:122
727299176,3cyclictest32-21ksoftirqd/207:19:242
727299175,3cyclictest86550irq/126-enp1s0-07:45:132
727299173,4cyclictest32-21ksoftirqd/211:47:472
727299173,4cyclictest32-21ksoftirqd/211:47:472
727299171,7cyclictest814-21sshd10:45:572
727299171,7cyclictest532-21cpuspeed_turbos10:55:122
727299171,6cyclictest768-21in:imjournal07:25:222
727299171,6cyclictest3298-21cut08:10:242
727299171,6cyclictest0-21swapper/212:05:572
727299171,6cyclictest0-21swapper/207:32:352
727299171,5cyclictest0-21swapper/208:55:182
727299171,3cyclictest0-21swapper/211:27:562
7272991712,3cyclictest23453-21/usr/sbin/munin10:30:162
7272991712,3cyclictest17186-21processes08:45:232
727299171,15cyclictest0-21swapper/211:13:442
7272991711,4cyclictest5838-21grep08:15:262
147422171,2sleep30-21swapper/307:25:083
7278991612,3cyclictest0-21swapper/308:26:533
7278991612,2cyclictest16953-21/usr/sbin/munin07:30:303
7278991610,3cyclictest2533-21entropy08:10:153
727299163,4cyclictest6566-21taskset08:20:102
727299161,5cyclictest0-21swapper/209:55:162
727299161,2cyclictest0-21swapper/208:29:172
7272991610,3cyclictest27560-21cut12:10:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional