You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 14:18
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Sat Jul 27, 2024 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1046221870,6sleep12775599cyclictest23:54:091
1046221870,6sleep12775599cyclictest23:54:091
505221730,4sleep42775999cyclictest22:25:5710
2487721650,31sleep00-21swapper/019:08:480
2723121320,4sleep112776899cyclictest23:56:443
2723121320,4sleep112776899cyclictest23:56:443
27051212492,27sleep50-21swapper/519:11:0211
27094212094,21sleep100-21swapper/1019:11:392
270572117104,8sleep90-21swapper/919:11:0615
26251211792,20sleep130-21swapper/1319:09:025
2857421140,1sleep140-21swapper/1423:38:366
2857421140,1sleep140-21swapper/1423:38:366
260021090,1sleep20-21swapper/200:19:038
27079210892,12sleep120-21swapper/1219:11:254
26962210790,8sleep80-21swapper/819:09:4514
3086521050,6sleep142777399cyclictest00:27:486
1315021040,2sleep91-21systemd23:59:0615
1315021040,2sleep91-21systemd23:59:0615
27091210391,7sleep70-21swapper/719:11:3613
1887421020,0sleep50-21swapper/500:15:5311
335521010,1sleep60-21swapper/621:26:3612
279142980,6sleep142777399cyclictest22:57:086
198982980,1sleep20-21swapper/222:51:358
83042960,1sleep10-21swapper/121:58:531
264642960,0sleep40-21swapper/421:57:0310
169172940,1sleep120-21swapper/1222:14:024
173012930,1sleep110-21swapper/1121:28:493
13552930,2sleep70-21swapper/721:39:5913
240902910,1sleep9891ktimersoftd/922:10:3115
2716528972,12sleep40-21swapper/419:12:3710
201402890,1sleep60-21swapper/600:04:2412
183512890,1sleep30-21swapper/321:28:579
94842880,1sleep60-21swapper/623:58:4712
94842880,1sleep60-21swapper/623:58:4712
2727528880,4sleep60-21swapper/619:13:2712
2716428875,7sleep30-21swapper/319:12:369
7422870,2sleep6740-21bash22:02:2612
271662870,1sleep100-21swapper/1023:19:322
2701128672,10sleep10-21swapper/119:10:261
144652860,1sleep60-21swapper/621:15:2212
144652860,1sleep60-21swapper/621:15:2212
126362860,1sleep60-21swapper/622:13:4012
123042860,0sleep20-21swapper/223:17:488
280062850,0sleep130-21swapper/1321:21:225
280062850,0sleep130-21swapper/1321:21:225
2707828575,6sleep110-21swapper/1119:11:243
2701228472,8sleep20-21swapper/219:10:278
218292840,1sleep20-21swapper/200:35:228
2711628375,4sleep140-21swapper/1419:11:596
226792820,1sleep30-21swapper/322:33:419
54612810,0sleep90-21swapper/921:26:5515
113372810,1sleep130-21swapper/1321:14:575
113372810,1sleep130-21swapper/1321:14:575
2699128072,4sleep150-21swapper/1519:10:087
148892750,0sleep100-21swapper/1023:04:012
115362580,1sleep150-21swapper/1523:58:567
115362580,1sleep150-21swapper/1523:58:567
299552520,1sleep40-21swapper/423:19:5810
305532500,1sleep8801rcuc/822:53:1714
2775599500,1cyclictest0-21swapper/122:55:271
2775599490,2cyclictest0-21swapper/123:32:551
2775599490,1cyclictest0-21swapper/122:19:591
2775599490,1cyclictest0-21swapper/100:15:231
2775599480,4cyclictest1-21systemd22:42:461
2775599470,2cyclictest0-21swapper/123:53:111
2775599470,2cyclictest0-21swapper/123:53:111
2775599470,2cyclictest0-21swapper/123:15:271
2775599470,1cyclictest0-21swapper/122:07:551
2775599470,1cyclictest0-21swapper/121:36:511
2775599470,1cyclictest0-21swapper/100:36:031
2775599461,35cyclictest31904-21gdbus21:51:141
2775599460,2cyclictest0-21swapper/123:42:551
2775599460,2cyclictest0-21swapper/123:42:551
2775599460,2cyclictest0-21swapper/123:09:351
2775599460,2cyclictest0-21swapper/122:15:551
2775599460,2cyclictest0-21swapper/121:16:311
2775599460,2cyclictest0-21swapper/121:16:311
2775599460,2cyclictest0-21swapper/100:05:231
2775599460,1cyclictest0-21swapper/123:24:391
2775599460,1cyclictest0-21swapper/123:00:071
2775599460,1cyclictest0-21swapper/122:28:591
2775599460,1cyclictest0-21swapper/121:32:231
2775599460,1cyclictest0-21swapper/100:40:391
2775599460,1cyclictest0-21swapper/100:09:591
2775599460,1cyclictest0-21swapper/100:03:021
2775599460,1cyclictest0-21swapper/100:03:021
27771994512,30cyclictest2019-21sshd00:33:565
2775699459,3cyclictest2019-21sshd21:19:508
2775699459,3cyclictest2019-21sshd21:19:508
2775599450,2cyclictest0-21swapper/100:28:071
2775599450,1cyclictest0-21swapper/123:43:431
2775599450,1cyclictest0-21swapper/123:43:431
2775599450,1cyclictest0-21swapper/123:19:551
2775599450,1cyclictest0-21swapper/122:49:231
2775599450,1cyclictest0-21swapper/122:46:441
2775599450,1cyclictest0-21swapper/122:09:471
2775599450,1cyclictest0-21swapper/121:25:271
2775599450,1cyclictest0-21swapper/120:19:471
2775599450,1cyclictest0-21swapper/119:22:191
2775599450,1cyclictest0-21swapper/119:14:031
2775599450,1cyclictest0-21swapper/100:31:151
2775599450,1cyclictest0-21swapper/100:21:071
62702440,1sleep130-21swapper/1323:16:555
2775599440,2cyclictest1014-21systemd-logind21:56:591
2775599440,2cyclictest0-21swapper/121:45:311
2775599440,1cyclictest0-21swapper/123:34:351
2775599440,1cyclictest0-21swapper/121:21:431
2775599440,1cyclictest0-21swapper/121:21:431
2775599440,1cyclictest0-21swapper/120:52:471
2775599440,1cyclictest0-21swapper/120:46:591
2777199430,40cyclictest0-21swapper/1300:18:535
2777199430,40cyclictest0-21swapper/1300:06:575
27756994311,29cyclictest1005-21NetworkManager21:18:258
27756994311,29cyclictest1005-21NetworkManager21:18:258
2775599431,40cyclictest5040-21nfsd19:40:211
2775599430,2cyclictest0-21swapper/120:43:151
2775599430,2cyclictest0-21swapper/120:33:351
2775599430,1cyclictest0-21swapper/123:03:591
2775599430,1cyclictest0-21swapper/122:34:311
2775599430,1cyclictest0-21swapper/122:24:071
2775599430,1cyclictest0-21swapper/119:24:351
2777199426,3cyclictest145-21kauditd00:30:295
2777199420,3cyclictest31904-21gdbus22:48:355
2775699427,4cyclictest1076-21gdbus22:15:458
27756994212,27cyclictest25511-21systemd-cgroups00:10:078
2775599420,40cyclictest545-21cp21:39:531
2775599420,1cyclictest0-21swapper/121:09:111
2775599420,1cyclictest0-21swapper/121:09:111
2775599420,1cyclictest0-21swapper/121:01:471
2775599420,1cyclictest0-21swapper/120:55:511
2775599420,1cyclictest0-21swapper/120:17:431
2775599420,1cyclictest0-21swapper/120:03:511
2775599420,1cyclictest0-21swapper/119:58:351
2775599420,1cyclictest0-21swapper/119:50:351
2777199419,4cyclictest998-21dbus-daemon22:17:055
2775699416,3cyclictest998-21dbus-daemon21:25:448
2775699414,34cyclictest11088-21kworker/2:222:09:018
2775699411,3cyclictest998-21dbus-daemon21:34:588
27756994112,26cyclictest22035-21sshd21:42:558
2775599410,1cyclictest0-21swapper/120:24:431
2775599410,1cyclictest0-21swapper/120:08:591
2777199409,3cyclictest998-21dbus-daemon22:31:085
2777199405,4cyclictest998-21dbus-daemon23:46:435
2777199405,4cyclictest998-21dbus-daemon23:46:435
2777199401,36cyclictest26114-21uname22:56:545
27771994012,26cyclictest0-21swapper/1300:17:185
27771994010,27cyclictest26790-21sshd23:28:565
27771994010,27cyclictest18629-21sshd23:04:315
2777199400,38cyclictest10402-21systemd-cgroups23:54:095
2777199400,38cyclictest10402-21systemd-cgroups23:54:095
2775699407,3cyclictest943-21run-parts21:31:058
2775699404,3cyclictest0-21swapper/221:52:438
2775599400,1cyclictest0-21swapper/121:03:551
2775599400,1cyclictest0-21swapper/120:29:591
2775599400,1cyclictest0-21swapper/119:43:551
321322390,1sleep30-21swapper/321:53:349
2777199399,3cyclictest1014-21systemd-logind22:08:015
2777199396,3cyclictest4274-21sshd22:21:055
2777199395,7cyclictest0-21swapper/1321:43:275
2777199395,4cyclictest24313-21run-parts00:40:095
2777199395,4cyclictest22241-21run-parts22:33:355
27771993911,26cyclictest5157-21sshd21:36:155
2777199390,10cyclictest8104-21sshd21:54:255
2775699399,3cyclictest30970-21sshd00:32:178
2775699398,3cyclictest17494-2110-uname23:59:418
2775699398,3cyclictest17494-2110-uname23:59:418
2775699394,7cyclictest0-21swapper/221:56:018
27756993911,24cyclictest17314-21sh23:32:358
2775699390,11cyclictest3054-21sshd23:02:408
2775699390,10cyclictest0-21swapper/222:47:398
2775599390,1cyclictest0-21swapper/119:57:591
2775599390,1cyclictest0-21swapper/119:33:591
2777199388,3cyclictest27620-2110-uname22:01:365
2777199387,3cyclictest21072-21sshd23:18:495
2777199386,3cyclictest1-21systemd21:52:195
2777199385,3cyclictest1005-21NetworkManager21:24:045
2777199384,4cyclictest0-21swapper/1300:27:515
2777199383,5cyclictest0-21swapper/1323:34:095
27771993814,22cyclictest0-21swapper/1322:59:125
27771993811,24cyclictest20719-21bash00:00:085
27771993811,24cyclictest20719-21bash00:00:085
2777199380,2cyclictest0-21swapper/1323:24:105
2777199380,10cyclictest4098-21sshd22:44:045
2775699387,3cyclictest998-21dbus-daemon22:01:108
2775699387,3cyclictest687-21sshd22:34:478
2775699387,3cyclictest28559-21sshd21:48:218
2775699380,11cyclictest0-21swapper/223:22:438
2775599380,1cyclictest0-21swapper/119:28:351
2777199378,2cyclictest0-21swapper/1322:27:525
2777199377,4cyclictest30783-21sshd22:38:565
2777199376,3cyclictest1005-21NetworkManager23:41:385
2777199376,3cyclictest1005-21NetworkManager23:41:385
2777199375,4cyclictest31904-21gdbus23:11:585
2777099378,22cyclictest0-21swapper/1221:58:314
2775699379,4cyclictest11525-21run-parts22:40:368
2775699379,3cyclictest20473-21sshd23:55:398
2775699379,3cyclictest20473-21sshd23:55:398
2775699379,3cyclictest12985-21sshd23:36:208
2775699378,3cyclictest18134-21sshd22:04:378
2775699378,3cyclictest1-21systemd00:43:028
2775699370,13cyclictest5039-21nfsd22:27:158
2777199365,4cyclictest28536-21gdbus00:12:375
2777199363,4cyclictest998-21dbus-daemon23:51:355
2777199363,4cyclictest998-21dbus-daemon23:51:355
2777199360,34cyclictest0-21swapper/1322:10:595
2777099362,27cyclictest12965-21sshd21:23:434
2775699369,3cyclictest20665-21sshd23:51:188
2775699369,3cyclictest20665-21sshd23:51:188
2775699367,3cyclictest23986-2110-uname23:42:158
2775699367,3cyclictest23986-2110-uname23:42:158
8702350,1sleep80-21swapper/823:25:0814
2777099351,27cyclictest0-21swapper/1221:19:104
2777099351,27cyclictest0-21swapper/1221:19:104
2777099351,18cyclictest0-21swapper/1221:52:024
2775699358,4cyclictest1-21systemd00:04:138
2775699357,3cyclictest21177-21sh23:23:368
53892340,1sleep00-21swapper/023:03:040
2777199346,3cyclictest13167-21sshd21:45:545
2777199344,4cyclictest1-21systemd21:29:375
2777099344,24cyclictest0-21swapper/1222:35:234
2777099343,24cyclictest17015-21systemd-cgroups22:32:514
27768993416,7cyclictest0-21swapper/1122:00:123
2775699348,3cyclictest7083-21sh22:31:158
2775699343,2cyclictest0-21swapper/200:16:418
2775699340,12cyclictest0-21swapper/222:54:528
220012340,29sleep5998-21dbus-daemon23:05:0211
172302340,1sleep317223-21sshd00:08:549
16192340,1sleep110-21swapper/1121:18:083
16192340,1sleep110-21swapper/1121:18:083
142202340,0sleep120-21swapper/1200:14:404
4102330,1sleep60-21swapper/600:18:5112
316672330,1sleep70-21swapper/723:02:0613
309592330,1sleep50-21swapper/521:21:4611
309592330,1sleep50-21swapper/521:21:4611
2777199335,3cyclictest10302-21cpuspeed_turbos20:48:455
2777099335,21cyclictest0-21swapper/1200:18:534
2777099334,22cyclictest14309-21sshd22:45:354
2777099332,25cyclictest998-21dbus-daemon23:47:284
2777099332,25cyclictest998-21dbus-daemon23:47:284
2777099332,24cyclictest5506-21sshd21:44:464
2777099331,25cyclictest0-21swapper/1200:33:344
2777099330,27cyclictest1007-21ModemManager22:57:364
2777099330,25cyclictest0-21swapper/1223:00:144
2777099330,24cyclictest0-21swapper/1222:21:444
27766993314,9cyclictest0-21swapper/921:39:2715
2775699336,3cyclictest24072-21sshd22:19:148
2775699330,12cyclictest8152-21sshd00:24:038
2775699330,11cyclictest2898-21diskstats20:38:468
2777199329,3cyclictest25803-21latency_hist20:28:345
2777199321,4cyclictest0-21swapper/1320:43:465
2777099325,20cyclictest12688-21sshd23:54:284
2777099325,20cyclictest12688-21sshd23:54:284
2777099322,23cyclictest0-21swapper/1222:08:474
2777099322,22cyclictest998-21dbus-daemon23:52:064
2777099322,22cyclictest998-21dbus-daemon23:52:064
2777099322,22cyclictest6690-2110-uname23:39:444
2777099322,22cyclictest6690-2110-uname23:39:444
2777099322,22cyclictest0-21swapper/1222:39:194
2777099322,19cyclictest1014-21systemd-logind22:27:004
27770993213,13cyclictest38750md124_raid123:15:014
2777099321,29cyclictest432-21sshd00:01:544
2777099321,29cyclictest432-21sshd00:01:544
27770993210,11cyclictest0-21swapper/1222:49:154
2777099320,24cyclictest1029-21gdbus22:05:064
2775699328,3cyclictest4174-21latency_hist20:03:348
2775699327,3cyclictest15736-21apt-key19:38:338
2775699326,3cyclictest15721-21cp23:08:588
225752320,2sleep341-21ksoftirqd/300:31:069
144412320,1sleep120-21swapper/1223:08:524
89622310,1sleep60-21swapper/623:31:1512
42712310,26sleep30-21swapper/322:44:089
2777199316,3cyclictest29024-21cpuspeed_turbos19:53:445
2777199316,3cyclictest2079-21latency19:58:555
2777199316,3cyclictest19452-21needreboot20:18:555
2777199315,3cyclictest3501-21latency20:38:535
2777199310,28cyclictest0-21swapper/1319:14:045
2777099317,18cyclictest0-21swapper/1221:37:244
2777099315,20cyclictest998-21dbus-daemon23:33:544
2777099314,19cyclictest16445-21env21:42:084
2777099313,22cyclictest0-21swapper/1223:20:074
2777099312,23cyclictest31902-21packagekitd00:38:484
2777099312,22cyclictest998-21dbus-daemon00:11:574
2777099312,21cyclictest25002-21sshd00:26:524
2777099311,23cyclictest1-21systemd21:33:294
2777099311,23cyclictest1014-21systemd-logind23:28:464
2777099311,22cyclictest27928-21gdbus23:26:114
2777099311,22cyclictest1-21systemd22:01:274
2775699319,4cyclictest27603-21sh23:05:558
185062310,27sleep00-21swapper/022:33:070
109332310,0sleep100-21swapper/1022:13:282
82122300,0sleep40-21swapper/400:28:5910
2777199307,3cyclictest8316-21turbostat.cron20:13:285
2777099303,16cyclictest4402-21sshd21:14:044
2777099303,16cyclictest4402-21sshd21:14:044
27770993012,11cyclictest1014-21systemd-logind23:04:174
2775699309,3cyclictest29922-21latency_hist20:33:348
29562290,3sleep143030-21systemd-cgroups23:12:086
2777099299,12cyclictest0-21swapper/1200:04:164
2777099293,18cyclictest1014-21systemd-logind00:33:524
2776699297,9cyclictest25111-21systemd-cgroups21:29:4415
2775699296,2cyclictest15239-21fschecks_count20:53:488
2775699292,5cyclictest0-21swapper/220:48:448
265062290,1sleep30-21swapper/300:40:299
215052290,3sleep021627-21run-parts23:23:400
186622290,25sleep140-21swapper/1423:46:066
186622290,25sleep140-21swapper/1423:46:066
31372280,1sleep80-21swapper/821:58:2914
2776899289,13cyclictest106-21ksoftirqd/1122:21:433
2776699283,5cyclictest90-21ksoftirqd/923:37:5415
2776699280,26cyclictest16465-21sshd21:33:3715
2775699287,2cyclictest2006-21runrttasks20:09:248
266152280,2sleep140-21swapper/1423:52:156
266152280,2sleep140-21swapper/1423:52:156
168012280,24sleep60-21swapper/622:27:5212
36432270,2sleep50-21swapper/522:25:4311
304672270,2sleep40-21swapper/400:23:0710
282702270,1sleep80-21swapper/800:05:4714
2777199277,3cyclictest27549-21kernelversion20:28:525
2776699275,16cyclictest90-21ksoftirqd/921:16:5015
2776699275,16cyclictest90-21ksoftirqd/921:16:5015
27759992710,6cyclictest141rcu_sched23:25:3710
2775999270,5cyclictest22295-21sshd23:46:4010
2775999270,5cyclictest22295-21sshd23:46:4010
2775699275,3cyclictest878-21cron23:48:338
2775699275,3cyclictest878-21cron23:48:338
225232270,2sleep4471rcuc/422:37:5910
21512270,2sleep110-21swapper/1121:26:263
168122270,25sleep140-21swapper/1420:18:026
163172270,0sleep30-21swapper/300:34:359
82752260,2sleep60-21swapper/621:49:5212
2777199265,3cyclictest2377-21switchtime19:19:025
2777199265,3cyclictest13953-21cpuspeed_turbos20:13:455
2777199261,3cyclictest0-21swapper/1319:38:335
2777099268,11cyclictest0-21swapper/1219:14:054
2777099266,10cyclictest0-21swapper/1220:30:134
2777099263,21cyclictest0-21swapper/1220:14:344
2777099262,22cyclictest0-21swapper/1220:53:584
27768992621,3cyclictest12700-21idleruntime-cro00:08:333
2776699267,11cyclictest90-21ksoftirqd/922:53:3615
2776699263,17cyclictest90-21ksoftirqd/923:03:3315
27766992623,2cyclictest881rcuc/922:23:2515
27766992619,5cyclictest891ktimersoftd/900:20:5715
27766992615,6cyclictest131rcu_preempt22:51:3215
2776599260,2cyclictest0-21swapper/823:13:5714
2775999269,8cyclictest141rcu_sched00:00:2910
2775999269,8cyclictest141rcu_sched00:00:2910
27272260,0sleep110-21swapper/1123:34:483
237052260,1sleep80-21swapper/823:37:5814
231592260,22sleep60-21swapper/621:20:4412
231592260,22sleep60-21swapper/621:20:4412
230202260,2sleep00-21swapper/022:52:060
57962250,1sleep150-21swapper/1521:14:117
57962250,1sleep150-21swapper/1521:14:117
54602250,2sleep140-21swapper/1422:21:196
49682250,2sleep100-21swapper/1000:42:122
2777199255,3cyclictest7492-21latency_hist19:28:335
2777199255,3cyclictest30183-21cpu20:33:455
27768992521,2cyclictest92850irq/36-eth0-TxR22:06:163
2776699257,10cyclictest90-21ksoftirqd/921:58:5515
2776699256,9cyclictest90-21ksoftirqd/923:16:1915
2776699256,11cyclictest90-21ksoftirqd/923:09:4815
2776699256,10cyclictest90-21ksoftirqd/922:18:1915
27766992519,5cyclictest90-21ksoftirqd/921:45:2615
2775999256,8cyclictest22492-21kworker/4:400:27:5410
27759992515,6cyclictest131rcu_preempt23:51:5810
27759992515,6cyclictest131rcu_preempt23:51:5810
27759992511,7cyclictest49-21ksoftirqd/422:59:1010
2775899250,22cyclictest0-21swapper/323:53:509
2775899250,22cyclictest0-21swapper/323:53:509
2775699257,3cyclictest9925-21latency_hist20:48:348
264632250,21sleep100-21swapper/1021:43:322
233812250,21sleep150-21swapper/1523:51:447
233812250,21sleep150-21swapper/1523:51:447
200582250,22sleep30-21swapper/323:04:449
171922250,1sleep110-21swapper/1120:54:043
162092250,0sleep00-21swapper/000:39:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional