You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 14:18
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Sat Jul 27, 2024 00:50:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
217122870,0sleep2311rcuc/220:45:112
2750428350,29sleep70-21swapper/719:09:087
2739528350,29sleep10-21swapper/119:07:351
2740928147,29sleep60-21swapper/619:07:476
2756627945,29sleep20-21swapper/219:09:592
2755627844,30sleep30-21swapper/319:09:533
2754827845,28sleep40-21swapper/419:09:464
2732527743,29sleep50-21swapper/519:06:345
2669027543,26sleep00-21swapper/019:05:250
287262500,0sleep00-21swapper/022:54:540
261592500,0sleep60-21swapper/620:50:196
261592500,0sleep60-21swapper/620:50:186
62712460,0sleep70-21swapper/722:28:127
90592440,0sleep10-21swapper/121:42:581
186582440,0sleep30-21swapper/323:51:433
28001991615,1cyclictest31554-21ssh21:22:394
28000991610,6cyclictest73550irq/125-eth023:35:473
27997991610,6cyclictest0-21swapper/023:26:020
27997991610,6cyclictest0-21swapper/022:20:150
28004991510,5cyclictest0-21swapper/700:30:047
28003991510,5cyclictest0-21swapper/600:13:466
28003991510,0cyclictest0-21swapper/622:44:486
28001991513,1cyclictest3809-21diskmemload21:17:004
2800199150,14cyclictest0-21swapper/420:45:104
28000991515,0cyclictest0-21swapper/323:27:083
28000991510,5cyclictest0-21swapper/322:48:543
28000991510,0cyclictest0-21swapper/323:21:053
28000991510,0cyclictest0-21swapper/300:26:573
27999991510,5cyclictest0-21swapper/200:34:002
27999991510,5cyclictest0-21swapper/200:29:312
27998991515,0cyclictest0-21swapper/121:54:591
27998991515,0cyclictest0-21swapper/121:10:491
27998991510,5cyclictest0-21swapper/100:17:251
27998991510,0cyclictest0-21swapper/123:58:481
27997991510,5cyclictest0-21swapper/023:13:050
27997991510,5cyclictest0-21swapper/021:37:080
27997991510,0cyclictest0-21swapper/022:19:250
27997991510,0cyclictest0-21swapper/022:06:030
27997991510,0cyclictest0-21swapper/000:29:490
85862140,0sleep50-21swapper/500:18:435
28004991414,0cyclictest0-21swapper/722:19:217
28004991414,0cyclictest0-21swapper/721:21:047
28004991414,0cyclictest0-21swapper/720:50:177
28004991414,0cyclictest0-21swapper/720:50:167
28004991412,1cyclictest18635-21nscd00:02:117
28004991410,4cyclictest0-21swapper/723:02:097
28003991414,0cyclictest0-21swapper/623:43:116
28003991413,1cyclictest9920-21ssh00:03:416
28003991410,0cyclictest0-21swapper/623:50:226
28003991410,0cyclictest0-21swapper/622:03:476
28003991410,0cyclictest0-21swapper/621:12:556
28002991414,0cyclictest0-21swapper/522:56:405
28002991414,0cyclictest0-21swapper/500:34:125
28002991412,1cyclictest3809-21diskmemload22:04:555
28002991412,1cyclictest3809-21diskmemload00:07:135
28002991410,4cyclictest0-21swapper/500:12:475
28002991410,0cyclictest0-21swapper/523:30:565
28002991410,0cyclictest0-21swapper/523:13:245
28001991413,1cyclictest22282-21ssh00:25:124
28000991414,0cyclictest0-21swapper/323:10:373
28000991414,0cyclictest0-21swapper/321:33:263
28000991414,0cyclictest0-21swapper/300:08:293
28000991412,1cyclictest73550irq/125-eth000:16:143
28000991411,2cyclictest682-21nscd23:06:563
27999991414,0cyclictest0-21swapper/223:22:182
27999991414,0cyclictest0-21swapper/222:20:032
27999991410,4cyclictest0-21swapper/222:26:512
27998991414,0cyclictest0-21swapper/123:49:571
27998991414,0cyclictest0-21swapper/123:20:051
27998991414,0cyclictest0-21swapper/122:55:381
27998991410,0cyclictest0-21swapper/123:53:291
27998991410,0cyclictest0-21swapper/123:41:041
27998991410,0cyclictest0-21swapper/122:34:091
27997991414,0cyclictest0-21swapper/023:43:220
27997991414,0cyclictest0-21swapper/023:16:320
27997991414,0cyclictest0-21swapper/023:00:580
27997991414,0cyclictest0-21swapper/021:42:520
27997991414,0cyclictest0-21swapper/000:08:490
162752140,0sleep10-21swapper/100:22:001
496221310,2sleep40-21swapper/419:23:384
2992121310,2sleep30-21swapper/322:24:163
28004991313,0cyclictest0-21swapper/723:37:517
28004991313,0cyclictest0-21swapper/723:11:457
28004991313,0cyclictest0-21swapper/721:45:507
28004991313,0cyclictest0-21swapper/721:17:007
28004991313,0cyclictest0-21swapper/700:22:077
28004991312,1cyclictest3809-21diskmemload23:30:157
28004991312,1cyclictest31001-21munin-node22:50:207
28004991312,1cyclictest26454-21kworker/u16:122:41:397
28004991312,1cyclictest16789-21ssh23:50:307
28004991312,1cyclictest12083-21snmpd21:30:357
28004991311,1cyclictest682-21nscd23:29:417
28004991311,1cyclictest682-21nscd23:16:457
28004991311,1cyclictest682-21nscd21:14:547
28004991311,1cyclictest3809-21diskmemload21:42:097
28004991311,1cyclictest30040-21grep19:10:227
28004991311,1cyclictest26937-21nscd22:31:037
28004991311,1cyclictest24053-21rm22:20:307
28004991311,1cyclictest18635-21nscd22:35:327
28004991311,1cyclictest18635-21nscd22:12:087
28004991310,3cyclictest0-21swapper/720:42:347
28004991310,1cyclictest0-21swapper/723:20:057
28004991310,0cyclictest0-21swapper/720:46:517
28003991313,0cyclictest0-21swapper/623:57:346
28003991313,0cyclictest0-21swapper/622:33:396
28003991313,0cyclictest0-21swapper/622:15:406
28003991313,0cyclictest0-21swapper/600:26:176
28003991312,1cyclictest682-21nscd21:25:276
28003991312,1cyclictest3809-21diskmemload00:09:526
28003991312,1cyclictest25635-21sh22:21:326
28003991312,1cyclictest2366-21sh22:10:206
28003991312,1cyclictest17883-21rm21:15:396
28003991312,1cyclictest17497-21ssh00:38:436
28003991312,1cyclictest16317-21ssh23:20:016
28003991312,1cyclictest10492-21sed19:30:226
28003991311,1cyclictest8376-21cut19:30:006
28003991311,1cyclictest682-21nscd23:47:296
28003991311,1cyclictest682-21nscd23:31:596
28003991311,1cyclictest682-21nscd00:33:126
28003991311,1cyclictest3809-21diskmemload23:10:046
28003991311,1cyclictest3809-21diskmemload21:51:056
28003991311,1cyclictest3809-21diskmemload21:20:336
28003991311,1cyclictest14893-21rm23:03:296
28002991313,0cyclictest0-21swapper/522:21:245
28002991312,1cyclictest3809-21diskmemload22:31:305
28002991312,1cyclictest0-21swapper/521:00:105
28002991311,1cyclictest9724-21rm21:12:105
28002991311,1cyclictest704-21ptp4l19:49:025
28002991311,1cyclictest682-21nscd22:07:525
28002991311,1cyclictest682-21nscd21:58:115
28002991311,1cyclictest3809-21diskmemload23:21:535
28002991311,1cyclictest1717-21ssh22:10:125
28002991311,1cyclictest13026-21missed_timers19:35:195
28002991310,3cyclictest24004-21kworker/u16:023:45:455
28002991310,3cyclictest0-21swapper/523:44:085
28002991310,3cyclictest0-21swapper/523:29:145
28002991310,3cyclictest0-21swapper/522:46:185
28002991310,3cyclictest0-21swapper/522:42:345
28002991310,3cyclictest0-21swapper/522:25:045
28002991310,3cyclictest0-21swapper/521:33:505
28002991310,3cyclictest0-21swapper/521:26:155
28002991310,3cyclictest0-21swapper/520:04:105
28002991310,3cyclictest0-21swapper/500:39:375
28002991310,3cyclictest0-21swapper/500:23:245
28002991310,0cyclictest0-21swapper/523:50:015
28002991310,0cyclictest0-21swapper/523:36:245
28002991310,0cyclictest0-21swapper/523:05:125
28002991310,0cyclictest0-21swapper/523:04:435
28002991310,0cyclictest0-21swapper/522:51:055
28002991310,0cyclictest0-21swapper/521:46:345
28002991310,0cyclictest0-21swapper/521:20:365
28002991310,0cyclictest0-21swapper/521:17:285
28002991310,0cyclictest0-21swapper/500:27:245
28002991310,0cyclictest0-21swapper/500:00:105
28001991312,1cyclictest481ktimersoftd/400:19:264
28001991311,1cyclictest704-21ptp4l00:20:564
28001991311,1cyclictest692-21nscd22:24:494
28001991311,1cyclictest481ktimersoftd/421:50:184
28001991311,1cyclictest3809-21diskmemload22:04:274
28001991311,1cyclictest31611-21nscd21:45:554
28000991313,0cyclictest0-21swapper/321:52:263
28000991313,0cyclictest0-21swapper/321:15:273
28000991312,1cyclictest73550irq/125-eth023:01:353
28000991312,1cyclictest73550irq/125-eth022:50:053
28000991312,1cyclictest73550irq/125-eth022:33:443
28000991312,1cyclictest73550irq/125-eth022:19:343
28000991312,1cyclictest73550irq/125-eth022:08:403
28000991312,1cyclictest73550irq/125-eth022:02:073
28000991312,1cyclictest73550irq/125-eth021:40:153
28000991312,1cyclictest73550irq/125-eth021:21:243
28000991312,1cyclictest73550irq/125-eth019:35:163
28000991312,1cyclictest73550irq/125-eth000:30:423
28000991312,1cyclictest73550irq/125-eth000:20:163
28000991312,1cyclictest20185-21wc22:35:143
28000991312,1cyclictest17771-21nscd21:12:233
28000991311,1cyclictest73550irq/125-eth023:55:253
28000991311,1cyclictest73550irq/125-eth023:30:023
28000991311,1cyclictest73550irq/125-eth022:25:273
28000991311,1cyclictest682-21nscd21:45:163
28000991310,3cyclictest0-21swapper/319:16:323
28000991310,0cyclictest121rcu_preempt23:41:573
27999991313,0cyclictest0-21swapper/223:57:342
27999991313,0cyclictest0-21swapper/223:15:182
27999991313,0cyclictest0-21swapper/200:35:482
27999991312,1cyclictest682-21nscd23:32:502
27999991312,1cyclictest32528-21/usr/sbin/munin20:10:182
27999991312,1cyclictest30620-21kworker/u16:300:19:162
27999991312,1cyclictest10840-1kworker/u17:222:07:382
27999991312,1cyclictest10630-21nscd20:35:012
27999991311,1cyclictest682-21nscd23:40:572
27999991311,1cyclictest682-21nscd23:05:012
27999991311,1cyclictest682-21nscd22:50:062
27999991311,1cyclictest682-21nscd22:40:292
27999991311,1cyclictest682-21nscd22:33:172
27999991311,1cyclictest682-21nscd21:21:032
27999991311,1cyclictest682-21nscd00:05:542
27999991311,1cyclictest6111-21tr21:10:152
27999991311,1cyclictest4856-21sh23:45:132
27999991311,1cyclictest25128-21ntp_states21:50:212
27999991311,1cyclictest22086-21taskset19:50:172
27998991313,0cyclictest0-21swapper/122:28:121
27998991313,0cyclictest0-21swapper/121:38:421
27998991312,1cyclictest692-21nscd22:37:131
27998991311,1cyclictest682-21nscd23:16:101
27998991311,1cyclictest682-21nscd21:51:141
27998991311,1cyclictest682-21nscd00:38:421
27998991311,1cyclictest682-21nscd00:25:171
27998991311,1cyclictest23078-21runrttasks19:53:471
27998991311,1cyclictest10630-21nscd23:03:131
27998991310,3cyclictest0-21swapper/121:05:541
27998991310,3cyclictest0-21swapper/120:58:311
27998991310,3cyclictest0-21swapper/119:28:111
27998991310,3cyclictest0-21swapper/119:20:091
27997991313,0cyclictest0-21swapper/023:30:180
27997991313,0cyclictest0-21swapper/021:26:280
27997991312,1cyclictest704-21ptp4l00:15:580
27997991312,1cyclictest30875-21rm23:57:570
27997991312,1cyclictest11043-21perf19:35:010
27997991311,1cyclictest704-21ptp4l22:35:180
27997991311,1cyclictest15479-21grep19:40:130
27997991310,1cyclictest0-21swapper/022:33:280
27997991310,0cyclictest0-21swapper/019:28:360
3000821210,0sleep30-21swapper/321:00:023
28004991211,1cyclictest9284-21cstates22:45:117
28004991211,1cyclictest8713-21cstates19:30:107
28004991211,1cyclictest704-21ptp4l20:05:037
28004991211,1cyclictest682-21nscd21:26:127
28004991211,1cyclictest682-21nscd00:10:067
28004991211,1cyclictest3809-21diskmemload21:55:047
28004991211,1cyclictest3579-21ssh23:45:057
28004991211,1cyclictest27995-21cyclictest20:55:217
28004991211,1cyclictest27995-21cyclictest20:30:337
28004991211,1cyclictest27995-21cyclictest20:20:357
28004991211,1cyclictest27995-21cyclictest20:11:037
28004991211,1cyclictest27995-21cyclictest20:00:377
28004991211,1cyclictest27995-21cyclictest19:37:487
28004991211,1cyclictest27995-21cyclictest19:15:017
28004991211,1cyclictest21633-21gltestperf19:50:137
28004991211,1cyclictest11965-21ssh22:00:087
28004991210,2cyclictest0-21swapper/721:50:517
28004991210,2cyclictest0-21swapper/720:35:147
28004991210,1cyclictest9748-21/usr/sbin/munin00:35:147
28004991210,1cyclictest704-21ptp4l19:22:297
28004991210,1cyclictest4201-21hwlatdetect20:15:147
28004991210,1cyclictest320-21systemd-journal21:00:027
28004991210,1cyclictest320-21systemd-journal19:40:027
28004991210,1cyclictest29267-21ssh22:55:027
28004991210,1cyclictest26469-21sed23:55:217
28004991210,1cyclictest26017-21netstat21:35:167
28004991210,1cyclictest24910-21missed_timers19:55:187
28004991210,1cyclictest22975-21grep00:25:157
28004991210,1cyclictest22149-21sh22:05:017
28004991210,1cyclictest1803-21sendmail-mta23:40:347
28004991210,1cyclictest17661-21diskmemload23:05:037
28004991210,1cyclictest17594-21apt-get19:45:107
28004991210,1cyclictest1624-21runrttasks21:05:107
28004991210,1cyclictest1624-21runrttasks00:15:267
28004991210,1cyclictest14151-21missed_timers00:05:177
28004991210,1cyclictest10087-21fschecks_count20:25:137
28004991210,0cyclictest0-21swapper/719:26:097
28003991211,1cyclictest704-21ptp4l23:05:126
28003991211,1cyclictest682-21nscd21:40:066
28003991211,1cyclictest6503-21ssh23:15:126
28003991211,1cyclictest5598-21sh21:56:196
28003991211,1cyclictest32639-21cpuspeed_turbos20:10:106
28003991211,1cyclictest305-21fschecks_time19:15:126
28003991211,1cyclictest27995-21cyclictest22:55:296
28003991211,1cyclictest27995-21cyclictest20:31:596
28003991211,1cyclictest27995-21cyclictest20:15:136
28003991211,1cyclictest27995-21cyclictest19:12:566
28003991211,1cyclictest21730-21cstates20:45:116
28003991211,1cyclictest21687-21ssh22:50:246
28003991211,1cyclictest18245-21runrttasks20:40:046
28003991211,1cyclictest1657-21lldpd21:08:336
28003991211,1cyclictest15189-21ssh23:35:016
28003991211,1cyclictest14613-21/usr/sbin/munin19:40:246
28003991211,1cyclictest14201-21runrttasks19:39:576
28003991211,1cyclictest13367-21runrttasks22:46:436
28003991211,1cyclictest0-21swapper/620:25:116
28003991211,0cyclictest3809-21diskmemload22:25:576
28003991210,2cyclictest201-21md0_raid121:35:136
28003991210,2cyclictest0-21swapper/620:55:176
28003991210,2cyclictest0-21swapper/619:55:226
28003991210,1cyclictest704-21ptp4l21:45:106
28003991210,1cyclictest6106-21/usr/sbin/munin20:20:116
28003991210,1cyclictest590-21ssh00:15:016
28003991210,1cyclictest28935-21chrt20:01:516
28003991210,1cyclictest2799-21expr19:20:126
28003991210,1cyclictest27012-21cpuspeed_turbos23:25:096
28003991210,1cyclictest22553-21timerandwakeup19:50:216
28003991210,1cyclictest22139-21ssh22:05:016
28003991210,1cyclictest19862-21ssh22:35:016
28003991210,1cyclictest16127-21sh21:30:376
28003991210,1cyclictest15456-21cpuspeed_turbos20:35:106
28003991210,1cyclictest11953-21/usr/sbin/munin00:20:106
28002991212,0cyclictest0-21swapper/522:35:015
28002991211,1cyclictest81350irq/129-enp2s0-20:16:065
28002991211,1cyclictest779-21gmain19:40:195
28002991211,1cyclictest704-21ptp4l20:59:275
28002991211,1cyclictest351-21cstates20:10:125
28002991211,1cyclictest27995-21cyclictest20:50:045
28002991211,1cyclictest27995-21cyclictest20:50:045
28002991211,1cyclictest27995-21cyclictest19:54:465
28002991211,1cyclictest27995-21cyclictest19:22:185
28002991211,1cyclictest27995-21cyclictest19:15:005
28002991211,1cyclictest24914-21cstates23:55:135
28002991211,1cyclictest15759-21cstates20:35:105
28002991211,1cyclictest0-21swapper/520:45:195
28002991210,2cyclictest0-21swapper/521:35:165
28002991210,2cyclictest0-21swapper/520:30:165
28002991210,2cyclictest0-21swapper/520:25:135
28002991210,2cyclictest0-21swapper/519:25:195
28002991210,1cyclictest7441-21latency20:20:165
28002991210,1cyclictest6200-21sh23:15:115
28002991210,1cyclictest3758-21runrttasks21:09:305
28002991210,1cyclictest30890-21grep20:05:165
28002991210,1cyclictest28934-21fschecks_time19:10:135
28002991210,1cyclictest23645-21cpuspeed_turbos19:55:125
28002991210,1cyclictest2294-21ssh21:40:055
28002991210,1cyclictest19352-21sort20:40:165
28002991210,1cyclictest10620-21ssh22:15:015
28002991210,1cyclictest10305-21switchtime19:30:225
28002991210,0cyclictest0-21swapper/521:50:375
28001991211,1cyclictest9120-21nscd23:56:504
28001991211,1cyclictest8123-21ssh23:46:344
28001991211,1cyclictest7889-21sh23:15:174
28001991211,1cyclictest704-21ptp4l23:36:294
28001991211,1cyclictest704-21ptp4l00:36:474
28001991211,1cyclictest5708-21rm00:00:574
28001991211,1cyclictest5466-21taskset21:56:154
28001991211,1cyclictest481ktimersoftd/422:58:144
28001991211,1cyclictest3809-21diskmemload23:33:084
28001991211,1cyclictest3809-21diskmemload21:41:334
28001991211,1cyclictest3483-1kworker/u17:223:25:114
28001991211,1cyclictest32370-21nscd23:01:264
28001991211,1cyclictest30718-1kworker/u17:000:30:364
28001991211,1cyclictest30620-21kworker/u16:300:12:014
28001991211,1cyclictest29348-21cut23:10:134
28001991211,1cyclictest27995-21cyclictest21:08:024
28001991211,1cyclictest25732-21/usr/sbin/munin23:40:114
28001991211,1cyclictest25409-21sh22:05:384
28001991211,1cyclictest24004-21kworker/u16:021:35:334
28001991211,1cyclictest23013-21taskset19:52:534
28001991211,1cyclictest20481-21nscd21:27:344
28001991211,1cyclictest20446-21sh23:05:244
28001991211,1cyclictest1856-21unixbench-2d22:25:264
28001991211,1cyclictest15097-21cut21:30:204
28001991211,1cyclictest13882-21sh22:15:414
28001991211,1cyclictest1385-21fschecks_time22:10:114
28001991211,1cyclictest13204-21sh22:30:584
28001991211,1cyclictest12121-21sed20:30:004
28001991211,1cyclictest10630-21nscd22:45:594
28001991210,2cyclictest0-21swapper/423:52:304
28001991210,2cyclictest0-21swapper/400:05:134
28001991210,1cyclictest5196-21fschecks_count21:10:124
28001991210,1cyclictest3809-21diskmemload22:50:084
28001991210,1cyclictest3712-21df20:15:134
28001991210,1cyclictest30609-21sh22:40:064
28001991210,1cyclictest26618-21ssh22:37:554
28001991210,1cyclictest26463-21latency_hist20:00:014
28001991210,1cyclictest2428-21latency_hist19:20:004
28000991211,1cyclictest73550irq/125-eth023:45:023
28000991211,1cyclictest73550irq/125-eth023:15:013
28000991211,1cyclictest73550irq/125-eth022:40:173
28000991211,1cyclictest73550irq/125-eth022:10:023
28000991211,1cyclictest73550irq/125-eth021:55:043
28000991211,1cyclictest73550irq/125-eth021:35:013
28000991211,1cyclictest73550irq/125-eth021:25:103
28000991211,1cyclictest73550irq/125-eth020:40:363
28000991211,1cyclictest73550irq/125-eth020:35:113
28000991211,1cyclictest73550irq/125-eth020:30:223
28000991211,1cyclictest73550irq/125-eth019:50:043
28000991211,1cyclictest73550irq/125-eth019:46:043
28000991211,1cyclictest73550irq/125-eth019:20:453
28000991211,1cyclictest73550irq/125-eth000:35:033
28000991211,1cyclictest73550irq/125-eth000:10:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional