You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-24 - 09:42
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sun Nov 24, 2024 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2504424334,0sleep30-21swapper/318:36:533
3178923829,0sleep10-21swapper/118:39:471
3172423825,0sleep00-21swapper/018:39:020
3157423425,0sleep20-21swapper/218:37:202
329970,0watchdog/314176-21aten_r1power_en22:16:583
29925070,0irq/32-eth0-rx-20839-21ssh21:02:343
29925070,0irq/32-eth0-rx-0-21swapper/321:35:493
29925070,0irq/32-eth0-rx-0-21swapper/320:57:043
339960,0migration/323618-21sh22:39:493
320329960,0cyclictest0-21swapper/120:47:021
320319966,0cyclictest2546-21nvme23:27:050
329960,0watchdog/331479-21python19:22:113
329960,0watchdog/311438-21fschecks_count19:11:593
329960,0watchdog/30-21swapper/323:57:303
329960,0watchdog/30-21swapper/323:55:103
329960,0watchdog/30-21swapper/323:47:063
329960,0watchdog/30-21swapper/323:28:383
329960,0watchdog/30-21swapper/323:17:383
329960,0watchdog/30-21swapper/323:15:263
329960,0watchdog/30-21swapper/323:03:063
329960,0watchdog/30-21swapper/322:56:583
329960,0watchdog/30-21swapper/322:56:383
329960,0watchdog/30-21swapper/322:47:063
329960,0watchdog/30-21swapper/322:29:183
329960,0watchdog/30-21swapper/322:09:543
329960,0watchdog/30-21swapper/322:03:303
329960,0watchdog/30-21swapper/321:44:233
329960,0watchdog/30-21swapper/321:23:513
329960,0watchdog/30-21swapper/321:19:153
329960,0watchdog/30-21swapper/321:11:553
329960,0watchdog/30-21swapper/319:41:593
329960,0watchdog/30-21swapper/319:37:033
329960,0watchdog/30-21swapper/318:46:353
329960,0watchdog/30-21swapper/300:07:543
329960,0watchdog/30-21swapper/300:06:423
29925060,0irq/32-eth0-rx-7553-21sh22:31:573
29925060,0irq/32-eth0-rx-4399-21smartctl21:52:023
29925060,0irq/32-eth0-rx-0-21swapper/323:22:563
29925060,0irq/32-eth0-rx-0-21swapper/323:09:023
29925060,0irq/32-eth0-rx-0-21swapper/322:42:063
29925060,0irq/32-eth0-rx-0-21swapper/322:24:373
29925060,0irq/32-eth0-rx-0-21swapper/321:57:133
29925060,0irq/32-eth0-rx-0-21swapper/321:50:143
29925060,0irq/32-eth0-rx-0-21swapper/321:27:383
29925060,0irq/32-eth0-rx-0-21swapper/321:10:383
29925060,0irq/32-eth0-rx-0-21swapper/320:44:373
29925060,0irq/32-eth0-rx-0-21swapper/318:47:043
149960,0migration/08004-21taskset20:17:000
149960,0migration/031063-21latency19:22:060
149960,0migration/014286-21taskset22:32:380
149960,0migration/013821-21aten_r1power_vo23:31:590
8315050,0irq/26-0000:00:8120-21hddtemp_smartct23:47:022
8315050,0irq/26-0000:00:32486-21crond00:01:462
339950,0migration/39912-21taskset21:36:573
339950,0migration/324090-21kthreadcore20:27:053
339950,0migration/316536-21kthreadcore18:52:053
339950,0migration/310225-21kthreadcore22:12:053
320339955,0cyclictest0-21swapper/219:22:142
320339954,0cyclictest3599-21lldpd21:51:562
320339954,0cyclictest23871-21snmpd19:59:162
320339954,0cyclictest15849-21awk20:22:022
320339950,0cyclictest0-21swapper/222:43:152
320339950,0cyclictest0-21swapper/220:27:112
320329950,0cyclictest0-21swapper/123:39:311
320329950,0cyclictest0-21swapper/121:18:061
320319955,0cyclictest0-21swapper/021:39:370
320319955,0cyclictest0-21swapper/021:30:180
320319950,0cyclictest0-21swapper/021:13:070
329950,0watchdog/331664-21python20:07:113
329950,0watchdog/30-21swapper/323:36:463
329950,0watchdog/30-21swapper/323:32:143
329950,0watchdog/30-21swapper/320:36:513
329950,0watchdog/30-21swapper/320:36:513
329950,0watchdog/30-21swapper/320:31:473
329950,0watchdog/30-21swapper/320:22:473
329950,0watchdog/30-21swapper/320:16:513
329950,0watchdog/30-21swapper/320:11:513
329950,0watchdog/30-21swapper/320:01:473
329950,0watchdog/30-21swapper/319:56:553
329950,0watchdog/30-21swapper/319:53:153
329950,0watchdog/30-21swapper/319:48:233
329950,0watchdog/30-21swapper/319:32:393
329950,0watchdog/30-21swapper/319:26:513
329950,0watchdog/30-21swapper/319:16:553
329950,0watchdog/30-21swapper/319:06:553
329950,0watchdog/30-21swapper/319:06:313
329950,0watchdog/30-21swapper/318:56:513
29935050,0irq/33-eth0-tx-8060-21cut22:12:042
29935050,0irq/33-eth0-tx-7373-21munin-node19:32:020
29935050,0irq/33-eth0-tx-23984-21iostat_ios21:07:032
29935050,0irq/33-eth0-tx-19452-21grep22:17:062
29935050,0irq/33-eth0-tx-18263-21munin-node20:02:020
29935050,0irq/33-eth0-tx-18093-21munin-node22:57:022
29935050,0irq/33-eth0-tx-18068-21munin-node19:17:010
29935050,0irq/33-eth0-tx-0-21swapper/223:34:342
29935050,0irq/33-eth0-tx-0-21swapper/223:12:152
29935050,0irq/33-eth0-tx-0-21swapper/221:38:412
29935050,0irq/33-eth0-tx-0-21swapper/020:42:300
29925050,0irq/32-eth0-rx-0-21swapper/323:41:463
29925050,0irq/32-eth0-rx-0-21swapper/320:51:533
29925050,0irq/32-eth0-rx-0-21swapper/320:47:003
259950,0migration/2333-21kthreadcore22:07:072
259950,0migration/222559-21latency_hist21:06:462
259950,0migration/22186-21missed_timers20:52:062
259950,0migration/213325-21latency22:32:052
249950,0watchdog/25443-21cut00:02:102
179950,0migration/18177-21idleruntime23:47:031
179950,0migration/17818-21kthreadcore19:52:051
179950,0migration/15920-21ntp_kernel_pll_18:42:071
179950,0migration/1478-21sh21:30:331
179950,0migration/131505-21kthreadcore23:22:081
179950,0migration/124865-21cpuspeed_turbos22:42:001
179950,0migration/11837-21taskset21:48:231
179950,0migration/110060-21kthreadcore22:32:041
169950,0watchdog/132604-21kthreadcore22:07:061
169950,0watchdog/128895-21tr23:57:061
169950,0watchdog/126851-21ssh23:17:141
169950,0watchdog/115144-21cut19:36:591
149950,0migration/0927-21fschecks_count00:02:040
149950,0migration/06755-21timerwakeupswit20:12:130
149950,0migration/06374-21kthreadcore21:52:060
149950,0migration/04098-21sh23:24:550
149950,0migration/03615-21kthreadcore19:27:070
149950,0migration/031772-21gltestperf23:42:020
149950,0migration/03074-21taskset19:04:090
149950,0migration/030392-21if_eth1.222:07:040
149950,0migration/029787-21kthreadcore21:47:050
149950,0migration/029407-21kthreadcore20:07:060
149950,0migration/028768-21aten_r1power_vo20:51:590
149950,0migration/027567-21kthreadcore20:27:060
149950,0migration/025523-21sh22:59:190
149950,0migration/025244-21munin-node23:57:020
149950,0migration/025012-21timerandwakeup21:42:140
149950,0migration/022859-21kthreadcore20:47:050
149950,0migration/022527-21cpuspeed_turbos23:36:590
149950,0migration/021790-21kthreadcore23:17:050
149950,0migration/021686-21idleruntime22:02:050
149950,0migration/021510-21taskset19:41:390
149950,0migration/013432-21kthreadcore00:07:050
149950,0migration/013001-21kthreadcore19:57:050
149950,0migration/010589-21sh23:09:500
149950,0migration/010192-21kthreadcore23:47:060
320339943,0cyclictest8273-21awk18:47:032
320339943,0cyclictest3599-21lldpd21:49:262
320339943,0cyclictest3599-21lldpd21:21:552
320339943,0cyclictest3599-21lldpd20:18:252
320339943,0cyclictest3599-21lldpd20:01:542
320339943,0cyclictest3599-21lldpd19:42:542
320339943,0cyclictest3599-21lldpd19:18:242
320339943,0cyclictest3599-21lldpd19:00:242
320339943,0cyclictest3262-21irqbalance23:27:002
320339943,0cyclictest30229-21awk20:32:032
320339943,0cyclictest23871-21snmpd20:14:162
320339943,0cyclictest23871-21snmpd20:10:162
320339943,0cyclictest23871-21snmpd18:55:152
320339943,0cyclictest20912-21awk20:47:022
320339943,0cyclictest15621-21awk19:37:012
320329944,0cyclictest0-21swapper/121:00:071
320329944,0cyclictest0-21swapper/119:22:321
320329943,0cyclictest4127-21awk23:07:031
320329943,0cyclictest23871-21snmpd22:54:471
320329943,0cyclictest21714-21awk22:02:051
320329943,0cyclictest19614-21awk21:42:041
320329943,0cyclictest19142-21awk22:57:021
320329943,0cyclictest15009-21awk21:02:041
320329943,0cyclictest1355-21awk20:12:031
320329943,0cyclictest10969-21awk21:37:021
320329940,0cyclictest0-21swapper/123:02:471
320319943,0cyclictest905-21awk18:42:010
320319943,0cyclictest2414-21awk21:32:040
320319943,0cyclictest23871-21snmpd20:38:010
320319943,0cyclictest23871-21snmpd20:38:010
320319943,0cyclictest23871-21snmpd20:35:160
320319943,0cyclictest22842-21awk19:42:020
320319943,0cyclictest1816-21awk22:47:020
29935040,0irq/33-eth0-tx-8907-21ssh00:06:572
29935040,0irq/33-eth0-tx-5421-21ssh20:56:582
29935040,0irq/33-eth0-tx-27669-21hwlatdetect23:02:052
29935040,0irq/33-eth0-tx-25084-21munin-node21:27:012
29935040,0irq/33-eth0-tx-23871-21snmpd23:21:562
29935040,0irq/33-eth0-tx-21598-21telnet18:56:580
29935040,0irq/33-eth0-tx-20763-21aten_r1power_po22:02:012
29935040,0irq/33-eth0-tx-16830-21fschecks_time22:37:012
29935040,0irq/33-eth0-tx-11105-21diskmemload21:12:032
29935040,0irq/33-eth0-tx-0-21swapper/223:52:152
29935040,0irq/33-eth0-tx-0-21swapper/223:41:592
29935040,0irq/33-eth0-tx-0-21swapper/223:36:482
29935040,0irq/33-eth0-tx-0-21swapper/223:07:122
29935040,0irq/33-eth0-tx-0-21swapper/222:52:042
29935040,0irq/33-eth0-tx-0-21swapper/222:47:512
29935040,0irq/33-eth0-tx-0-21swapper/221:57:382
29935040,0irq/33-eth0-tx-0-21swapper/221:41:552
29935040,0irq/33-eth0-tx-0-21swapper/221:31:592
29935040,0irq/33-eth0-tx-0-21swapper/019:52:090
29935040,0irq/33-eth0-tx-0-21swapper/019:09:540
29935040,0irq/33-eth0-tx-0-21swapper/018:47:130
259940,0migration/27661-21cpuspeed_turbos19:31:592
259940,0migration/27201-21kthreadcore20:37:072
259940,0migration/27201-21kthreadcore20:37:072
259940,0migration/2422-21kthreadcore22:27:052
259940,0migration/23811-21kthreadcore18:42:042
259940,0migration/225779-21kthreadcore23:17:062
259940,0migration/215798-21kthreadcore20:42:052
259940,0migration/210047-21kthreadcore21:17:052
249940,0watchdog/23734-21aten_r1power_vo19:06:592
249940,0watchdog/228959-21munin-node19:47:032
249940,0watchdog/222618-21ssh22:21:502
249940,0watchdog/20-21swapper/219:11:592
19140,0ktimersoftd/12123-21jbd2/md0-821:21:461
179940,0migration/19694-21kthreadcore20:17:051
179940,0migration/19654-21kthreadcore18:47:041
179940,0migration/18911-21irqstats19:32:031
179940,0migration/16250-21kthreadcore21:32:071
179940,0migration/16044-21hddtemp_smartct23:27:021
179940,0migration/14312-21kthreadcore21:12:041
179940,0migration/13623-21kthreadcore00:02:101
179940,0migration/13212-21taskset19:48:121
179940,0migration/13206-21kthreadcore22:47:041
179940,0migration/130916-21sh22:26:451
179940,0migration/128143-21taskset18:57:571
179940,0migration/127735-21kthreadcore20:27:061
179940,0migration/127631-21kthreadcore20:07:061
179940,0migration/121064-21runrttasks22:18:031
179940,0migration/116227-21kthreadcore23:12:031
179940,0migration/115886-21taskset00:07:571
179940,0migration/113828-21aten_r1power_vo23:31:591
179940,0migration/111593-21needreboot22:12:061
179940,0migration/110465-21taskset19:10:101
169940,0watchdog/17064-21ssh22:31:301
169940,0watchdog/128715-21munin-node19:01:591
169940,0watchdog/118910-21fschecks_time20:01:591
169940,0watchdog/116743-21hwlatdetect23:52:021
169940,0watchdog/112550-21interrupts20:42:031
169940,0watchdog/111439-21fschecks_count19:11:591
169940,0watchdog/10-21swapper/123:42:061
169940,0watchdog/10-21swapper/119:27:071
149940,0migration/028928-21kthreadcore22:42:060
149940,0migration/020679-21timerandwakeup18:52:120
149940,0migration/018373-21kthreadcore23:52:050
149940,0migration/017241-21kthreadcore21:57:050
320339932,0cyclictest3599-21lldpd19:52:542
320329933,0cyclictest30284-21awk20:32:041
320329932,0cyclictest19132-21awk19:17:021
320329932,0cyclictest16834-21fschecks_time22:37:011
320329930,0cyclictest22757-21idleruntime19:42:021
320329930,0cyclictest17103-21pmu-power19:57:081
320319933,0cyclictest6946-21kworker/0:220:22:010
320319933,0cyclictest6946-21kworker/0:219:47:010
320319933,0cyclictest3023-21kworker/0:022:52:270
320319933,0cyclictest3023-21kworker/0:022:12:030
320319933,0cyclictest3023-21kworker/0:021:06:530
320319933,0cyclictest3023-21kworker/0:021:02:490
320319933,0cyclictest17592-21awk21:22:060
320319932,0cyclictest23871-21snmpd22:36:470
320319932,0cyclictest23871-21snmpd21:17:010
320319932,0cyclictest15179-21awk22:17:030
320319932,0cyclictest13230-21cut23:12:020
320319930,0cyclictest5521-21diskstats20:56:590
320319930,0cyclictest32018-21perl22:27:010
320319930,0cyclictest23393-21perl22:22:040
320319930,0cyclictest11523-21perl19:11:590
249930,0watchdog/20-21swapper/219:27:072
249930,0watchdog/20-21swapper/219:01:592
169930,0watchdog/111105-21diskmemload21:51:471
169930,0watchdog/111105-21diskmemload20:51:591
169930,0watchdog/10-21swapper/121:57:221
169930,0watchdog/10-21swapper/121:07:031
169930,0watchdog/10-21swapper/120:37:031
169930,0watchdog/10-21swapper/120:37:031
169930,0watchdog/10-21swapper/120:21:511
169930,0watchdog/10-21swapper/118:52:071
149930,0migration/026674-21runrttasks23:01:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional