You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-25 - 11:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Mon Nov 25, 2024 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1718625958,0sleep00-21swapper/018:36:590
2305824839,0sleep10-21swapper/118:41:161
2301823728,0sleep30-21swapper/318:40:463
2305223627,0sleep20-21swapper/218:41:132
329980,0watchdog/30-21swapper/322:01:593
29925070,0irq/32-eth0-rx-0-21swapper/320:58:163
8315060,0irq/26-0000:00:1654-21crond19:11:422
4160,0ktimersoftd/010650-21grep22:57:010
339960,0migration/327376-21cpuspeed_turbos22:11:563
339960,0migration/32082-21irqrtprio23:52:013
329960,0watchdog/323483-21aten_r1power_po18:41:553
329960,0watchdog/30-21swapper/323:38:073
329960,0watchdog/30-21swapper/323:28:433
329960,0watchdog/30-21swapper/323:28:433
329960,0watchdog/30-21swapper/323:03:313
329960,0watchdog/30-21swapper/322:51:473
329960,0watchdog/30-21swapper/322:29:593
329960,0watchdog/30-21swapper/322:11:393
329960,0watchdog/30-21swapper/321:56:553
329960,0watchdog/30-21swapper/321:54:193
329960,0watchdog/30-21swapper/321:37:353
329960,0watchdog/30-21swapper/321:19:553
329960,0watchdog/30-21swapper/321:08:593
329960,0watchdog/30-21swapper/320:52:073
329960,0watchdog/30-21swapper/320:11:513
329960,0watchdog/30-21swapper/320:02:033
329960,0watchdog/30-21swapper/319:51:553
329960,0watchdog/30-21swapper/319:22:073
329960,0watchdog/30-21swapper/300:01:553
29935060,0irq/33-eth0-tx-18466-21cut23:42:012
29935060,0irq/33-eth0-tx-0-21swapper/223:01:502
29925060,0irq/32-eth0-rx-45822sleep322:35:133
29925060,0irq/32-eth0-rx-4214-21perl22:16:563
29925060,0irq/32-eth0-rx-0-21swapper/323:48:083
29925060,0irq/32-eth0-rx-0-21swapper/323:42:023
29925060,0irq/32-eth0-rx-0-21swapper/323:34:513
29925060,0irq/32-eth0-rx-0-21swapper/323:08:043
29925060,0irq/32-eth0-rx-0-21swapper/322:56:523
29925060,0irq/32-eth0-rx-0-21swapper/322:47:303
29925060,0irq/32-eth0-rx-0-21swapper/322:37:043
29925060,0irq/32-eth0-rx-0-21swapper/322:24:293
29925060,0irq/32-eth0-rx-0-21swapper/321:48:343
29925060,0irq/32-eth0-rx-0-21swapper/321:42:003
29925060,0irq/32-eth0-rx-0-21swapper/321:32:113
29925060,0irq/32-eth0-rx-0-21swapper/321:12:543
29925060,0irq/32-eth0-rx-0-21swapper/320:43:273
231659960,0cyclictest0-21swapper/023:15:570
231659960,0cyclictest0-21swapper/020:46:380
149960,0migration/05430-21munin-node22:36:530
149960,0migration/01286-21latency_hist21:56:430
8315050,0irq/26-0000:00:28349-21irqcore00:07:012
339950,0migration/37524-21kthreadcore18:52:013
339950,0migration/34532-21latency20:17:013
339950,0migration/33698-21kthreadcore19:32:023
339950,0migration/331805-21kthreadcore19:07:013
339950,0migration/329394-21kthreadcore20:37:003
339950,0migration/319920-21munin-node19:01:553
339950,0migration/314026-21munin-node22:41:573
339950,0migration/313390-21hwlatdetect23:21:563
339950,0migration/310630-21kthreadcore21:22:053
329950,0watchdog/322714-21port_sshpriv20:07:073
329950,0watchdog/316194-21munin-node21:26:593
329950,0watchdog/30-21swapper/323:16:473
329950,0watchdog/30-21swapper/323:11:473
329950,0watchdog/30-21swapper/321:02:353
329950,0watchdog/30-21swapper/320:31:433
329950,0watchdog/30-21swapper/320:26:593
329950,0watchdog/30-21swapper/320:21:593
329950,0watchdog/30-21swapper/319:56:473
329950,0watchdog/30-21swapper/319:46:433
329950,0watchdog/30-21swapper/319:41:033
329950,0watchdog/30-21swapper/319:27:193
329950,0watchdog/30-21swapper/319:12:593
329950,0watchdog/30-21swapper/318:56:473
329950,0watchdog/30-21swapper/318:47:153
329950,0watchdog/30-21swapper/300:07:073
29935050,0irq/33-eth0-tx-6315-21iceccd21:08:332
29935050,0irq/33-eth0-tx-28674-21ntpdc22:47:022
29935050,0irq/33-eth0-tx-28228-21ssh23:27:292
29935050,0irq/33-eth0-tx-28228-21ssh23:27:292
29935050,0irq/33-eth0-tx-17579-21ssh22:04:402
29935050,0irq/33-eth0-tx-14933-21grep22:42:012
29935050,0irq/33-eth0-tx-12706-21munin-node18:57:000
29935050,0irq/33-eth0-tx-0-21swapper/223:12:012
29935050,0irq/33-eth0-tx-0-21swapper/223:09:402
29935050,0irq/33-eth0-tx-0-21swapper/222:38:502
29935050,0irq/33-eth0-tx-0-21swapper/222:30:462
29935050,0irq/33-eth0-tx-0-21swapper/221:59:222
29935050,0irq/33-eth0-tx-0-21swapper/221:51:062
29935050,0irq/33-eth0-tx-0-21swapper/220:56:582
29935050,0irq/33-eth0-tx-0-21swapper/019:52:050
29935050,0irq/33-eth0-tx-0-21swapper/019:46:550
29935050,0irq/33-eth0-tx-0-21swapper/019:41:570
29935050,0irq/33-eth0-tx-0-21swapper/019:36:570
29935050,0irq/33-eth0-tx-0-21swapper/019:26:570
29925050,0irq/32-eth0-rx-13941-21munin-node19:41:573
29925050,0irq/32-eth0-rx-11942-21grep19:17:013
29925050,0irq/32-eth0-rx-0-21swapper/323:57:083
29925050,0irq/32-eth0-rx-0-21swapper/320:47:563
259950,0migration/24260-21kthreadcore23:52:022
259950,0migration/229641-21sh21:15:122
259950,0migration/229073-21kthreadcore21:32:032
259950,0migration/21766-21kthreadcore22:32:022
231679955,0cyclictest0-21swapper/219:32:092
231679950,0cyclictest0-21swapper/222:22:342
231669955,0cyclictest23871-21snmpd23:58:121
231669955,0cyclictest0-21swapper/122:41:021
231669950,0cyclictest0-21swapper/122:30:471
231669950,0cyclictest0-21swapper/121:36:431
231659955,0cyclictest0-21swapper/023:36:260
231659955,0cyclictest0-21swapper/023:22:340
231659955,0cyclictest0-21swapper/021:29:410
231659955,0cyclictest0-21swapper/000:11:360
179950,0migration/17300-21fschecks_time21:22:011
179950,0migration/16303-21sh21:21:411
179950,0migration/14978-21kthreadcore23:52:021
179950,0migration/131241-21taskset21:34:431
179950,0migration/130269-21timerwakeupswit20:12:101
179950,0migration/129879-21munin-node19:27:071
179950,0migration/127928-21fschecks_time19:06:571
179950,0migration/121221-21kthreadcore23:42:021
179950,0migration/11264-21kthreadcore22:32:021
169950,0watchdog/132570-21sed22:51:591
149950,0migration/08443-21kthreadcore22:17:010
149950,0migration/05682-21taskset20:21:190
149950,0migration/05215-21fschecks_count21:01:590
149950,0migration/04827-21kthreadcore19:12:020
149950,0migration/030719-21kthreadcore20:37:010
149950,0migration/030628-21munin-node23:47:050
149950,0migration/030040-21kthreadcore22:12:010
149950,0migration/027815-21switchtime23:27:080
149950,0migration/027815-21switchtime23:27:080
149950,0migration/027269-21taskset20:35:320
149950,0migration/024903-21kthreadcore19:02:030
149950,0migration/022569-21sh23:42:500
149950,0migration/021847-21kthreadcore00:02:040
149950,0migration/020281-21fschecks_time20:51:580
149950,0migration/018956-21kthreadcore20:07:040
149950,0migration/018744-21kthreadcore21:47:010
149950,0migration/016736-21kthreadcore22:42:050
149950,0migration/01641-21kthreadcore21:37:040
149950,0migration/014357-21sh21:42:060
149950,0migration/012918-21ntp_states23:37:020
149950,0migration/012036-21unixbench_singl18:52:090
149950,0migration/011337-21munin-node20:46:540
149950,0migration/010897-21fschecks_time22:02:020
8315040,0irq/26-0000:00:0-21swapper/219:51:432
29935040,0irq/33-eth0-tx-6386-21runrttasks20:51:572
29935040,0irq/33-eth0-tx-5671-21ssh23:34:462
29935040,0irq/33-eth0-tx-2210-21diskmemload23:18:002
29935040,0irq/33-eth0-tx-0-21swapper/223:56:522
29935040,0irq/33-eth0-tx-0-21swapper/223:37:052
29935040,0irq/33-eth0-tx-0-21swapper/222:56:442
29935040,0irq/33-eth0-tx-0-21swapper/221:51:502
29935040,0irq/33-eth0-tx-0-21swapper/220:43:252
259940,0migration/231610-21kthreadcore22:12:022
259940,0migration/227032-21kthreadcore20:12:022
259940,0migration/222034-21kthreadcore19:22:052
259940,0migration/221879-21kthreadcore19:02:022
259940,0migration/220892-21kthreadcore21:27:032
259940,0migration/220788-21kthreadcore00:02:032
259940,0migration/216963-21missed_timers20:47:022
259940,0migration/21339-21kthreadcore18:47:022
259940,0migration/211622-21sh21:02:502
249940,0watchdog/28576-21cpu21:41:592
249940,0watchdog/223553-21grep20:31:592
249940,0watchdog/215489-21open_inodes20:02:072
249940,0watchdog/20-21swapper/222:51:592
231679943,0cyclictest3599-21lldpd23:21:482
231679943,0cyclictest3599-21lldpd22:16:452
231679943,0cyclictest3599-21lldpd20:21:572
231679943,0cyclictest3599-21lldpd19:42:572
231679943,0cyclictest3599-21lldpd19:40:572
231679943,0cyclictest3599-21lldpd19:12:572
231679943,0cyclictest3599-21lldpd18:51:572
231679943,0cyclictest3599-21lldpd18:45:572
231679943,0cyclictest3262-21irqbalance23:46:472
231679943,0cyclictest3262-21irqbalance22:06:472
231679943,0cyclictest3262-21irqbalance21:36:472
231679943,0cyclictest3262-21irqbalance21:21:472
231679943,0cyclictest31589-21awk21:16:582
231679943,0cyclictest24766-21awk19:27:002
231679943,0cyclictest13834-21awk18:56:582
231679940,0cyclictest0-21swapper/220:10:522
231669944,0cyclictest0-21swapper/119:37:031
231669944,0cyclictest0-21swapper/100:09:191
231669943,0cyclictest3599-21lldpd23:24:181
231669943,0cyclictest3599-21lldpd20:51:561
231669943,0cyclictest3599-21lldpd20:16:561
231669943,0cyclictest3599-21lldpd19:41:571
231669943,0cyclictest2984-21awk19:12:001
231669943,0cyclictest23871-21snmpd20:40:101
231669943,0cyclictest23871-21snmpd19:49:251
231669943,0cyclictest23871-21snmpd18:49:401
231669943,0cyclictest23622-21awk22:46:571
231669943,0cyclictest15098-21awk22:42:021
231669943,0cyclictest14301-21awk20:26:581
231659944,0cyclictest0-21swapper/023:59:270
231659943,0cyclictest7702-21awk21:22:020
231659943,0cyclictest3599-21lldpd21:15:320
231659943,0cyclictest31920-21awk19:31:580
231659943,0cyclictest28816-21sadc22:31:420
231659943,0cyclictest28214-21awk19:06:590
231659943,0cyclictest19614-21awk22:06:570
19140,0ktimersoftd/10-21swapper/121:53:251
179940,0migration/18443-21kthreadcore18:52:011
179940,0migration/17488-21kthreadcore21:02:041
179940,0migration/17340-21irqcore20:21:591
179940,0migration/16654-21latency_hist23:36:431
179940,0migration/14434-21kthreadcore23:12:001
179940,0migration/14406-21kthreadcore21:57:011
179940,0migration/13977-21kthreadcore19:32:021
179940,0migration/13267-21taskset22:16:461
179940,0migration/13012-21fschecks_count20:41:591
179940,0migration/128769-21runrttasks23:28:481
179940,0migration/128769-21runrttasks23:28:481
179940,0migration/127257-21runrttasks20:35:221
179940,0migration/126677-21kthreadcore18:42:021
179940,0migration/123081-21taskset20:07:121
179940,0migration/122829-21kthreadcore19:02:031
179940,0migration/122521-21fschecks_count21:11:551
179940,0migration/122199-21kthreadcore00:02:041
179940,0migration/120553-21kthreadcore21:47:011
179940,0migration/120412-21kthreadcore21:27:031
179940,0migration/119461-21kthreadcore23:02:051
179940,0migration/116201-21kthreadcore20:47:011
179940,0migration/112643-21kthreadcore22:57:021
179940,0migration/112058-21kthreadcore20:02:041
169940,0watchdog/132074-21chrt23:09:071
169940,0watchdog/123871-21snmpd21:00:351
169940,0watchdog/119005-21ssh22:23:151
169940,0watchdog/114792-21irqrtprio21:06:591
169940,0watchdog/10-21swapper/123:46:591
169940,0watchdog/10-21swapper/119:16:591
149940,0migration/04417-21kthreadcore23:52:020
149940,0migration/029702-21kthreadcore23:07:010
149940,0migration/029153-21fschecks_time20:56:560
149940,0migration/018846-21kthreadcore19:22:040
8315030,0irq/26-0000:00:0-21swapper/219:56:482
29935030,0irq/33-eth0-tx-30727-21aten2.4-expect18:46:560
29935030,0irq/33-eth0-tx-0-21swapper/018:41:560
231679932,0cyclictest9376-21aten_r1power_vo19:16:552
231679932,0cyclictest3599-21lldpd19:46:572
231679932,0cyclictest14247-21awk20:26:572
231679932,0cyclictest1422-21sensors20:37:052
231679931,0cyclictest424-21kthreadcore20:16:592
231669933,0cyclictest3599-21lldpd22:11:441
231669933,0cyclictest0-21swapper/121:42:041
231669932,0cyclictest4238-21sensors23:32:051
231669932,0cyclictest3599-21lldpd23:16:481
231669932,0cyclictest17526-21awk19:22:021
231659933,0cyclictest734-21kworker/0:220:26:530
231659933,0cyclictest26613-21kworker/0:023:16:470
231659933,0cyclictest26613-21kworker/0:023:01:510
231659933,0cyclictest26613-21kworker/0:022:51:570
231659933,0cyclictest26613-21kworker/0:022:31:430
231659933,0cyclictest26613-21kworker/0:022:22:410
231659933,0cyclictest26613-21kworker/0:021:16:450
231659933,0cyclictest26613-21kworker/0:021:06:570
231659933,0cyclictest23638-21awk22:46:570
231659933,0cyclictest0-21swapper/020:21:530
231659932,0cyclictest30002-21sensors21:32:060
231659932,0cyclictest26613-21kworker/0:021:51:510
231659932,0cyclictest24913-21awk20:11:590
231659931,0cyclictest6470-21grep19:57:020
231659930,0cyclictest0-21swapper/020:01:590
179930,0migration/126562-21sh22:10:591
169930,0watchdog/118920-21python18:57:071
169930,0watchdog/10-21swapper/122:01:591
169930,0watchdog/10-21swapper/119:57:031
169930,0watchdog/10-21swapper/119:51:551
231659921,0cyclictest9136-21date19:16:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional