You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-16 - 08:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Tue Jul 16, 2024 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1696423827,0sleep10-21swapper/118:47:541
1710123728,0sleep30-21swapper/318:49:323
1690023626,0sleep20-21swapper/218:47:112
1711423524,0sleep00-21swapper/018:49:370
8315060,0irq/26-0000:00:9130-21crond21:34:422
4160,0ktimersoftd/02065-21md0_raid123:25:170
339960,0migration/39329-21sh22:49:523
29925060,0irq/32-eth0-rx-30642-21python23:55:263
29925060,0irq/32-eth0-rx-2972-21munin-run19:24:433
171929960,0cyclictest0-21swapper/121:13:351
171919965,0cyclictest11069-21ps21:35:080
149960,0migration/03470-21sh23:21:050
149960,0migration/02412-21taskset23:04:300
8315050,0irq/26-0000:00:7019-21sort23:44:432
8315050,0irq/26-0000:00:17238-21yum20:15:272
4150,0ktimersoftd/00-21swapper/021:23:360
35150,0ktimersoftd/30-21swapper/319:34:353
339950,0migration/39499-21fschecks_count00:04:583
339950,0migration/3630-21kthreadcore23:40:033
339950,0migration/35926-21munin-node20:30:183
339950,0migration/35035-21taskset23:24:463
339950,0migration/331233-21taskset18:55:383
339950,0migration/328838-21aten_r1power_en20:49:563
339950,0migration/328377-21taskset20:48:183
339950,0migration/325756-21taskset21:42:443
339950,0migration/325342-21gltestperf22:40:023
339950,0migration/325184-21taskset00:13:283
339950,0migration/325125-21cpuspeed_turbos19:40:003
339950,0migration/323868-21kthreadcore22:00:133
339950,0migration/323797-21munin-node19:35:203
339950,0migration/322449-21aten_r1power_vo22:19:563
339950,0migration/321824-21kthreadcore23:15:083
339950,0migration/318283-21munin-node20:20:023
339950,0migration/317058-21irqcore22:35:063
339950,0migration/31685-21irqcore00:00:033
339950,0migration/316461-21latency20:15:133
339950,0migration/312078-21sh22:11:013
329950,0watchdog/39670-21python20:10:153
329950,0watchdog/38437-21iostat_ios22:30:073
329950,0watchdog/34095-21sh23:05:033
329950,0watchdog/314963-21iostat_ios23:30:073
329950,0watchdog/314744-21perl19:10:003
329950,0watchdog/312522-21perl21:55:033
329950,0watchdog/30-21swapper/320:39:553
29935050,0irq/33-eth0-tx-6386-21runrttasks23:53:352
29935050,0irq/33-eth0-tx-5092-21munin-node23:25:182
29935050,0irq/33-eth0-tx-28754-21munin-node23:20:012
29935050,0irq/33-eth0-tx-2788-21munin-node23:05:122
29935050,0irq/33-eth0-tx-24440-21munin-node21:24:582
29935050,0irq/33-eth0-tx-2418-21grep22:05:062
29935050,0irq/33-eth0-tx-21868-21ssh21:04:272
29935050,0irq/33-eth0-tx-21568-21munin-node20:44:590
29935050,0irq/33-eth0-tx-19384-21ssh23:13:232
29935050,0irq/33-eth0-tx-17085-21ssh00:09:262
29935050,0irq/33-eth0-tx-0-21swapper/223:31:222
29935050,0irq/33-eth0-tx-0-21swapper/221:46:482
29935050,0irq/33-eth0-tx-0-21swapper/221:13:472
29935050,0irq/33-eth0-tx-0-21swapper/220:55:092
29935050,0irq/33-eth0-tx-0-21swapper/200:02:522
29935050,0irq/33-eth0-tx-0-21swapper/019:39:330
29925050,0irq/32-eth0-rx-9266-21ntpq19:25:153
29925050,0irq/32-eth0-rx-7700-21fschecks_count23:45:023
29925050,0irq/32-eth0-rx-5404-21diskstats20:55:013
29925050,0irq/32-eth0-rx-5346-21memory19:00:113
29925050,0irq/32-eth0-rx-31327-21diskstats22:24:573
29925050,0irq/32-eth0-rx-2989-21cut22:45:093
29925050,0irq/32-eth0-rx-27351-21grep21:25:103
29925050,0irq/32-eth0-rx-23501-21ssh23:53:363
29925050,0irq/32-eth0-rx-16957-21if_err_eth121:20:043
29925050,0irq/32-eth0-rx-0-21swapper/323:35:143
29925050,0irq/32-eth0-rx-0-21swapper/322:15:063
29925050,0irq/32-eth0-rx-0-21swapper/322:08:233
29925050,0irq/32-eth0-rx-0-21swapper/321:54:293
29925050,0irq/32-eth0-rx-0-21swapper/321:34:513
259950,0migration/25844-21kthreadcore21:50:112
259950,0migration/229107-21kthreadcore00:15:122
259950,0migration/227933-21taskset19:17:342
259950,0migration/219493-21kthreadcore19:55:082
259950,0migration/21583-21kthreadcore22:25:072
249950,0watchdog/210687-21iostat21:35:072
171939955,0cyclictest0-21swapper/222:30:362
171939954,0cyclictest17466-21munin-node00:10:222
171929955,0cyclictest0-21swapper/122:43:301
171929955,0cyclictest0-21swapper/121:45:571
171929955,0cyclictest0-21swapper/119:35:251
171929950,0cyclictest0-21swapper/123:47:591
171929950,0cyclictest0-21swapper/122:30:531
171929950,0cyclictest0-21swapper/122:11:111
171919955,0cyclictest0-21swapper/023:55:450
171919950,0cyclictest0-21swapper/019:05:020
179950,0migration/18574-21sh22:48:241
179950,0migration/13880-21idleruntime23:04:581
179950,0migration/122169-21munin-node21:20:161
179950,0migration/118595-21munin-node00:10:041
179950,0migration/117710-21kthreadcore22:35:071
179950,0migration/115138-21fschecks_time20:39:581
179950,0migration/113594-21sh21:55:101
179950,0migration/110060-21taskset20:10:411
149950,0migration/09739-21sh23:05:300
149950,0migration/07410-21kthreadcore20:10:070
149950,0migration/06285-21kthreadcore22:45:100
149950,0migration/05221-21kthreadcore19:25:090
149950,0migration/0424-21sh23:59:540
149950,0migration/03698-21kthreadcore21:10:100
149950,0migration/031824-21sh21:26:580
149950,0migration/031325-21sh22:40:450
149950,0migration/029729-21sh21:07:080
149950,0migration/029234-21kthreadcore21:45:080
149950,0migration/028207-21kthreadcore00:15:120
149950,0migration/027772-21kthreadcore18:55:160
149950,0migration/02491-21kthreadcore21:30:040
149950,0migration/023815-21kthreadcore21:40:090
149950,0migration/023229-21kthreadcore19:15:090
149950,0migration/023179-21kthreadcore23:15:080
149950,0migration/021134-21taskset19:59:310
149950,0migration/020113-21kthreadcore18:50:070
149950,0migration/016208-21df_abs22:34:560
149950,0migration/014469-21sh22:31:470
149950,0migration/014234-21kthreadcore21:55:100
149950,0migration/01420-21kthreadcore19:00:080
149950,0migration/012723-21kthreadcore20:15:120
149950,0migration/011898-21sh22:10:300
149950,0migration/010284-21kthreadcore20:35:140
8315040,0irq/26-0000:00:0-21swapper/222:54:422
8315040,0irq/26-0000:00:0-21swapper/219:49:432
8315040,0irq/26-0000:00:0-21swapper/218:54:432
339940,0migration/331613-21kthreadcore00:15:133
339940,0migration/325937-21fschecks_time20:24:583
339940,0migration/32254-21kthreadcore19:45:113
339940,0migration/319349-21kthreadcore18:50:073
329940,0watchdog/326242-21kthreadcore20:00:113
329940,0watchdog/310305-21kthreadcore19:05:123
329940,0watchdog/30-21swapper/321:00:073
329940,0watchdog/30-21swapper/319:15:003
29935040,0irq/33-eth0-tx-28522-21diskmemload21:54:542
29935040,0irq/33-eth0-tx-15766-21ssh22:34:502
29935040,0irq/33-eth0-tx-0-21swapper/223:34:542
29935040,0irq/33-eth0-tx-0-21swapper/223:15:022
29935040,0irq/33-eth0-tx-0-21swapper/223:00:012
29935040,0irq/33-eth0-tx-0-21swapper/222:50:092
29935040,0irq/33-eth0-tx-0-21swapper/222:44:522
29935040,0irq/33-eth0-tx-0-21swapper/220:50:572
29935040,0irq/33-eth0-tx-0-21swapper/019:10:010
29925040,0irq/32-eth0-rx-27017-21sh21:44:583
29925040,0irq/32-eth0-rx-0-21swapper/323:20:053
29925040,0irq/32-eth0-rx-0-21swapper/323:10:243
29925040,0irq/32-eth0-rx-0-21swapper/323:00:113
29925040,0irq/32-eth0-rx-0-21swapper/322:54:533
29925040,0irq/32-eth0-rx-0-21swapper/321:30:023
29925040,0irq/32-eth0-rx-0-21swapper/321:15:013
29925040,0irq/32-eth0-rx-0-21swapper/321:10:103
29925040,0irq/32-eth0-rx-0-21swapper/321:05:353
29925040,0irq/32-eth0-rx-0-21swapper/320:04:563
29925040,0irq/32-eth0-rx-0-21swapper/320:04:563
29925040,0irq/32-eth0-rx-0-21swapper/319:49:553
259940,0migration/28394-21kthreadcore19:25:102
259940,0migration/227776-21kthreadcore21:05:092
259940,0migration/227182-21kthreadcore20:25:072
259940,0migration/226413-21kthreadcore20:45:082
259940,0migration/225230-21idleruntime23:55:052
259940,0migration/219584-21kthreadcore18:50:072
259940,0migration/218816-21kthreadcore21:20:072
259940,0migration/217637-21latency_hist19:34:432
249940,0watchdog/28373-21if_err_eth1.321:15:032
249940,0watchdog/231955-21idleruntime23:39:592
249940,0watchdog/210304-21kthreadcore19:05:122
171939943,0cyclictest3599-21lldpd22:40:572
171939943,0cyclictest3599-21lldpd22:19:572
171939943,0cyclictest3599-21lldpd22:09:572
171939943,0cyclictest3599-21lldpd20:40:242
171939943,0cyclictest3599-21lldpd19:49:242
171939943,0cyclictest3262-21irqbalance22:14:472
171939943,0cyclictest3262-21irqbalance20:09:472
171939943,0cyclictest3262-21irqbalance20:04:472
171939943,0cyclictest3262-21irqbalance20:04:472
171939943,0cyclictest3262-21irqbalance20:01:172
171939943,0cyclictest29543-21awk19:20:072
171939943,0cyclictest25999-21awk19:40:082
171939943,0cyclictest21369-21awk22:00:082
171939943,0cyclictest19227-21awk21:40:022
171939943,0cyclictest15172-21grep19:10:052
171939943,0cyclictest11491-21awk19:30:022
171929944,0cyclictest0-21swapper/123:35:131
171929943,0cyclictest7812-21awk19:05:051
171929943,0cyclictest4283-21awk19:25:031
171929943,0cyclictest3599-21lldpd23:42:591
171929943,0cyclictest3599-21lldpd23:29:591
171929943,0cyclictest3599-21lldpd23:19:581
171929943,0cyclictest3599-21lldpd22:49:571
171929943,0cyclictest3599-21lldpd22:01:571
171929943,0cyclictest3599-21lldpd21:41:561
171929943,0cyclictest3599-21lldpd21:27:561
171929943,0cyclictest3599-21lldpd20:49:551
171929943,0cyclictest3599-21lldpd20:37:541
171929943,0cyclictest3599-21lldpd20:22:241
171929943,0cyclictest3599-21lldpd00:03:291
171929943,0cyclictest32114-21awk22:25:051
171929943,0cyclictest29629-21awk20:05:031
171929943,0cyclictest29629-21awk20:05:031
171929943,0cyclictest27746-21awk23:00:091
171929943,0cyclictest23871-21snmpd19:23:321
171929943,0cyclictest23871-21snmpd19:12:311
171929943,0cyclictest23871-21snmpd19:01:311
171929943,0cyclictest21251-21awk23:15:011
171929943,0cyclictest19142-21awk22:55:031
171929943,0cyclictest1588-21sadc00:19:421
171929943,0cyclictest12565-21ls23:10:021
171929943,0cyclictest11693-21awk20:15:061
171929943,0cyclictest0-21swapper/119:55:071
171919944,0cyclictest0-21swapper/022:16:540
171919943,0cyclictest9675-21kworker/0:120:50:480
171919943,0cyclictest9675-21kworker/0:120:20:160
171919943,0cyclictest848-21awk20:30:050
171919943,0cyclictest3262-21irqbalance23:36:370
171919943,0cyclictest3262-21irqbalance23:10:270
171919943,0cyclictest3262-21irqbalance22:50:270
171919943,0cyclictest31991-21awk22:25:020
171919943,0cyclictest29874-21awk22:04:590
171919943,0cyclictest29756-21awk20:05:040
171919943,0cyclictest29756-21awk20:05:040
171919943,0cyclictest26160-21awk20:25:030
171919943,0cyclictest23871-21snmpd21:02:020
171919943,0cyclictest23871-21snmpd19:44:170
171919943,0cyclictest15580-21sadc21:19:420
171919943,0cyclictest14931-21awk23:30:060
171919943,0cyclictest10026-21awk00:05:020
171919940,0cyclictest32093-21munin-node19:45:120
179940,0migration/18133-21kthreadcore21:50:121
179940,0migration/1561-21runrttasks21:29:451
179940,0migration/131805-21kthreadcore22:05:041
179940,0migration/131082-21taskset19:40:231
179940,0migration/127557-21rt-features20:00:181
179940,0migration/127290-21kthreadcore23:55:131
179940,0migration/124401-21kthreadcore22:20:071
179940,0migration/117052-21kthreadcore22:15:061
169940,0watchdog/19591-21ssh21:34:551
169940,0watchdog/15215-21aten_r1power_po23:24:591
169940,0watchdog/15048-21tr20:30:111
169940,0watchdog/13652-21grep19:45:121
169940,0watchdog/128839-21kthreadcore20:25:071
169940,0watchdog/127334-21kthreadcore18:55:161
169940,0watchdog/123148-21kthreadcore20:45:071
169940,0watchdog/121656-21cut19:15:001
169940,0watchdog/115254-21processes00:05:151
169940,0watchdog/115170-21ssh21:18:351
149940,0migration/029153-21sh22:20:340
149940,0migration/025504-21kthreadcore22:00:140
149940,0migration/020797-21kthreadcore00:10:080
149940,0migration/017991-21kthreadcore20:40:050
329930,0watchdog/30-21swapper/320:35:033
29925030,0irq/32-eth0-rx-0-21swapper/319:55:233
249930,0watchdog/20-21swapper/220:35:032
249930,0watchdog/20-21swapper/220:19:432
171939933,0cyclictest0-21swapper/218:59:582
171939932,0cyclictest32176-21cat20:29:432
171929932,0cyclictest3599-21lldpd23:52:591
171929932,0cyclictest14612-21awk21:00:021
171929932,0cyclictest14165-21kthreadcore19:30:071
171929932,0cyclictest11387-21sensors20:55:251
171929930,0cyclictest7809-21perl19:49:581
171919933,0cyclictest9675-21kworker/0:120:54:440
171919933,0cyclictest8713-21kworker/0:123:49:470
171919933,0cyclictest8713-21kworker/0:123:44:470
171919933,0cyclictest8713-21kworker/0:123:39:470
171919933,0cyclictest8713-21kworker/0:122:54:430
171919933,0cyclictest8713-21kworker/0:121:50:010
171919933,0cyclictest12989-21kworker/0:019:59:460
171919933,0cyclictest12989-21kworker/0:019:49:440
171919933,0cyclictest12989-21kworker/0:019:29:500
171919933,0cyclictest12989-21kworker/0:019:19:520
179930,0migration/123741-21irqrtprio21:05:081
169930,0watchdog/10-21swapper/118:49:441
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional