You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:29
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2.osadl.org (updated Thu Apr 03, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2412024644,0sleep10-21swapper/118:33:291
2405523928,0sleep30-21swapper/318:32:433
2388623927,0sleep20-21swapper/218:30:462
2408723625,0sleep00-21swapper/018:33:080
329970,0watchdog/30-21swapper/323:04:223
329970,0watchdog/30-21swapper/322:34:183
329970,0watchdog/30-21swapper/321:50:503
329970,0watchdog/30-21swapper/321:14:153
329970,0watchdog/30-21swapper/320:36:593
329970,0watchdog/30-21swapper/320:36:593
29925070,0irq/32-eth0-rx-0-21swapper/322:21:073
29925070,0irq/32-eth0-rx-0-21swapper/321:24:023
339960,0migration/3509-21gltestperf20:09:183
339960,0migration/322387-21irqrtprio20:44:223
339960,0migration/312766-21sh22:11:243
339960,0migration/310832-21sh21:20:083
329960,0watchdog/35584-21ssh00:01:103
329960,0watchdog/321579-21fschecks_time18:54:193
329960,0watchdog/30-21swapper/323:56:543
329960,0watchdog/30-21swapper/323:33:343
329960,0watchdog/30-21swapper/323:24:023
329960,0watchdog/30-21swapper/323:15:023
329960,0watchdog/30-21swapper/323:10:583
329960,0watchdog/30-21swapper/323:00:583
329960,0watchdog/30-21swapper/322:57:543
329960,0watchdog/30-21swapper/322:44:023
329960,0watchdog/30-21swapper/322:39:223
329960,0watchdog/30-21swapper/322:27:423
329960,0watchdog/30-21swapper/322:04:023
329960,0watchdog/30-21swapper/321:45:383
329960,0watchdog/30-21swapper/321:39:383
329960,0watchdog/30-21swapper/321:34:383
329960,0watchdog/30-21swapper/321:31:183
329960,0watchdog/30-21swapper/321:12:193
329960,0watchdog/30-21swapper/321:07:033
329960,0watchdog/30-21swapper/320:57:033
329960,0watchdog/30-21swapper/320:49:433
329960,0watchdog/30-21swapper/318:59:033
29925060,0irq/32-eth0-rx-33552sleep322:51:013
29925060,0irq/32-eth0-rx-14052-21tr20:39:203
29925060,0irq/32-eth0-rx-0-21swapper/323:34:563
29925060,0irq/32-eth0-rx-0-21swapper/323:22:483
29925060,0irq/32-eth0-rx-0-21swapper/321:56:573
29925060,0irq/32-eth0-rx-0-21swapper/319:54:183
242379965,0cyclictest3207-21diskmemload21:51:382
242379960,0cyclictest0-21swapper/221:35:242
149960,0migration/031491-21runrttasks23:12:160
149960,0migration/025229-21sh21:54:270
8315050,0irq/26-0000:00:7158-21perf22:33:562
339950,0migration/38696-21kthreadcore20:14:233
339950,0migration/34224-21hddtemp_smartct19:49:173
339950,0migration/332446-21kthreadcore19:44:213
339950,0migration/32327-21kthreadcore18:39:233
339950,0migration/315006-21kthreadcore19:09:223
339950,0migration/313420-21munin-node23:44:193
329950,0watchdog/30-21swapper/323:49:103
329950,0watchdog/30-21swapper/322:29:343
329950,0watchdog/30-21swapper/322:14:583
329950,0watchdog/30-21swapper/320:59:473
329950,0watchdog/30-21swapper/320:29:153
329950,0watchdog/30-21swapper/320:24:233
329950,0watchdog/30-21swapper/320:22:313
329950,0watchdog/30-21swapper/320:06:353
329950,0watchdog/30-21swapper/320:01:393
329950,0watchdog/30-21swapper/319:40:353
329950,0watchdog/30-21swapper/319:34:033
329950,0watchdog/30-21swapper/319:29:113
329950,0watchdog/30-21swapper/319:27:073
329950,0watchdog/30-21swapper/319:19:193
329950,0watchdog/30-21swapper/319:14:113
329950,0watchdog/30-21swapper/319:04:153
329950,0watchdog/30-21swapper/318:49:273
329950,0watchdog/30-21swapper/318:47:153
329950,0watchdog/30-21swapper/318:34:033
29935050,0irq/33-eth0-tx-28607-21munin-node20:29:210
29935050,0irq/33-eth0-tx-24615-21munin-node19:19:180
29935050,0irq/33-eth0-tx-24014-21cut20:44:232
29935050,0irq/33-eth0-tx-20943-21munin-node18:54:310
29935050,0irq/33-eth0-tx-19253-21tune2fs21:54:122
29935050,0irq/33-eth0-tx-16992-21munin-run23:24:012
29935050,0irq/33-eth0-tx-15730-21swap22:34:222
29935050,0irq/33-eth0-tx-0-21swapper/223:34:422
29935050,0irq/33-eth0-tx-0-21swapper/222:45:272
29935050,0irq/33-eth0-tx-0-21swapper/222:40:222
29935050,0irq/33-eth0-tx-0-21swapper/222:15:312
29935050,0irq/33-eth0-tx-0-21swapper/020:34:230
29935050,0irq/33-eth0-tx-0-21swapper/020:34:230
29925050,0irq/32-eth0-rx-0-21swapper/323:39:043
29925050,0irq/32-eth0-rx-0-21swapper/321:59:263
27150,0ktimersoftd/20-21swapper/222:07:332
259950,0migration/2507-21gltestperf20:09:182
259950,0migration/227138-21sendmail_mailst18:54:272
259950,0migration/21471-21kthreadcore20:29:232
259950,0migration/211398-21unixbench_singl22:54:322
259950,0migration/210828-21sh20:36:122
259950,0migration/210828-21sh20:36:122
242379955,0cyclictest0-21swapper/222:27:062
242379955,0cyclictest0-21swapper/220:42:162
242379950,0cyclictest0-21swapper/221:08:202
242379950,0cyclictest0-21swapper/220:49:592
242369955,0cyclictest0-21swapper/121:39:151
242359955,0cyclictest0-21swapper/022:52:130
242359954,0cyclictest23015-21awk23:04:190
242359950,0cyclictest0-21swapper/022:45:360
242359950,0cyclictest0-21swapper/021:07:220
242359950,0cyclictest0-21swapper/020:50:400
249950,0watchdog/23153-21cpu19:04:152
179950,0migration/1399-21kthreadcore23:59:221
179950,0migration/122449-21kthreadcore23:24:221
179950,0migration/122293-21munin-node23:04:161
169950,0watchdog/127043-21memory19:39:231
149950,0migration/08772-21kthreadcore19:04:210
149950,0migration/08035-21sh22:30:570
149950,0migration/05311-21timerandwakeup22:24:310
149950,0migration/03544-21kthreadcore20:54:200
149950,0migration/032557-21fschecks_count20:09:170
149950,0migration/031650-21kthreadcore19:44:200
149950,0migration/027254-21sh21:59:140
149950,0migration/022583-21kthreadcore20:44:220
149950,0migration/022504-21taskset22:19:040
149950,0migration/020567-21taskset23:49:020
149950,0migration/017603-21kthreadcore20:19:210
149950,0migration/017226-21sh23:24:080
149950,0migration/016393-21kthreadcore18:49:210
149950,0migration/014222-21gltestperf22:59:160
149950,0migration/012589-21aten_r1power_fr21:24:190
149950,0migration/012116-21sh22:56:270
149950,0migration/011731-21kthreadcore22:34:150
8315040,0irq/26-0000:00:0-21swapper/223:54:132
29935040,0irq/33-eth0-tx-7826-21ls21:44:012
29935040,0irq/33-eth0-tx-2697-21ssh22:49:102
29935040,0irq/33-eth0-tx-10643-21ssh21:19:342
29935040,0irq/33-eth0-tx-0-21swapper/223:59:082
29935040,0irq/33-eth0-tx-0-21swapper/223:30:272
29935040,0irq/33-eth0-tx-0-21swapper/222:09:082
29935040,0irq/33-eth0-tx-0-21swapper/221:59:112
29935040,0irq/33-eth0-tx-0-21swapper/221:39:162
29935040,0irq/33-eth0-tx-0-21swapper/221:24:192
29935040,0irq/33-eth0-tx-0-21swapper/221:14:112
29935040,0irq/33-eth0-tx-0-21swapper/221:09:082
29935040,0irq/33-eth0-tx-0-21swapper/220:59:072
29935040,0irq/33-eth0-tx-0-21swapper/020:04:260
29935040,0irq/33-eth0-tx-0-21swapper/019:49:140
29935040,0irq/33-eth0-tx-0-21swapper/019:29:230
29935040,0irq/33-eth0-tx-0-21swapper/019:24:140
29935040,0irq/33-eth0-tx-0-21swapper/018:34:150
259940,0migration/232430-21fschecks_count20:54:162
259940,0migration/231768-21runrttasks20:08:212
259940,0migration/223719-21if_eth1.222:19:172
259940,0migration/222260-21fschecks_time20:24:162
259940,0migration/217505-21kthreadcore22:59:202
259940,0migration/21669-21kthreadcore23:14:152
259940,0migration/213842-21latency_hist19:34:022
259940,0migration/210043-21kthreadcore18:44:212
242379943,0cyclictest8030-21awk20:14:212
242379943,0cyclictest4416-21awk19:49:192
242379943,0cyclictest3599-21lldpd23:09:482
242379943,0cyclictest3599-21lldpd21:29:102
242379943,0cyclictest3599-21lldpd18:37:032
242379943,0cyclictest3262-21irqbalance23:49:262
242379943,0cyclictest3262-21irqbalance19:29:172
242379943,0cyclictest3262-21irqbalance19:24:372
242379943,0cyclictest3262-21irqbalance19:19:072
242379943,0cyclictest3262-21irqbalance19:14:072
242379943,0cyclictest3262-21irqbalance19:09:172
242379943,0cyclictest22205-21awk19:39:192
242379943,0cyclictest15320-21awk20:19:192
242379943,0cyclictest14821-21awk18:49:192
242379943,0cyclictest13444-21awk23:44:192
242379940,0cyclictest2722-21munin-run19:04:022
242379940,0cyclictest0-21swapper/220:00:392
242369943,0cyclictest3599-21lldpd23:29:511
242369943,0cyclictest3599-21lldpd22:13:441
242369943,0cyclictest3599-21lldpd21:08:401
242369943,0cyclictest3599-21lldpd20:50:341
242369943,0cyclictest3599-21lldpd20:48:041
242369943,0cyclictest3599-21lldpd19:46:031
242369943,0cyclictest3599-21lldpd19:38:031
242369943,0cyclictest23871-21snmpd19:31:281
242369943,0cyclictest23871-21snmpd18:39:131
242369943,0cyclictest11364-21awk19:09:191
242369940,0cyclictest0-21swapper/123:47:441
242359944,0cyclictest0-21swapper/022:18:290
242359944,0cyclictest0-21swapper/021:53:430
242359944,0cyclictest0-21swapper/021:32:360
242359943,0cyclictest9819-21awk23:19:190
242359943,0cyclictest9427-21awk20:59:190
242359943,0cyclictest4983-21awk21:19:170
242359943,0cyclictest22466-21awk20:24:170
242359943,0cyclictest18542-21awk19:14:210
242359943,0cyclictest15553-21kworker/0:018:39:190
242359943,0cyclictest13683-21sadc22:14:010
249940,0watchdog/221922-21aten_r1power_en23:04:142
249940,0watchdog/20-21swapper/223:39:222
249940,0watchdog/20-21swapper/223:24:222
2220240,0sleep00-21swapper/021:15:200
179940,0migration/19952-21kthreadcore22:54:251
179940,0migration/15262-21kthreadcore22:04:211
179940,0migration/132295-21cpuspeed_turbos20:09:161
179940,0migration/132281-21kthreadcore23:34:171
179940,0migration/131835-21aten_r1power_cu19:24:131
179940,0migration/131020-21latency_hist22:24:011
179940,0migration/128749-21aten_r1power_vo20:29:161
179940,0migration/128328-21fschecks_time21:14:121
179940,0migration/125803-21gltestperf20:04:171
179940,0migration/124069-21idleruntime21:34:131
179940,0migration/123387-21kthreadcore20:24:191
179940,0migration/123240-21kthreadcore21:54:171
179940,0migration/121563-21kthreadcore19:59:211
179940,0migration/117266-21kthreadcore21:24:261
179940,0migration/117038-21sh21:50:371
179940,0migration/114809-21fschecks_time22:14:111
179940,0migration/113063-21kthreadcore22:34:161
179940,0migration/112136-21kthreadcore19:54:241
179940,0migration/111367-21kthreadcore18:44:221
179940,0migration/111165-21kthreadcore20:59:211
179940,0migration/11025-21sh22:45:051
169940,0watchdog/18001-21rm22:30:501
169940,0watchdog/15732-21kthreadcore21:19:191
169940,0watchdog/125489-21perl19:19:151
169940,0watchdog/122701-21ssh21:33:501
169940,0watchdog/119640-21ssh22:59:261
169940,0watchdog/118630-21tune2fs22:39:141
169940,0watchdog/11815-21rm22:00:381
169940,0watchdog/114651-21memory23:19:221
169940,0watchdog/111130-21ssh20:36:591
169940,0watchdog/111130-21ssh20:36:591
169940,0watchdog/10-21swapper/123:39:221
169940,0watchdog/10-21swapper/122:20:301
169940,0watchdog/10-21swapper/120:54:111
169940,0watchdog/10-21swapper/119:04:151
169940,0watchdog/10-21swapper/118:54:231
149940,0migration/023737-21kthreadcore19:39:210
149940,0migration/018949-21runrttasks22:39:150
149940,0migration/018846-21if_eth1.219:59:190
149940,0migration/013601-21kthreadcore21:44:250
149940,0migration/010927-21kthreadcore18:44:210
29935030,0irq/33-eth0-tx-13987-21aten2.4-expect19:34:170
242379932,0cyclictest3262-21irqbalance19:44:072
242379932,0cyclictest13706-21grep19:54:242
242369932,0cyclictest8045-21awk20:14:211
242369932,0cyclictest3599-21lldpd23:15:181
242369932,0cyclictest3599-21lldpd22:49:171
242369932,0cyclictest3599-21lldpd20:40:341
242369932,0cyclictest3599-21lldpd19:49:031
242369932,0cyclictest3599-21lldpd18:59:331
242369932,0cyclictest23871-21snmpd20:23:281
242369932,0cyclictest14893-21awk18:49:191
242359933,0cyclictest9491-21kworker/0:121:09:190
242359933,0cyclictest9491-21kworker/0:120:14:050
242359933,0cyclictest9491-21kworker/0:119:54:050
242359933,0cyclictest20778-21kworker/0:023:59:020
242359933,0cyclictest20778-21kworker/0:023:29:020
242359933,0cyclictest20778-21kworker/0:023:14:080
242359933,0cyclictest20778-21kworker/0:022:04:070
242359933,0cyclictest20778-21kworker/0:021:34:210
242359933,0cyclictest15553-21kworker/0:019:09:090
242359933,0cyclictest15553-21kworker/0:018:59:110
242359932,0cyclictest5034-21awk23:39:200
242359932,0cyclictest28283-21tail23:54:170
242359932,0cyclictest27889-21stty23:34:100
242359932,0cyclictest2761-21cat21:39:220
242359932,0cyclictest13290-21awk20:39:180
242359930,0cyclictest12728-21diskstats23:44:160
249930,0watchdog/20-21swapper/218:39:192
179930,0migration/19399-21irqrtprio21:44:231
179930,0migration/130526-21diskmemload23:09:291
169930,0watchdog/13207-21diskmemload21:09:111
169930,0watchdog/10-21swapper/123:51:261
169930,0watchdog/10-21swapper/119:14:271
169930,0watchdog/10-21swapper/118:34:151
242369920,0cyclictest0-21swapper/123:54:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional