You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 07:01
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot2.osadl.org (updated Wed Apr 02, 2025 00:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
105642201184,12sleep10-21swapper/119:06:481
105112196179,13sleep00-21swapper/019:06:050
104942193176,12sleep30-21swapper/319:05:523
105132165147,13sleep20-21swapper/219:06:072
21262470,0sleep20-21swapper/221:25:242
64282420,0sleep10-21swapper/100:17:131
96302400,0sleep10-21swapper/122:02:451
267952400,0sleep20-21swapper/222:11:592
232742130,0sleep20-21swapper/219:30:212
52992110,0sleep10-21swapper/100:00:001
1099099103,1cyclictest3-21ksoftirqd/023:15:010
109909991,1cyclictest3-21ksoftirqd/000:29:590
26422280,0sleep10-21swapper/100:10:361
109909981,2cyclictest0-21swapper/020:45:130
109909972,1cyclictest3-21ksoftirqd/020:10:160
109909971,2cyclictest3-21ksoftirqd/023:40:130
109909970,1cyclictest81rcu_preempt00:20:110
109909970,1cyclictest10691-21kworker/0:119:35:000
109909965,0cyclictest3-21ksoftirqd/000:00:210
109909964,1cyclictest3-21ksoftirqd/021:40:120
109909964,0cyclictest3-21ksoftirqd/023:25:000
109909964,0cyclictest3-21ksoftirqd/022:50:090
109909964,0cyclictest3-21ksoftirqd/022:35:000
109909964,0cyclictest3-21ksoftirqd/021:30:150
109909964,0cyclictest3-21ksoftirqd/021:05:210
109909964,0cyclictest3-21ksoftirqd/019:20:000
109909963,1cyclictest3-21ksoftirqd/022:55:120
109909962,1cyclictest3-21ksoftirqd/023:10:000
109909962,0cyclictest81rcu_preempt23:20:010
109909962,0cyclictest81rcu_preempt21:10:220
109909961,3cyclictest3-21ksoftirqd/023:30:430
109909960,6cyclictest0-21swapper/022:09:150
109909960,3cyclictest809-21basename22:50:010
109909960,2cyclictest3-21ksoftirqd/020:10:000
109909960,2cyclictest3-21ksoftirqd/000:10:000
109909960,1cyclictest6955-21pmu-power22:35:180
109909960,1cyclictest0-21swapper/000:10:110
109909960,0cyclictest10639-21smartctl22:20:150
23898250,0sleep10-21swapper/121:36:201
110049954,1cyclictest19054-21ssh21:34:433
110049950,5cyclictest9573-21ssh22:20:083
110049950,5cyclictest0-21swapper/323:26:013
110049950,5cyclictest0-21swapper/323:00:343
110049950,5cyclictest0-21swapper/300:39:493
110049950,4cyclictest2295-21apt-get19:55:133
110049950,4cyclictest21595-21df_inode20:40:143
109999953,1cyclictest20785-21ls23:00:012
109999950,5cyclictest0-21swapper/223:45:512
109999950,5cyclictest0-21swapper/223:19:482
109999950,5cyclictest0-21swapper/222:46:292
109969951,2cyclictest0-21swapper/119:15:131
109969950,5cyclictest0-21swapper/123:34:471
109969950,5cyclictest0-21swapper/122:57:401
109969950,5cyclictest0-21swapper/121:42:101
109909954,0cyclictest3-21ksoftirqd/022:30:000
109909954,0cyclictest3-21ksoftirqd/022:10:090
109909953,1cyclictest3-21ksoftirqd/021:55:130
109909953,0cyclictest3-21ksoftirqd/023:50:090
109909953,0cyclictest3-21ksoftirqd/022:00:110
109909952,1cyclictest3-21ksoftirqd/020:15:220
109909951,1cyclictest3-21ksoftirqd/021:28:210
109909951,1cyclictest3-21ksoftirqd/020:40:140
109909951,1cyclictest3-21ksoftirqd/000:15:140
109909951,0cyclictest81rcu_preempt22:15:590
109909951,0cyclictest3-21ksoftirqd/020:30:090
109909951,0cyclictest3-21ksoftirqd/020:00:130
109909950,5cyclictest0-21swapper/021:53:420
109909950,1cyclictest3-21ksoftirqd/023:55:050
109909950,1cyclictest3-21ksoftirqd/023:30:000
109909950,1cyclictest3-21ksoftirqd/023:00:110
109909950,1cyclictest3-21ksoftirqd/019:40:150
109909950,1cyclictest26420-21ssh23:36:170
109909950,1cyclictest10985-21cyclictest19:50:210
109909950,1cyclictest0-21swapper/021:35:130
109909950,0cyclictest81rcu_preempt00:30:140
109909950,0cyclictest23456-21apt-get21:20:020
110049944,0cyclictest13573-21kworker/3:121:08:423
110049944,0cyclictest13573-21kworker/3:119:50:193
110049944,0cyclictest0-21swapper/323:10:273
110049944,0cyclictest0-21swapper/322:32:033
110049944,0cyclictest0-21swapper/322:14:353
110049944,0cyclictest0-21swapper/322:07:273
110049944,0cyclictest0-21swapper/321:10:163
110049944,0cyclictest0-21swapper/320:55:353
110049943,1cyclictest24067-21diskstats20:45:143
110049943,1cyclictest19107-21df_inode20:35:133
110049942,1cyclictest13575-21fw_packets19:15:003
110049941,3cyclictest12040-21perl22:55:143
110049940,4cyclictest30993-21ssh21:57:073
110049940,4cyclictest19536-21apt-get21:35:013
110049940,4cyclictest0-21swapper/323:53:113
110049940,4cyclictest0-21swapper/323:20:153
110049940,4cyclictest0-21swapper/323:05:143
110049940,4cyclictest0-21swapper/322:45:383
110049940,4cyclictest0-21swapper/322:42:453
110049940,4cyclictest0-21swapper/322:25:083
110049940,4cyclictest0-21swapper/322:15:123
110049940,4cyclictest0-21swapper/321:45:153
110049940,4cyclictest0-21swapper/320:54:513
110049940,4cyclictest0-21swapper/319:40:133
110049940,4cyclictest0-21swapper/300:33:243
110049940,4cyclictest0-21swapper/300:25:293
110049940,4cyclictest0-21swapper/300:24:263
110049940,4cyclictest0-21swapper/300:19:173
110049940,3cyclictest7021-21apt-get20:05:013
110049940,3cyclictest5415-21ssh22:35:113
110049940,3cyclictest31542-21ls21:40:173
110049940,3cyclictest30188-21ssh23:55:203
110049940,3cyclictest16799-21cat23:15:023
110049940,3cyclictest12828-21ssh23:30:073
110049940,3cyclictest10787-21ssh00:02:103
110049940,3cyclictest0-21swapper/321:15:123
110049940,1cyclictest29998-21cstates19:45:123
110049940,0cyclictest0-21swapper/322:01:123
110049940,0cyclictest0-21swapper/321:26:343
109999944,0cyclictest0-21swapper/222:54:202
109999944,0cyclictest0-21swapper/221:12:292
109999942,1cyclictest30997-21snmpd21:31:112
109999942,1cyclictest30997-21snmpd00:18:572
109999941,2cyclictest0-21swapper/220:05:022
109999940,4cyclictest5417-21apt-get22:35:112
109999940,4cyclictest291rcuc/222:05:002
109999940,4cyclictest13674-21sed23:30:092
109999940,4cyclictest0-21swapper/223:40:102
109999940,4cyclictest0-21swapper/223:12:532
109999940,4cyclictest0-21swapper/222:15:082
109999940,4cyclictest0-21swapper/221:37:202
109999940,4cyclictest0-21swapper/200:10:072
109999940,4cyclictest0-21swapper/200:08:122
109999940,3cyclictest3482-21cut21:10:002
109999940,3cyclictest31262-21cat19:45:192
109999940,3cyclictest28471-21sh23:55:132
109999940,3cyclictest27956-21ssh23:55:012
109999940,3cyclictest25651-21ntp_states21:20:192
109999940,3cyclictest21017-21sh23:00:022
109999940,3cyclictest18272-21sh22:40:492
109999940,3cyclictest14469-21df_inode21:15:132
109999940,3cyclictest14340-21apt19:15:122
109999940,3cyclictest10824-21irqstats22:20:162
109999940,3cyclictest10221-21ssh00:01:492
109999940,3cyclictest0-21swapper/221:50:072
109999940,1cyclictest8061-21/usr/sbin/munin20:10:182
109999940,1cyclictest30302-21/usr/sbin/munin21:40:152
109999940,1cyclictest28850-21apt00:29:592
109999940,1cyclictest28371-21if_eth023:20:182
109999940,1cyclictest27764-21/usr/sbin/munin22:30:102
109999940,0cyclictest0-21swapper/221:46:412
109969944,0cyclictest0-21swapper/123:16:441
109969944,0cyclictest0-21swapper/122:52:081
109969944,0cyclictest0-21swapper/122:46:361
109969944,0cyclictest0-21swapper/121:30:131
109969944,0cyclictest0-21swapper/100:29:381
109969941,2cyclictest0-21swapper/123:20:141
109969941,2cyclictest0-21swapper/123:10:081
109969941,2cyclictest0-21swapper/122:40:121
109969941,2cyclictest0-21swapper/122:10:091
109969941,2cyclictest0-21swapper/121:55:131
109969941,2cyclictest0-21swapper/121:20:131
109969941,2cyclictest0-21swapper/121:00:121
109969941,2cyclictest0-21swapper/120:45:121
109969941,2cyclictest0-21swapper/120:35:121
109969941,2cyclictest0-21swapper/120:30:071
109969941,2cyclictest0-21swapper/120:20:121
109969941,2cyclictest0-21swapper/120:10:111
109969941,2cyclictest0-21swapper/120:00:131
109969941,2cyclictest0-21swapper/119:30:151
109969941,2cyclictest0-21swapper/119:20:131
109969940,4cyclictest0-21swapper/121:10:531
109969940,0cyclictest0-21swapper/121:52:031
109909944,0cyclictest3-21ksoftirqd/021:15:130
109909944,0cyclictest3-21ksoftirqd/020:55:170
109909944,0cyclictest3-21ksoftirqd/020:25:010
109909944,0cyclictest0-21swapper/021:45:590
109909944,0cyclictest0-21swapper/019:45:110
109909943,0cyclictest3-21ksoftirqd/019:35:120
109909943,0cyclictest3-21ksoftirqd/000:35:140
109909942,0cyclictest3-21ksoftirqd/019:20:140
109909941,2cyclictest3-21ksoftirqd/020:53:530
109909941,1cyclictest3-21ksoftirqd/020:30:010
109909940,4cyclictest0-21swapper/021:00:110
109909940,4cyclictest0-21swapper/020:35:110
109909940,1cyclictest20095-21uniq19:25:150
109909940,0cyclictest81rcu_preempt23:45:270
109909940,0cyclictest81rcu_preempt22:40:130
109909940,0cyclictest81rcu_preempt19:55:190
109909940,0cyclictest81rcu_preempt19:10:160
3851230,0sleep10-21swapper/122:34:201
110049932,1cyclictest23728-21ssh00:10:053
110049932,1cyclictest16356-21tr21:50:093
110049932,1cyclictest14775-21grep20:20:213
110049932,1cyclictest10713-21ssh23:45:283
110049931,1cyclictest32657-21ssh23:40:073
110049931,1cyclictest0-21swapper/321:20:143
110049930,3cyclictest5958-21grep20:00:193
110049930,3cyclictest3276-21diskmemload00:05:203
110049930,3cyclictest1189-21ssh22:50:083
110049930,3cyclictest11106-21apt-get19:10:133
110049930,3cyclictest0-21swapper/323:35:113
110049930,3cyclictest0-21swapper/321:00:123
110049930,3cyclictest0-21swapper/320:30:093
110049930,3cyclictest0-21swapper/320:26:013
110049930,3cyclictest0-21swapper/320:15:133
110049930,3cyclictest0-21swapper/320:10:103
110049930,3cyclictest0-21swapper/319:35:123
110049930,3cyclictest0-21swapper/319:30:173
110049930,3cyclictest0-21swapper/319:25:133
110049930,3cyclictest0-21swapper/319:20:013
109999932,1cyclictest30997-21snmpd20:50:152
109999931,2cyclictest0-21swapper/222:26:072
109999931,1cyclictest57250irq/125-eth023:35:012
109999931,1cyclictest23536-21apt-get20:45:122
109999931,1cyclictest0-21swapper/200:35:132
109999930,3cyclictest32173-21egrep19:50:182
109999930,3cyclictest29162-21perl20:55:152
109999930,3cyclictest27642-21perl19:40:162
109999930,3cyclictest26150-21apt-get21:55:132
109999930,3cyclictest2295-21apt-get19:55:122
109999930,3cyclictest11374-21apt-get00:20:122
109999930,3cyclictest0-21swapper/222:00:102
109999930,3cyclictest0-21swapper/221:00:302
109999930,3cyclictest0-21swapper/220:40:122
109999930,3cyclictest0-21swapper/220:30:082
109999930,3cyclictest0-21swapper/220:25:242
109999930,3cyclictest0-21swapper/220:20:122
109999930,3cyclictest0-21swapper/220:15:002
109999930,3cyclictest0-21swapper/220:00:122
109999930,3cyclictest0-21swapper/219:35:192
109999930,3cyclictest0-21swapper/219:25:122
109999930,3cyclictest0-21swapper/219:20:122
109999930,3cyclictest0-21swapper/219:10:132
109999930,3cyclictest0-21swapper/200:30:122
109999930,2cyclictest4311-21cron23:25:022
109999930,1cyclictest31141-21ssh23:05:062
109999930,0cyclictest18585-21apt-get20:35:122
109969933,0cyclictest0-21swapper/123:51:471
109969933,0cyclictest0-21swapper/123:28:131
109969933,0cyclictest0-21swapper/122:16:211
109969931,1cyclictest0-21swapper/123:40:101
109969931,1cyclictest0-21swapper/121:15:121
109969931,1cyclictest0-21swapper/120:50:171
109969931,1cyclictest0-21swapper/119:55:121
109969931,1cyclictest0-21swapper/119:50:181
109969931,1cyclictest0-21swapper/119:45:021
109969931,1cyclictest0-21swapper/119:35:121
109969931,1cyclictest0-21swapper/119:25:121
109969931,1cyclictest0-21swapper/100:35:131
109969931,1cyclictest0-21swapper/100:30:131
109969931,1cyclictest0-21swapper/100:20:111
109969930,3cyclictest0-21swapper/119:40:481
109969930,2cyclictest14388-21chrt21:49:161
109969930,2cyclictest0-21swapper/123:35:141
109969930,2cyclictest0-21swapper/122:35:121
109969930,2cyclictest0-21swapper/122:20:121
109969930,2cyclictest0-21swapper/120:15:141
109969921,1cyclictest0-21swapper/122:05:021
109969921,1cyclictest0-21swapper/100:05:241
109969920,1cyclictest5643-21chrt00:00:041
109969920,1cyclictest21151-21chrt23:00:061
109969920,1cyclictest19714-21chrt22:25:111
109969920,1cyclictest0-21swapper/123:48:401
109969920,1cyclictest0-21swapper/123:10:011
109969920,1cyclictest0-21swapper/121:25:461
109969920,1cyclictest0-21swapper/121:05:141
109969920,1cyclictest0-21swapper/120:55:111
109969920,1cyclictest0-21swapper/120:40:131
109969920,1cyclictest0-21swapper/120:05:011
109969920,1cyclictest0-21swapper/119:10:141
109969910,1cyclictest15487-21chrt20:25:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional