You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 09:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot2.osadl.org (updated Fri Nov 22, 2024 00:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
94342197181,11sleep10-21swapper/119:08:161
94032190173,12sleep00-21swapper/019:07:500
92832179163,11sleep30-21swapper/319:06:223
95372165145,15sleep20-21swapper/219:09:372
204752480,0sleep30-21swapper/320:35:003
209672460,0sleep10-21swapper/121:50:021
209672460,0sleep10-21swapper/121:50:011
120972460,0sleep10-21swapper/123:55:281
222362440,0sleep00-21swapper/022:06:010
97419970,5cyclictest0-21swapper/000:00:100
97419970,3cyclictest404-21perf21:00:000
97499961,4cyclictest15760-21apt-get20:25:093
97449966,0cyclictest31-21ksoftirqd/200:35:012
97449965,1cyclictest31-21ksoftirqd/222:00:012
97449965,1cyclictest301ktimersoftd/221:05:012
97449965,0cyclictest31-21ksoftirqd/222:40:112
97419960,5cyclictest0-21swapper/023:05:110
97419960,3cyclictest5372-21diskmemload00:10:000
97499955,0cyclictest12334-21kworker/3:023:44:293
97499955,0cyclictest12334-21kworker/3:023:33:183
97499955,0cyclictest12334-21kworker/3:022:59:433
97499955,0cyclictest12334-21kworker/3:021:30:103
97499955,0cyclictest12334-21kworker/3:021:18:583
97499955,0cyclictest12334-21kworker/3:000:18:053
97499955,0cyclictest0-21swapper/323:52:003
97499950,5cyclictest16388-21ssh22:35:273
97499950,5cyclictest0-21swapper/323:03:233
97499950,5cyclictest0-21swapper/300:40:003
97499950,4cyclictest0-21swapper/323:10:093
97449955,0cyclictest31-21ksoftirqd/223:20:002
97449955,0cyclictest31-21ksoftirqd/221:45:162
97449955,0cyclictest31-21ksoftirqd/221:45:162
97449955,0cyclictest31-21ksoftirqd/219:25:012
97449955,0cyclictest31-21ksoftirqd/200:35:142
97449955,0cyclictest31-21ksoftirqd/200:20:002
97449955,0cyclictest301ktimersoftd/220:40:002
97449955,0cyclictest0-21swapper/223:59:232
97449954,1cyclictest31-21ksoftirqd/223:35:142
97449954,1cyclictest31-21ksoftirqd/222:55:112
97449954,1cyclictest31-21ksoftirqd/222:00:132
97449954,1cyclictest31-21ksoftirqd/200:10:182
97449953,2cyclictest31-21ksoftirqd/223:13:392
97449951,4cyclictest0-21swapper/221:38:242
97449950,5cyclictest6017-21ssh23:03:212
97449950,5cyclictest0-21swapper/223:06:092
97449950,5cyclictest0-21swapper/221:26:012
97449950,5cyclictest0-21swapper/200:03:202
97449950,4cyclictest30835-21apt-get19:50:122
97449950,4cyclictest15760-21apt-get20:25:092
97449950,4cyclictest0-21swapper/200:25:132
97449950,3cyclictest81rcu_preempt19:25:012
97449950,0cyclictest0-21swapper/200:07:152
97429951,2cyclictest0-21swapper/100:00:101
97429950,5cyclictest0-21swapper/122:43:371
97419955,0cyclictest0-21swapper/023:16:230
97419951,3cyclictest8023-21grep23:20:180
97419951,3cyclictest4334-21fschecks_count21:25:150
97419950,5cyclictest0-21swapper/023:46:580
97419950,5cyclictest0-21swapper/023:43:010
97419950,4cyclictest5372-21diskmemload21:20:120
97419950,4cyclictest21259-21df_inode19:30:120
97419950,4cyclictest19219-21irqstats20:30:160
97419950,4cyclictest0-21swapper/022:35:110
97419950,4cyclictest0-21swapper/000:20:120
97419950,1cyclictest0-21swapper/021:38:460
97419950,0cyclictest5372-21diskmemload23:10:080
97419950,0cyclictest5372-21diskmemload22:21:160
97419950,0cyclictest23414-21timerandwakeup22:55:180
97419950,0cyclictest0-21swapper/023:35:130
24114250,4sleep024139-21ssh21:50:420
97499944,0cyclictest12334-21kworker/3:023:22:073
97499944,0cyclictest12334-21kworker/3:022:26:083
97499944,0cyclictest12334-21kworker/3:020:22:583
97499944,0cyclictest0-21swapper/321:48:013
97499944,0cyclictest0-21swapper/321:48:013
97499941,2cyclictest23207-21aten2_r5power_c19:35:113
97499940,4cyclictest6160-21ssh23:35:203
97499940,4cyclictest61-21kswapd022:05:133
97499940,4cyclictest30594-21apt-get20:55:083
97499940,4cyclictest13099-21apt-get19:15:123
97499940,4cyclictest0-21swapper/322:50:153
97499940,4cyclictest0-21swapper/322:47:203
97499940,4cyclictest0-21swapper/322:40:073
97499940,4cyclictest0-21swapper/322:20:123
97499940,4cyclictest0-21swapper/321:40:373
97499940,4cyclictest0-21swapper/321:27:173
97499940,4cyclictest0-21swapper/321:11:303
97499940,4cyclictest0-21swapper/321:00:113
97499940,4cyclictest0-21swapper/320:40:113
97499940,4cyclictest0-21swapper/319:25:003
97499940,4cyclictest0-21swapper/300:10:173
97499940,4cyclictest0-21swapper/300:00:503
97499940,3cyclictest8776-21cat22:00:023
97499940,3cyclictest6651-21/usr/sbin/munin22:15:113
97499940,3cyclictest660-21sh00:06:103
97499940,3cyclictest3661-21diskstats21:05:153
97499940,3cyclictest30997-21snmpd00:26:153
97499940,3cyclictest30348-21missed_timers22:10:183
97499940,3cyclictest29502-21/usr/sbin/munin19:45:193
97499940,3cyclictest27128-21grep00:20:133
97499940,3cyclictest23071-21date20:40:013
97499940,3cyclictest21180-21apt-config21:50:013
97499940,3cyclictest18443-21ssh00:31:293
97499940,3cyclictest1390-21df19:55:113
97499940,3cyclictest13828-21ssh23:06:583
97499940,3cyclictest13039-21latency_hist20:20:003
97499940,3cyclictest12293-21grep19:15:013
97499940,3cyclictest11023-21memory23:55:163
97499940,3cyclictest10169-21modprobe22:00:163
97499940,3cyclictest0-21swapper/323:15:183
97499940,3cyclictest0-21swapper/320:10:123
97499940,2cyclictest0-21swapper/321:20:333
97499940,1cyclictest25654-21/usr/sbin/munin20:45:113
97499940,1cyclictest23599-21/usr/sbin/munin21:35:133
97499940,0cyclictest0-21swapper/322:30:483
97449944,0cyclictest31-21ksoftirqd/223:20:122
97449944,0cyclictest31-21ksoftirqd/221:50:022
97449944,0cyclictest31-21ksoftirqd/221:20:152
97449944,0cyclictest31-21ksoftirqd/220:55:002
97449944,0cyclictest31-21ksoftirqd/220:05:102
97449944,0cyclictest31-21ksoftirqd/219:50:012
97449944,0cyclictest31-21ksoftirqd/219:30:102
97449944,0cyclictest31-21ksoftirqd/200:20:142
97449944,0cyclictest0-21swapper/222:45:112
97449944,0cyclictest0-21swapper/221:10:202
97449943,1cyclictest31-21ksoftirqd/222:35:152
97449943,1cyclictest31-21ksoftirqd/219:15:002
97449943,1cyclictest0-21swapper/219:40:142
97449943,0cyclictest31-21ksoftirqd/223:40:172
97449943,0cyclictest31-21ksoftirqd/222:20:172
97449943,0cyclictest31-21ksoftirqd/220:30:112
97449942,2cyclictest31-21ksoftirqd/222:30:112
97449942,1cyclictest31-21ksoftirqd/222:15:112
97449942,1cyclictest30997-21snmpd22:05:092
97449940,4cyclictest32319-21cpuspeed_turbos23:50:092
97449940,4cyclictest301ktimersoftd/219:15:122
97449940,4cyclictest13778-21apt-get21:30:162
97449940,4cyclictest0-21swapper/222:50:142
97449940,4cyclictest0-21swapper/222:25:102
97449940,4cyclictest0-21swapper/221:15:132
97449940,4cyclictest0-21swapper/220:15:132
97449940,4cyclictest0-21swapper/220:00:132
97449940,3cyclictest5248-21ssh21:41:342
97449940,3cyclictest432-21ssh22:11:282
97449940,3cyclictest3196-21cut21:05:122
97449940,3cyclictest24234-21ssh23:45:192
97449940,3cyclictest23405-21expr19:35:112
97449940,3cyclictest14809-21python20:20:192
97449940,3cyclictest0-21swapper/220:10:122
97449940,2cyclictest0-21swapper/223:30:132
97449940,1cyclictest16840-21tr23:25:112
97449940,0cyclictest0-21swapper/219:55:122
97429944,0cyclictest0-21swapper/123:49:221
97429944,0cyclictest0-21swapper/123:34:231
97429944,0cyclictest0-21swapper/121:43:481
97429944,0cyclictest0-21swapper/100:20:041
97429941,2cyclictest0-21swapper/122:50:141
97429941,2cyclictest0-21swapper/100:30:141
97429941,2cyclictest0-21swapper/100:15:111
97429941,1cyclictest0-21swapper/123:35:111
97429940,4cyclictest0-21swapper/123:28:071
97429940,4cyclictest0-21swapper/121:21:161
97429940,4cyclictest0-21swapper/119:14:561
97429940,0cyclictest0-21swapper/122:48:151
97429940,0cyclictest0-21swapper/121:51:221
97429940,0cyclictest0-21swapper/121:39:171
97429940,0cyclictest0-21swapper/121:11:301
97419944,0cyclictest0-21swapper/000:30:090
97419942,2cyclictest14564-21ssh21:46:060
97419942,2cyclictest14564-21ssh21:46:060
97419942,1cyclictest19087-21head19:25:140
97419941,3cyclictest30997-21snmpd22:15:350
97419941,3cyclictest0-21swapper/020:50:140
97419940,4cyclictest5372-21diskmemload00:35:010
97419940,4cyclictest3483-21aten2_r5power_v20:00:130
97419940,4cyclictest19247-21munin-run00:00:000
97419940,4cyclictest0-21swapper/023:50:090
97419940,4cyclictest0-21swapper/023:30:150
97419940,4cyclictest0-21swapper/022:52:470
97419940,4cyclictest0-21swapper/022:45:120
97419940,4cyclictest0-21swapper/022:30:150
97419940,4cyclictest0-21swapper/021:15:210
97419940,4cyclictest0-21swapper/021:14:140
97419940,4cyclictest0-21swapper/021:00:120
97419940,3cyclictest8421-21irqstats00:10:210
97419940,3cyclictest8313-21apt20:10:120
97419940,3cyclictest5372-21diskmemload22:25:050
97419940,3cyclictest4907-21rm22:14:160
97419940,3cyclictest4280-21ssh00:25:000
97419940,3cyclictest32181-21perl21:55:130
97419940,3cyclictest31233-21expr19:50:120
97419940,3cyclictest30997-21snmpd00:15:040
97419940,3cyclictest24912-21perl22:40:140
97419940,3cyclictest24265-21sh23:28:510
97419940,3cyclictest16328-21fschecks_time19:20:130
97419940,3cyclictest15190-21ssh21:30:230
97419940,3cyclictest12454-21netstat19:15:000
97419940,3cyclictest0-21swapper/019:40:140
8523240,0sleep10-21swapper/123:20:211
97499932,1cyclictest30997-21snmpd19:30:043
97499931,1cyclictest0-21swapper/320:00:133
97499931,0cyclictest0-21swapper/323:25:113
97499930,3cyclictest28658-21diskstats20:50:143
97499930,3cyclictest0-21swapper/323:45:283
97499930,3cyclictest0-21swapper/320:05:023
97499930,3cyclictest0-21swapper/319:50:073
97499930,3cyclictest0-21swapper/319:40:083
97499930,3cyclictest0-21swapper/319:20:113
97449932,1cyclictest30594-21apt-get20:55:082
97449930,3cyclictest0-21swapper/220:45:112
97449930,3cyclictest0-21swapper/220:40:102
97429933,0cyclictest0-21swapper/123:17:561
97429933,0cyclictest0-21swapper/100:35:041
97429931,1cyclictest0-21swapper/123:40:121
97429931,1cyclictest0-21swapper/123:10:101
97429931,1cyclictest0-21swapper/123:05:121
97429931,1cyclictest0-21swapper/123:00:101
97429931,1cyclictest0-21swapper/122:35:111
97429931,1cyclictest0-21swapper/122:15:121
97429931,1cyclictest0-21swapper/122:10:131
97429931,1cyclictest0-21swapper/121:15:121
97429931,1cyclictest0-21swapper/121:05:121
97429931,1cyclictest0-21swapper/120:35:101
97429931,1cyclictest0-21swapper/120:15:121
97429931,1cyclictest0-21swapper/120:00:131
97429931,1cyclictest0-21swapper/119:55:111
97429931,1cyclictest0-21swapper/119:45:121
97429931,1cyclictest0-21swapper/119:30:111
97429931,1cyclictest0-21swapper/119:25:021
97429931,1cyclictest0-21swapper/100:05:121
97429930,3cyclictest0-21swapper/120:58:331
97429930,3cyclictest0-21swapper/119:53:031
97429930,2cyclictest7833-21taskset22:31:201
97429930,2cyclictest6924-21chrt00:10:021
97429930,2cyclictest27154-21chrt19:40:221
97429930,2cyclictest21653-21chrt22:22:121
97429930,2cyclictest0-21swapper/122:55:091
97429930,2cyclictest0-21swapper/120:40:101
97429930,2cyclictest0-21swapper/120:05:101
97429930,2cyclictest0-21swapper/119:15:121
97429930,2cyclictest0-21swapper/100:25:111
97419932,1cyclictest30997-21snmpd19:35:080
97419930,3cyclictest8958-21ssh22:00:060
97419930,3cyclictest31356-21ssh23:00:010
97419930,3cyclictest0-21swapper/021:40:080
97419930,3cyclictest0-21swapper/021:05:120
97419930,3cyclictest0-21swapper/020:45:100
97419930,3cyclictest0-21swapper/020:35:100
97419930,3cyclictest0-21swapper/020:25:090
97419930,3cyclictest0-21swapper/020:20:110
97419930,3cyclictest0-21swapper/020:15:120
97419930,3cyclictest0-21swapper/020:05:110
97419930,3cyclictest0-21swapper/019:55:100
97419930,3cyclictest0-21swapper/019:45:040
97419930,3cyclictest0-21swapper/019:10:110
448230,0sleep10-21swapper/123:50:111
23184231,1sleep041ktimersoftd/020:40:100
18403230,0sleep10-21swapper/122:04:541
97429921,1cyclictest132852chrt20:20:101
97429921,1cyclictest0-21swapper/122:05:131
97429921,1cyclictest0-21swapper/121:30:161
97429921,1cyclictest0-21swapper/121:00:111
97429921,1cyclictest0-21swapper/120:45:101
97429921,0cyclictest0-21swapper/120:10:121
97429920,1cyclictest28115-21chrt20:50:041
97429920,1cyclictest18220-21chrt20:30:051
97429920,1cyclictest0-21swapper/122:25:101
97429920,1cyclictest0-21swapper/121:55:101
97429920,1cyclictest0-21swapper/121:25:141
97429920,1cyclictest0-21swapper/120:25:101
97429920,1cyclictest0-21swapper/119:40:001
97429920,1cyclictest0-21swapper/119:20:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional