You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:41
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Thu Apr 03, 2025 00:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
164252204189,10sleep10-21swapper/119:09:321
161762194177,12sleep30-21swapper/319:06:213
140002194178,11sleep00-21swapper/019:05:020
142342182165,13sleep20-21swapper/219:05:042
93592810,0sleep361-21kswapd023:06:213
325422540,0sleep10-21swapper/122:10:151
142402470,0sleep00-21swapper/000:34:340
299822420,0sleep30-21swapper/323:34:353
256352170,0sleep10-21swapper/122:23:271
1663499115,5cyclictest30899-21ssh22:10:080
166349993,3cyclictest17196-21munin-run20:15:010
166349991,4cyclictest7419-21munin-run21:00:000
166399976,0cyclictest31-21ksoftirqd/200:10:082
166349971,3cyclictest19547-21basename22:55:010
166349970,4cyclictest27379-21munin-run19:30:010
12657270,0sleep10-21swapper/123:25:001
166449960,6cyclictest0-21swapper/320:10:143
166449960,5cyclictest5154-21apt-get20:55:013
166449960,1cyclictest0-21swapper/323:17:273
166399960,5cyclictest2485-21apt-get20:50:002
166399960,3cyclictest12930-21/usr/sbin/munin22:00:242
166349960,3cyclictest12312-21sendmail-msp22:00:010
8282250,4sleep00-21swapper/022:31:310
16808250,0sleep10-21swapper/122:53:171
166449955,0cyclictest0-21swapper/322:51:213
166449954,1cyclictest7297-21munin-run19:55:013
166449954,1cyclictest26191-21sh23:50:003
166449950,5cyclictest2357-21latency_hist19:45:013
166449950,5cyclictest0-21swapper/323:12:153
166449950,5cyclictest0-21swapper/322:49:453
166449950,5cyclictest0-21swapper/300:30:183
166449950,1cyclictest0-21swapper/323:36:113
166399954,1cyclictest0-21swapper/200:16:312
166399951,3cyclictest0-21swapper/223:15:102
166399950,0cyclictest0-21swapper/222:52:422
166359951,2cyclictest0-21swapper/123:15:101
166359950,5cyclictest0-21swapper/123:57:301
166359950,5cyclictest0-21swapper/100:11:191
166349950,5cyclictest0-21swapper/023:15:090
166349950,5cyclictest0-21swapper/022:36:440
24493240,0sleep10-21swapper/121:50:011
166449944,0cyclictest0-21swapper/322:29:583
166449944,0cyclictest0-21swapper/322:12:393
166449943,1cyclictest10045-21ssh22:15:193
166449942,1cyclictest30997-21snmpd23:27:203
166449941,3cyclictest0-21swapper/320:19:593
166449941,2cyclictest0-21swapper/319:55:103
166449940,4cyclictest1946-21users00:10:173
166449940,4cyclictest16448-21ssh22:01:193
166449940,4cyclictest13525-21df_inode21:10:163
166449940,4cyclictest12365-21diskmemload23:50:073
166449940,4cyclictest0-21swapper/323:55:023
166449940,4cyclictest0-21swapper/323:41:193
166449940,4cyclictest0-21swapper/323:21:153
166449940,4cyclictest0-21swapper/322:40:133
166449940,4cyclictest0-21swapper/322:35:123
166449940,4cyclictest0-21swapper/322:20:163
166449940,4cyclictest0-21swapper/321:55:113
166449940,4cyclictest0-21swapper/321:50:083
166449940,4cyclictest0-21swapper/321:47:413
166449940,4cyclictest0-21swapper/321:30:123
166449940,4cyclictest0-21swapper/321:27:123
166449940,4cyclictest0-21swapper/321:22:213
166449940,4cyclictest0-21swapper/319:21:263
166449940,4cyclictest0-21swapper/319:15:273
166449940,4cyclictest0-21swapper/300:20:123
166449940,4cyclictest0-21swapper/300:15:353
166449940,3cyclictest4741-21latency_hist20:55:003
166449940,3cyclictest28441-21hddtemp_smartct19:30:173
166449940,3cyclictest25668-21diskstats19:25:143
166449940,3cyclictest25076-21cat00:40:003
166449940,3cyclictest24037-21rm00:05:003
166449940,3cyclictest19114-21munin-node19:15:013
166449940,3cyclictest14115-21ssh00:00:013
166449940,0cyclictest0-21swapper/322:33:543
166449940,0cyclictest0-21swapper/321:39:203
166399944,0cyclictest31-21ksoftirqd/221:10:012
166399944,0cyclictest0-21swapper/223:45:492
166399944,0cyclictest0-21swapper/223:29:392
166399944,0cyclictest0-21swapper/223:11:532
166399944,0cyclictest0-21swapper/222:55:542
166399944,0cyclictest0-21swapper/222:30:262
166399944,0cyclictest0-21swapper/222:15:262
166399944,0cyclictest0-21swapper/221:35:092
166399943,1cyclictest32351-21memory23:35:152
166399943,1cyclictest31996-21irqstats22:10:122
166399942,1cyclictest15207-21expr22:35:142
166399941,3cyclictest26567-21sh22:24:032
166399941,1cyclictest0-21swapper/219:55:092
166399940,4cyclictest30068-21apt-get19:35:132
166399940,4cyclictest25836-21perl21:50:102
166399940,4cyclictest19265-21ssh00:02:072
166399940,4cyclictest12365-21diskmemload00:35:132
166399940,4cyclictest0-21swapper/223:30:112
166399940,4cyclictest0-21swapper/223:20:122
166399940,4cyclictest0-21swapper/222:43:492
166399940,4cyclictest0-21swapper/222:09:172
166399940,4cyclictest0-21swapper/221:55:082
166399940,4cyclictest0-21swapper/221:23:222
166399940,4cyclictest0-21swapper/221:10:162
166399940,4cyclictest0-21swapper/220:45:052
166399940,4cyclictest0-21swapper/220:25:122
166399940,4cyclictest0-21swapper/220:15:132
166399940,4cyclictest0-21swapper/219:15:152
166399940,4cyclictest0-21swapper/200:30:062
166399940,4cyclictest0-21swapper/200:07:252
166399940,3cyclictest8711-21/usr/sbin/munin23:40:182
166399940,3cyclictest8226-21sh22:49:122
166399940,3cyclictest4782-21grep23:55:142
166399940,3cyclictest30207-21sh00:25:252
166399940,3cyclictest30083-21ssh23:51:102
166399940,3cyclictest24669-21sed21:50:002
166399940,3cyclictest15941-21irqstats20:10:182
166399940,1cyclictest30082-21/usr/sbin/munin23:00:222
166359944,0cyclictest0-21swapper/122:58:261
166359944,0cyclictest0-21swapper/122:36:451
166359941,2cyclictest0-21swapper/121:55:101
166359941,2cyclictest0-21swapper/120:15:131
166359941,2cyclictest0-21swapper/119:35:121
166359941,2cyclictest0-21swapper/119:15:151
166359941,2cyclictest0-21swapper/119:10:131
166359941,1cyclictest0-21swapper/121:35:131
166359940,4cyclictest0-21swapper/123:53:261
166359940,4cyclictest0-21swapper/123:40:291
166359940,4cyclictest0-21swapper/122:49:331
166359940,4cyclictest0-21swapper/121:41:021
166359940,2cyclictest0-21swapper/120:10:141
166359940,0cyclictest0-21swapper/122:05:111
166359940,0cyclictest0-21swapper/100:33:091
166349944,0cyclictest0-21swapper/023:00:450
166349944,0cyclictest0-21swapper/022:28:530
166349944,0cyclictest0-21swapper/022:19:250
166349944,0cyclictest0-21swapper/000:10:090
166349941,3cyclictest32195-21cat23:35:150
166349941,2cyclictest0-21swapper/020:30:150
166349941,1cyclictest0-21swapper/023:55:130
166349940,4cyclictest8759-21ssh23:05:580
166349940,4cyclictest30997-21snmpd23:40:080
166349940,4cyclictest21184-21ntp_states20:20:180
166349940,4cyclictest0-21swapper/023:52:440
166349940,4cyclictest0-21swapper/023:34:200
166349940,4cyclictest0-21swapper/023:12:040
166349940,4cyclictest0-21swapper/022:55:110
166349940,4cyclictest0-21swapper/021:40:140
166349940,4cyclictest0-21swapper/021:34:480
166349940,4cyclictest0-21swapper/021:17:300
166349940,4cyclictest0-21swapper/000:28:370
166349940,4cyclictest0-21swapper/000:15:120
166349940,4cyclictest0-21swapper/000:09:100
166349940,3cyclictest8109-21ssh23:22:090
166349940,3cyclictest24614-21munin-run20:30:000
166349940,3cyclictest2359-21apt-config20:50:000
166349940,3cyclictest19898-21ssh23:46:000
166349940,3cyclictest19150-21cat19:15:000
166349940,3cyclictest16575-21ssh22:01:240
166349940,3cyclictest16501-21rm21:45:010
166349940,3cyclictest14408-21rm23:25:590
166349940,3cyclictest0-21swapper/019:50:150
166349940,1cyclictest29562-21cut21:35:130
166349940,0cyclictest0-21swapper/022:45:280
166349940,0cyclictest0-21swapper/022:24:570
26228230,0sleep10-21swapper/100:06:251
17188230,0sleep10-21swapper/121:29:061
166449932,1cyclictest30997-21snmpd20:40:043
166449932,1cyclictest30997-21snmpd19:35:033
166449930,3cyclictest7656-21apt21:00:113
166449930,3cyclictest321-21df_inode20:45:153
166449930,3cyclictest22540-21awk20:25:133
166449930,3cyclictest16342-21kworker/3:022:55:083
166449930,3cyclictest12365-21diskmemload21:40:023
166449930,3cyclictest0-21swapper/323:00:013
166449930,3cyclictest0-21swapper/322:05:253
166449930,3cyclictest0-21swapper/321:15:013
166449930,3cyclictest0-21swapper/321:05:133
166449930,3cyclictest0-21swapper/320:35:123
166449930,3cyclictest0-21swapper/320:30:103
166449930,3cyclictest0-21swapper/320:05:133
166449930,3cyclictest0-21swapper/320:00:113
166449930,3cyclictest0-21swapper/319:45:123
166449930,3cyclictest0-21swapper/300:25:103
166449930,2cyclictest0-21swapper/320:20:073
166399932,1cyclictest29792-21/usr/sbin/munin20:40:212
166399932,1cyclictest17674-21awk19:10:172
166399931,1cyclictest27606-21apt-get19:30:142
166399931,1cyclictest19912-21apt-get20:20:122
166399931,1cyclictest0-21swapper/221:00:112
166399930,3cyclictest7598-21sh23:05:142
166399930,3cyclictest6592-21/usr/sbin/munin19:50:232
166399930,3cyclictest5171-21apt20:55:122
166399930,3cyclictest0-21swapper/221:30:142
166399930,3cyclictest0-21swapper/221:15:012
166399930,3cyclictest0-21swapper/220:35:122
166399930,3cyclictest0-21swapper/220:30:152
166399930,3cyclictest0-21swapper/220:05:132
166399930,3cyclictest0-21swapper/220:00:112
166399930,3cyclictest0-21swapper/219:45:132
166399930,3cyclictest0-21swapper/219:40:122
166399930,3cyclictest0-21swapper/219:25:122
166399930,3cyclictest0-21swapper/219:20:012
166399930,3cyclictest0-21swapper/200:20:122
166399930,2cyclictest29255-21ssh22:25:362
166399930,1cyclictest6508-21ssh21:40:022
166399930,1cyclictest0-21swapper/221:25:202
166359931,2cyclictest0-21swapper/119:45:131
166359931,1cyclictest0-21swapper/123:30:111
166359931,1cyclictest0-21swapper/123:00:151
166359931,1cyclictest0-21swapper/122:40:131
166359931,1cyclictest0-21swapper/121:50:021
166359931,1cyclictest0-21swapper/121:30:121
166359931,1cyclictest0-21swapper/121:05:131
166359931,1cyclictest0-21swapper/120:45:131
166359931,1cyclictest0-21swapper/120:35:121
166359931,1cyclictest0-21swapper/120:25:121
166359931,1cyclictest0-21swapper/120:20:121
166359931,1cyclictest0-21swapper/120:05:121
166359931,1cyclictest0-21swapper/119:55:101
166359931,1cyclictest0-21swapper/119:30:141
166359931,1cyclictest0-21swapper/119:25:131
166359931,1cyclictest0-21swapper/119:20:021
166359931,1cyclictest0-21swapper/100:35:131
166359931,1cyclictest0-21swapper/100:20:121
166359930,2cyclictest4947-21chrt00:29:591
166359930,2cyclictest4309-21chrt23:37:341
166359930,2cyclictest3937-21chrt21:20:581
166359930,2cyclictest19977-21chrt23:46:031
166359930,2cyclictest16524-21chrt23:27:211
166359930,2cyclictest13881-21chrt00:16:331
166349932,1cyclictest23767-21ssh22:05:460
166349932,1cyclictest17504-21wget20:15:130
166349931,1cyclictest8521-21cut19:55:140
166349931,1cyclictest0-21swapper/021:05:120
166349931,1cyclictest0-21swapper/020:05:130
166349931,1cyclictest0-21swapper/019:35:130
166349930,3cyclictest3515-21if_err_eth419:45:150
166349930,3cyclictest19751-21if_err_eth019:15:000
166349930,3cyclictest19686-21ssh00:20:030
166349930,3cyclictest15331-21ssh00:35:060
166349930,3cyclictest0-21swapper/022:40:030
166349930,3cyclictest0-21swapper/021:50:020
166349930,3cyclictest0-21swapper/021:25:240
166349930,3cyclictest0-21swapper/021:20:060
166349930,3cyclictest0-21swapper/021:10:160
166349930,3cyclictest0-21swapper/021:00:110
166349930,3cyclictest0-21swapper/020:45:220
166349930,3cyclictest0-21swapper/020:40:120
166349930,3cyclictest0-21swapper/020:35:070
166349930,3cyclictest0-21swapper/020:00:150
166349930,3cyclictest0-21swapper/019:40:110
166349930,3cyclictest0-21swapper/019:30:140
166349930,3cyclictest0-21swapper/019:20:050
166349930,3cyclictest0-21swapper/000:00:130
13234230,0sleep10-21swapper/122:00:181
166359922,0cyclictest0-21swapper/121:10:161
166359921,1cyclictest78402chrt21:00:121
166359921,1cyclictest289382chrt22:25:241
166359921,1cyclictest0-21swapper/123:10:071
166359921,0cyclictest0-21swapper/122:30:081
166359920,2cyclictest0-21swapper/100:00:141
166359920,1cyclictest0-21swapper/123:06:431
166359920,1cyclictest0-21swapper/122:15:131
166359920,1cyclictest0-21swapper/121:15:021
166359920,1cyclictest0-21swapper/120:55:011
166359920,1cyclictest0-21swapper/120:50:011
166359920,1cyclictest0-21swapper/120:40:131
166359920,1cyclictest0-21swapper/120:30:141
166359920,1cyclictest0-21swapper/120:00:111
166359920,1cyclictest0-21swapper/119:50:151
166359920,1cyclictest0-21swapper/119:45:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional