You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 02:48
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackdslot2.osadl.org (updated Thu Nov 21, 2024 12:45:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
170662235170,22sleep10-21swapper/107:06:221
170932231163,22sleep30-21swapper/307:06:443
170422204170,22sleep20-21swapper/207:06:032
171652202170,22sleep00-21swapper/007:07:410
2480621520,2sleep20-21swapper/209:44:592
1733828161,11sleep20-21swapper/207:10:012
1733828161,11sleep20-21swapper/207:10:012
1762999716,16cyclictest5716-21cstates11:20:140
17629997010,18cyclictest19371-21tune2fs10:40:160
1763099670,25cyclictest23333-21ps11:55:231
17632996637,3cyclictest41-21ksoftirqd/308:50:013
1762999669,11cyclictest26380-21sed07:25:270
17629996611,11cyclictest17811-21grep10:35:220
17631996522,6cyclictest33-21ksoftirqd/207:50:132
1763099650,24cyclictest31475-21users09:55:291
1763099650,24cyclictest19742-21sed10:40:191
1762999649,11cyclictest4304-21grep08:55:230
17631996325,6cyclictest33-21ksoftirqd/210:30:182
17631996324,6cyclictest33-21ksoftirqd/212:25:242
1763099630,35cyclictest3839-21/usr/sbin/munin10:05:261
1763099630,25cyclictest13479-21sed10:25:271
1763099630,24cyclictest12819-21cstates09:15:151
1763099630,23cyclictest23953-21df_abs10:50:131
1762999639,11cyclictest447-21sed08:50:010
1762999638,12cyclictest11319-21ls11:30:220
1762999638,11cyclictest6606-21latency_hist07:55:010
17629996310,11cyclictest11619-21tr08:05:150
17632996240,3cyclictest41-21ksoftirqd/312:30:293
17632996230,4cyclictest41-21ksoftirqd/307:35:213
17632996229,4cyclictest41-21ksoftirqd/309:20:193
1762999629,16cyclictest9490-21awk10:20:130
17629996211,11cyclictest9586-21tune2fs08:00:190
17629996211,11cyclictest3436-21/usr/sbin/munin10:05:210
17632996130,9cyclictest41-21ksoftirqd/308:35:233
17632996130,7cyclictest41-21ksoftirqd/311:05:173
17632996130,4cyclictest41-21ksoftirqd/307:20:193
17632996129,9cyclictest41-21ksoftirqd/312:25:173
17632996127,4cyclictest41-21ksoftirqd/311:35:233
17631996120,6cyclictest33-21ksoftirqd/211:10:272
1763099610,30cyclictest6386-21date12:30:001
1763099610,22cyclictest15000-21/usr/sbin/munin11:40:221
1763099610,18cyclictest0-21swapper/107:45:141
1762999619,11cyclictest22024-21sort08:25:190
17632996030,4cyclictest41-21ksoftirqd/310:45:003
17632996030,4cyclictest41-21ksoftirqd/309:05:003
17632996029,4cyclictest41-21ksoftirqd/308:10:133
17632996029,4cyclictest41-21ksoftirqd/308:10:133
17632996028,4cyclictest41-21ksoftirqd/310:30:013
17632996027,4cyclictest41-21ksoftirqd/311:45:143
17631996017,6cyclictest33-21ksoftirqd/207:25:232
17631996016,5cyclictest33-21ksoftirqd/212:35:162
1763099601,21cyclictest0-21swapper/108:00:301
1763099600,29cyclictest2743-21latency12:20:211
1763099600,25cyclictest12415-21cat09:15:001
1762999609,12cyclictest25758-21grep08:35:140
17632995930,4cyclictest41-21ksoftirqd/311:25:263
17632995929,4cyclictest41-21ksoftirqd/309:30:153
17632995927,4cyclictest41-21ksoftirqd/310:30:183
17632995927,11cyclictest41-21ksoftirqd/310:10:193
17632995926,4cyclictest41-21ksoftirqd/309:45:243
17632995925,7cyclictest41-21ksoftirqd/308:30:183
17631995918,6cyclictest33-21ksoftirqd/211:35:002
17631995918,6cyclictest33-21ksoftirqd/209:15:152
17631995918,6cyclictest33-21ksoftirqd/208:55:142
17631995917,6cyclictest33-21ksoftirqd/208:10:232
17631995917,6cyclictest33-21ksoftirqd/208:10:232
17631995916,8cyclictest33-21ksoftirqd/209:30:142
1763099590,26cyclictest742-21/usr/sbin/munin07:40:241
1763099590,26cyclictest742-21/usr/sbin/munin07:40:241
1762999599,16cyclictest29449-21/usr/sbin/munin07:35:150
1762999599,11cyclictest6811-21sh10:14:590
1762999599,11cyclictest20749-21fschecks_time09:35:150
1762999598,11cyclictest24088-21irqrtprio08:30:190
17629995911,12cyclictest15348-21/usr/sbin/munin08:10:250
17629995911,12cyclictest15348-21/usr/sbin/munin08:10:250
17629995910,11cyclictest20846-21irqrtprio07:15:180
17632995830,4cyclictest41-21ksoftirqd/311:30:213
17632995829,9cyclictest41-21ksoftirqd/312:10:303
17632995828,9cyclictest41-21ksoftirqd/307:40:313
17632995828,9cyclictest41-21ksoftirqd/307:40:313
17632995828,7cyclictest41-21ksoftirqd/311:25:013
17632995828,4cyclictest41-21ksoftirqd/308:10:003
17632995828,4cyclictest41-21ksoftirqd/307:25:213
17632995827,10cyclictest41-21ksoftirqd/312:05:233
17632995825,8cyclictest41-21ksoftirqd/309:10:203
17632995824,3cyclictest41-21ksoftirqd/311:40:133
17631995830,4cyclictest33-21ksoftirqd/210:45:012
17631995826,5cyclictest33-21ksoftirqd/207:45:002
17631995826,5cyclictest33-21ksoftirqd/207:45:002
17631995816,7cyclictest33-21ksoftirqd/209:35:162
1763099584,28cyclictest613-21gmain11:25:011
1763099580,25cyclictest29336-21cut07:35:001
1763099580,24cyclictest17447-21sed10:35:181
1763099580,23cyclictest31980-21cstates12:15:151
1762999589,11cyclictest21848-21wc11:55:010
1762999588,12cyclictest25063-21mailstats10:50:240
17629995812,18cyclictest12349-21fw_packets10:25:140
17632995729,4cyclictest41-21ksoftirqd/308:40:013
17632995726,5cyclictest41-21ksoftirqd/307:15:243
17632995726,4cyclictest41-21ksoftirqd/311:10:303
17632995726,10cyclictest41-21ksoftirqd/308:25:253
17632995725,4cyclictest41-21ksoftirqd/309:55:243
17632995722,10cyclictest41-21ksoftirqd/308:15:213
17631995727,4cyclictest33-21ksoftirqd/208:45:012
17631995719,6cyclictest33-21ksoftirqd/209:55:142
17631995717,6cyclictest33-21ksoftirqd/211:00:102
17631995717,5cyclictest33-21ksoftirqd/212:10:202
17631995716,6cyclictest33-21ksoftirqd/211:05:232
1763099570,23cyclictest6927-21cat10:15:011
1762999579,11cyclictest25139-21sort09:45:120
1762999578,11cyclictest6986-21cstates07:55:130
17629995711,18cyclictest11181-21/usr/sbin/munin09:10:180
17632995631,3cyclictest41-21ksoftirqd/307:45:013
17632995630,7cyclictest41-21ksoftirqd/308:20:203
17632995628,4cyclictest41-21ksoftirqd/307:55:253
17632995627,4cyclictest41-21ksoftirqd/311:00:233
17632995627,4cyclictest41-21ksoftirqd/309:35:143
17632995626,9cyclictest41-21ksoftirqd/309:40:253
17631995621,4cyclictest33-21ksoftirqd/208:15:212
17631995616,6cyclictest33-21ksoftirqd/209:20:222
17631995616,5cyclictest33-21ksoftirqd/209:00:172
17632995530,4cyclictest41-21ksoftirqd/308:00:183
17632995527,8cyclictest41-21ksoftirqd/309:50:283
17632995523,8cyclictest41-21ksoftirqd/312:15:163
17631995523,5cyclictest33-21ksoftirqd/211:40:272
17631995513,6cyclictest33-21ksoftirqd/212:00:122
17629995510,17cyclictest19702-21taskset08:20:240
17629995510,10cyclictest9054-21seq12:35:040
17632995427,4cyclictest41-21ksoftirqd/310:20:203
17632995425,4cyclictest41-21ksoftirqd/309:05:153
17632995421,9cyclictest41-21ksoftirqd/310:45:193
17632995421,4cyclictest41-21ksoftirqd/311:50:253
17631995421,4cyclictest33-21ksoftirqd/209:45:132
17631995419,6cyclictest33-21ksoftirqd/208:25:222
17631995412,7cyclictest33-21ksoftirqd/210:45:142
17631995410,7cyclictest33-21ksoftirqd/208:45:162
1762999549,16cyclictest19740-21taskset09:31:060
17632995328,4cyclictest41-21ksoftirqd/309:15:153
17632995322,4cyclictest41-21ksoftirqd/310:05:153
17632995321,8cyclictest41-21ksoftirqd/310:50:223
1763299530,6cyclictest748-21polkitd11:20:013
1763199539,6cyclictest33-21ksoftirqd/211:35:242
1763099530,22cyclictest16078-21latency_hist08:14:591
1763099530,22cyclictest16078-21latency_hist08:14:591
17632995223,10cyclictest41-21ksoftirqd/307:30:133
17632995219,5cyclictest41-21ksoftirqd/312:20:213
1763199527,9cyclictest33-21ksoftirqd/211:20:142
17631995223,4cyclictest33-21ksoftirqd/211:55:242
17631995221,3cyclictest33-21ksoftirqd/208:30:262
17631995217,6cyclictest33-21ksoftirqd/208:50:252
17631995210,6cyclictest33-21ksoftirqd/208:05:272
1762999529,11cyclictest8578-21sed10:15:300
1762999529,10cyclictest29226-21cut11:00:170
17632995126,4cyclictest41-21ksoftirqd/312:35:013
17632995122,9cyclictest41-21ksoftirqd/310:35:263
17632995122,4cyclictest41-21ksoftirqd/311:55:273
17632995120,8cyclictest41-21ksoftirqd/308:55:133
17631995119,4cyclictest33-21ksoftirqd/212:30:182
17631995116,7cyclictest33-21ksoftirqd/211:15:172
17631995111,6cyclictest33-21ksoftirqd/210:00:242
1763299500,5cyclictest41-21ksoftirqd/310:05:003
17631995022,6cyclictest33-21ksoftirqd/212:15:192
17631995022,5cyclictest33-21ksoftirqd/210:55:162
17631995010,6cyclictest33-21ksoftirqd/210:15:162
1763099501,24cyclictest6772-21gzip09:00:231
1762999507,16cyclictest18049-21diskstats07:10:150
1762999507,16cyclictest18049-21diskstats07:10:150
1763199493,6cyclictest321ktimersoftd/208:20:322
1763199492,5cyclictest321ktimersoftd/211:50:312
17631994911,8cyclictest33-21ksoftirqd/209:50:292
1762999499,11cyclictest15225-21cstates11:40:140
1762999498,12cyclictest1918-21apt-key07:45:010
1762999498,12cyclictest1698-21cut12:20:010
1763299486,3cyclictest41-21ksoftirqd/310:59:563
1763199485,7cyclictest33-21ksoftirqd/210:35:212
17631994822,6cyclictest33-21ksoftirqd/207:35:172
17631994818,6cyclictest33-21ksoftirqd/207:15:222
17631994818,5cyclictest33-21ksoftirqd/207:55:162
17631994817,5cyclictest33-21ksoftirqd/209:15:002
17631994814,6cyclictest33-21ksoftirqd/210:10:232
17631994813,6cyclictest33-21ksoftirqd/207:30:192
1762999489,11cyclictest738-21grep11:10:190
1762999489,11cyclictest5364-21cut11:20:000
1762999489,11cyclictest27433-21awk09:50:160
17629994810,12cyclictest12988-21grep09:15:160
17629994810,10cyclictest18199-21taskset08:17:160
17632994735,3cyclictest41-21ksoftirqd/308:50:273
1763199473,5cyclictest321ktimersoftd/212:05:252
17631994718,7cyclictest33-21ksoftirqd/208:35:172
17631994714,4cyclictest33-21ksoftirqd/210:05:132
1763199470,6cyclictest321ktimersoftd/210:50:152
1762999479,11cyclictest6425-21cat12:30:010
1762999479,11cyclictest29230-21cut08:40:180
17629994710,11cyclictest15573-21grep10:30:230
1763299464,3cyclictest41-21ksoftirqd/310:15:183
1763199462,5cyclictest33-21ksoftirqd/210:25:182
1762999469,9cyclictest23461-21sed07:20:210
1762999469,11cyclictest28624-21timerandwakeup07:30:260
17629994611,9cyclictest1718-21sed07:45:000
17629994611,9cyclictest1718-21sed07:45:000
17631994530,4cyclictest33-21ksoftirqd/209:09:592
17631994529,5cyclictest33-21ksoftirqd/210:20:202
1763199450,7cyclictest321ktimersoftd/212:20:252
1762999459,7cyclictest23914-21sshd11:56:160
1762999458,11cyclictest30855-21switchtime12:10:290
1762999458,11cyclictest26792-21cat12:05:000
1762999458,11cyclictest10148-21cat11:30:000
1763299443,3cyclictest121rcu_preempt07:10:223
1763299443,3cyclictest121rcu_preempt07:10:223
1763299441,8cyclictest41-21ksoftirqd/312:00:193
1763199440,5cyclictest321ktimersoftd/211:25:152
1762999449,9cyclictest30816-21missed_timers09:55:210
1762999449,12cyclictest13499-21sed11:35:210
1762999449,11cyclictest6956-21tune2fs12:30:210
1762999449,11cyclictest23043-21/usr/sbin/munin10:45:270
1762999449,11cyclictest22711-21egrep09:40:110
1762999448,11cyclictest9206-21mailstats09:05:230
17631994340,2cyclictest311rcuc/207:20:232
1763199432,8cyclictest33-21ksoftirqd/211:45:142
1763099430,25cyclictest10531-21/usr/sbin/munin12:35:231
133291430,0ptp4l401ktimersoftd/309:25:453
1763099421,24cyclictest30178-21irqrtprio12:10:231
17629994210,11cyclictest31562-21irqrtprio11:05:200
17632994111,6cyclictest41-21ksoftirqd/307:50:173
1763099412,27cyclictest20666-1kworker/1:2H09:45:011
1763099411,37cyclictest27854-21cron08:39:591
1763099411,26cyclictest4476-21/usr/sbin/munin07:50:121
1762999419,11cyclictest26610-21awk10:55:170
17631994036,2cyclictest321ktimersoftd/209:29:592
1763199401,4cyclictest321ktimersoftd/207:45:172
1762999409,11cyclictest3912-21latency_hist12:25:010
1762999409,10cyclictest2312-21timerwakeupswit08:50:280
17631993923,4cyclictest33-21ksoftirqd/208:00:172
17630993631,3cyclictest25-21ksoftirqd/112:05:111
1763099360,18cyclictest0-21swapper/108:25:141
1763099360,17cyclictest0-21swapper/109:05:321
1762999367,11cyclictest0-21swapper/009:20:120
17630993530,3cyclictest25-21ksoftirqd/107:40:001
17630993529,4cyclictest25-21ksoftirqd/111:31:531
17630993514,18cyclictest31911-21nfs4_client11:05:241
1763099351,20cyclictest0-21swapper/111:55:001
1763099350,22cyclictest23541-21netstat07:20:221
1762999356,9cyclictest0-21swapper/009:00:260
17630993429,3cyclictest25-21ksoftirqd/108:57:021
17630993428,4cyclictest25-21ksoftirqd/110:48:021
17630993428,4cyclictest25-21ksoftirqd/108:07:591
17630993427,4cyclictest25-21ksoftirqd/107:10:111
17630993427,4cyclictest25-21ksoftirqd/107:10:111
1763099341,29cyclictest28997-21grep08:40:181
1763099340,19cyclictest0-21swapper/110:15:181
1763099340,18cyclictest0-21swapper/111:10:181
1762999347,16cyclictest0-21swapper/012:05:200
17630993327,4cyclictest25-21ksoftirqd/111:03:021
17630993327,4cyclictest25-21ksoftirqd/108:53:531
17630993327,4cyclictest25-21ksoftirqd/108:30:011
17630993327,4cyclictest25-21ksoftirqd/107:57:521
1763099331,19cyclictest0-21swapper/109:35:001
17630993311,19cyclictest26873-21latency_hist07:30:001
1763099330,20cyclictest0-21swapper/107:15:121
1763099330,17cyclictest0-21swapper/109:45:291
1762999338,8cyclictest32189-21cpuspeed10:00:120
17629993328,3cyclictest9-21ksoftirqd/011:45:370
17629993310,10cyclictest17374-21taskset09:26:050
1763099329,20cyclictest17092-21cut08:15:201
17630993226,4cyclictest25-21ksoftirqd/110:04:331
17630993226,4cyclictest25-21ksoftirqd/109:20:561
17630993225,5cyclictest25-21ksoftirqd/110:57:121
17630993210,19cyclictest18468-21cat11:45:211
17630993210,19cyclictest14346-21cpuspeed_turbos10:30:131
1763099320,22cyclictest0-21swapper/109:50:301
1763099319,19cyclictest6683-21cpuspeed_turbos12:30:171
17630993126,3cyclictest25-21ksoftirqd/111:18:221
17630993125,4cyclictest25-21ksoftirqd/109:26:501
17630993125,4cyclictest25-21ksoftirqd/108:23:071
1763099307,21cyclictest468-21dbus-daemon09:35:251
1763099306,21cyclictest30892-21seq08:45:131
1763099297,20cyclictest9180-21ptp4l-jitter11:25:231
1763099291,24cyclictest25752-21/usr/sbin/munin12:00:241
1763099281,24cyclictest13209-21awk11:35:181
1763099281,23cyclictest9593-21cut10:20:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional