You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Thu Apr 03, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22702862840,0sleep50-21swapper/506:59:205
188838326049,7sleep50-21swapper/503:35:375
188862925140,8sleep60-21swapper/603:39:076
188841824619,7sleep40-21swapper/403:36:084
18876862461,38sleep70-21swapper/703:34:437
23522152440,0sleep72352214-21head07:44:337
188893499440,42cyclictest1934029-21latency_hist03:59:077
1888934993433,0cyclictest0-21swapper/706:04:107
1888934993128,2cyclictest2407776-21cat08:24:207
188893499300,28cyclictest1995903-21latency_hist04:29:087
1888934992928,0cyclictest0-21swapper/704:09:027
188893499280,26cyclictest2057764-21latency_hist04:59:077
1888934992723,3cyclictest2090605-21sed05:14:267
188893499270,25cyclictest2078380-21latency_hist05:09:077
1888927992624,2cyclictest1836336-21kworker/u16:4+flush-8:003:40:134
1888934992423,0cyclictest0-21swapper/703:44:417
1888934992420,3cyclictest2041235-21grep04:49:277
1888930992422,2cyclictest2269174-21kworker/u16:3+flush-8:008:14:076
1888930992321,2cyclictest1898185-21kworker/u16:0+flush-8:004:14:256
20207682220,0chrt0-21swapper/404:39:294
1888934992221,0cyclictest0-21swapper/706:59:177
1888934992220,1cyclictest2140353-21ssh05:39:077
188893499220,20cyclictest2449115-21latency_hist08:49:077
188893499220,20cyclictest1923725-21latency_hist03:54:087
1888930992220,2cyclictest2025544-21kworker/u16:0+flush-8:004:49:386
1888934992120,0cyclictest0-21swapper/708:29:257
1888934992120,0cyclictest0-21swapper/708:04:207
1888934992120,0cyclictest0-21swapper/705:09:377
1888934992119,1cyclictest2301169-21/usr/sbin/munin07:19:317
1888934992119,1cyclictest2301169-21/usr/sbin/munin07:19:317
1888934992118,2cyclictest2458467-21nfs4_client08:49:327
1888930992118,2cyclictest2069565-21kworker/u16:2+events_unbound05:39:086
1888934992019,0cyclictest0-21swapper/706:09:097
1888934992017,2cyclictest1943427-21ntpq03:59:377
188893499200,18cyclictest2006222-21latency_hist04:34:087
188893499200,18cyclictest1983058-21grep04:19:337
1888930992018,2cyclictest1880012-21kworker/u16:3+flush-8:003:41:086
1888934991918,0cyclictest0-21swapper/708:24:027
1888934991918,0cyclictest0-21swapper/706:44:367
188893499180,16cyclictest2293075-21kthreadcore07:09:277
1888934991713,2cyclictest1964003-21ntpq04:09:337
1888934991713,2cyclictest1964003-21ntpq04:09:337
188893499170,15cyclictest2470826-21turbostat09:04:067
1888934991613,2cyclictest1975311-21idleruntime-cro04:19:077
1888930991614,2cyclictest2069565-21kworker/u16:2+flush-8:005:46:396
1888930991614,2cyclictest2069565-21kworker/u16:2+flush-8:005:46:396
188893099160,15cyclictest0-21swapper/606:59:186
1888927991615,1cyclictest0-21swapper/408:14:314
1888927991615,1cyclictest0-21swapper/406:06:314
1888927991615,1cyclictest0-21swapper/405:32:444
1888927991615,1cyclictest0-21swapper/405:09:114
1888927991615,1cyclictest0-21swapper/405:08:094
1888927991615,1cyclictest0-21swapper/404:36:254
1888927991615,1cyclictest0-21swapper/403:55:274
1888934991513,1cyclictest2214737-21awk06:24:077
188893499150,13cyclictest2018125-21sed04:39:277
1888930991514,1cyclictest2036157-21kworker/u16:5+events_unbound04:54:326
1888930991513,2cyclictest1898186-21kworker/u16:2+flush-8:003:44:256
1888930991512,2cyclictest2205114-21kworker/u16:0+flush-8:007:04:216
188893099150,14cyclictest0-21swapper/603:49:116
188892999150,1cyclictest0-21swapper/506:10:025
1888927991514,1cyclictest0-21swapper/405:58:204
1888927991514,1cyclictest0-21swapper/403:46:144
1888927991512,3cyclictest2320781-21kworker/u16:0+flush-8:007:59:324
188893499140,13cyclictest2037125-21turbostat04:49:077
188893499140,12cyclictest2163335-21tr05:49:277
188893099140,1cyclictest0-21swapper/605:09:396
188893099140,1cyclictest0-21swapper/604:24:396
1888929991412,2cyclictest0-21swapper/507:44:375
188892999140,12cyclictest0-21swapper/505:47:195
188892999140,12cyclictest0-21swapper/505:47:195
188892799140,14cyclictest0-21swapper/404:24:354
1888934991312,0cyclictest0-21swapper/708:34:387
1888934991310,2cyclictest2319079-21cut07:24:287
188893499130,4cyclictest2437858-21ntp_states08:39:307
188893499130,12cyclictest1313-21snmpd03:42:187
188893499130,11cyclictest2238377-21kthreadcore06:39:217
188893499130,11cyclictest1313-21snmpd08:16:227
1888930991312,1cyclictest0-21swapper/608:39:256
188893099130,13cyclictest0-21swapper/604:06:506
188893099130,12cyclictest0-21swapper/608:55:396
188893099130,12cyclictest0-21swapper/608:55:396
188893099130,12cyclictest0-21swapper/606:16:266
1888929991312,1cyclictest0-21swapper/508:37:165
1888929991312,1cyclictest0-21swapper/508:31:175
1888929991312,1cyclictest0-21swapper/505:29:215
1888929991311,2cyclictest0-21swapper/507:22:235
1888929991311,2cyclictest0-21swapper/507:22:235
1888929991311,2cyclictest0-21swapper/506:58:355
1888929991311,2cyclictest0-21swapper/505:23:225
1888929991311,2cyclictest0-21swapper/504:24:345
188892999130,13cyclictest0-21swapper/508:47:315
188892999130,13cyclictest0-21swapper/507:41:095
188892999130,12cyclictest0-21swapper/505:25:065
1888927991312,1cyclictest0-21swapper/408:49:164
1888927991312,1cyclictest0-21swapper/408:24:454
1888927991312,1cyclictest0-21swapper/404:54:324
1888927991312,1cyclictest0-21swapper/404:52:484
188892799130,1cyclictest0-21swapper/407:01:174
188892799130,1cyclictest0-21swapper/404:29:374
188893499129,2cyclictest2267379-21basename06:54:077
188893499128,3cyclictest27430-21gmain05:39:327
188893499126,5cyclictest27430-21gmain05:03:487
1888934991212,0cyclictest0-21swapper/705:44:377
1888934991212,0cyclictest0-21swapper/705:44:377
188893499120,2cyclictest2353817-21/usr/sbin/munin07:49:337
188893499120,11cyclictest2322664-21sed07:29:277
188893499120,10cyclictest2009933-21cut04:34:287
1888930991211,1cyclictest0-21swapper/608:37:416
1888930991211,1cyclictest0-21swapper/607:58:066
1888930991211,1cyclictest0-21swapper/607:15:526
1888930991211,1cyclictest0-21swapper/606:57:256
1888930991211,1cyclictest0-21swapper/606:39:176
1888930991211,1cyclictest0-21swapper/605:35:366
188893099120,1cyclictest0-21swapper/608:19:426
188893099120,12cyclictest0-21swapper/608:47:556
1888929991211,1cyclictest0-21swapper/508:21:495
1888929991211,1cyclictest0-21swapper/507:49:495
1888929991210,2cyclictest0-21swapper/508:10:375
1888929991210,2cyclictest0-21swapper/507:36:535
188892999120,1cyclictest0-21swapper/503:51:275
188892999120,12cyclictest0-21swapper/508:00:135
188892999120,12cyclictest0-21swapper/506:24:035
188892999120,12cyclictest0-21swapper/505:13:545
188892999120,11cyclictest0-21swapper/504:39:515
1888927991211,1cyclictest0-21swapper/405:28:074
188892799120,1cyclictest0-21swapper/406:34:104
188893499117,3cyclictest27904-21gmain07:39:167
1888934991110,0cyclictest0-21swapper/708:55:177
1888934991110,0cyclictest0-21swapper/708:55:177
1888934991110,0cyclictest0-21swapper/706:34:597
188893499110,9cyclictest2215501-21awk06:24:187
188893499110,3cyclictest2386710-21sed08:09:247
188893499110,3cyclictest2225749-21/usr/sbin/munin06:29:427
188893499110,2cyclictest2367631-21grep07:59:277
188893499110,2cyclictest2279470-21/usr/sbin/munin07:04:367
188893499110,2cyclictest2181775-21ntp_states05:59:347
188893499110,2cyclictest2120373-21/usr/sbin/munin05:29:217
188893499110,10cyclictest2472363-21cat09:04:247
188893499110,10cyclictest1313-21snmpd07:56:427
188893499110,10cyclictest1313-21snmpd06:16:247
1888930991110,1cyclictest0-21swapper/607:28:096
1888930991110,1cyclictest0-21swapper/605:22:316
1888930991110,1cyclictest0-21swapper/605:16:316
1888930991110,1cyclictest0-21swapper/603:54:526
188893099110,1cyclictest0-21swapper/608:05:036
188893099110,1cyclictest0-21swapper/605:49:576
188893099110,1cyclictest0-21swapper/604:37:166
188893099110,11cyclictest0-21swapper/607:33:126
188893099110,11cyclictest0-21swapper/606:13:356
188893099110,11cyclictest0-21swapper/606:00:116
188893099110,11cyclictest0-21swapper/605:26:466
188892999119,2cyclictest0-21swapper/504:47:445
1888929991110,1cyclictest0-21swapper/509:02:585
1888929991110,1cyclictest0-21swapper/508:50:595
1888929991110,1cyclictest0-21swapper/508:05:255
1888929991110,1cyclictest0-21swapper/506:38:345
1888929991110,1cyclictest0-21swapper/505:53:385
1888929991110,1cyclictest0-21swapper/505:38:205
1888929991110,1cyclictest0-21swapper/504:36:235
1888929991110,1cyclictest0-21swapper/504:20:185
188892999110,11cyclictest0-21swapper/507:25:045
188892999110,11cyclictest0-21swapper/506:41:145
188892999110,11cyclictest0-21swapper/506:03:155
1888927991110,1cyclictest0-21swapper/409:08:494
1888927991110,1cyclictest0-21swapper/408:43:164
1888927991110,1cyclictest0-21swapper/408:04:164
1888927991110,1cyclictest0-21swapper/407:49:224
1888927991110,1cyclictest0-21swapper/407:34:514
1888927991110,1cyclictest0-21swapper/406:39:314
1888927991110,1cyclictest0-21swapper/405:59:384
1888927991110,1cyclictest0-21swapper/405:47:394
1888927991110,1cyclictest0-21swapper/405:47:394
1888927991110,1cyclictest0-21swapper/404:44:364
1888927991110,1cyclictest0-21swapper/404:19:324
1888927991110,1cyclictest0-21swapper/404:03:394
188892799110,1cyclictest0-21swapper/407:55:214
188892799110,1cyclictest0-21swapper/406:22:204
188892799110,11cyclictest0-21swapper/407:45:534
188892799110,11cyclictest0-21swapper/407:29:494
188892799110,11cyclictest0-21swapper/405:53:014
188893499109,0cyclictest0-21swapper/707:15:307
188893499100,9cyclictest2110901-21iostat_ios05:24:247
188893499100,9cyclictest1313-21snmpd07:38:337
188893499100,3cyclictest2172102-21sshd-session05:54:387
188893499100,2cyclictest2104352-21cut05:19:317
188893099109,1cyclictest0-21swapper/608:51:236
188893099109,1cyclictest0-21swapper/608:03:196
188893099109,1cyclictest0-21swapper/607:19:296
188893099109,1cyclictest0-21swapper/607:19:296
188893099109,1cyclictest0-21swapper/606:36:366
188893099108,2cyclictest2472380-21kworker/u16:1+flush-8:009:04:396
188893099108,2cyclictest2224081-21kworker/u16:4+flush-8:006:49:336
188893099108,2cyclictest2069565-21kworker/u16:2+flush-8:007:09:256
188892999109,1cyclictest0-21swapper/507:17:025
188892999109,1cyclictest0-21swapper/507:09:465
188892999109,1cyclictest0-21swapper/505:41:585
188892999109,1cyclictest0-21swapper/504:59:055
188892999109,1cyclictest0-21swapper/504:52:565
188892999109,1cyclictest0-21swapper/504:09:075
188892999106,1cyclictest0-21swapper/508:56:585
188892999106,1cyclictest0-21swapper/508:56:585
188892999105,1cyclictest0-21swapper/503:39:275
188892999103,3cyclictest1906360-21grep03:44:295
188892999100,9cyclictest0-21swapper/506:51:475
188892999100,6cyclictest0-21swapper/505:06:025
188892799109,1cyclictest0-21swapper/408:45:014
188892799109,1cyclictest0-21swapper/408:39:014
188892799109,1cyclictest0-21swapper/407:07:444
188892799109,1cyclictest0-21swapper/406:12:064
18889349998,0cyclictest0-21swapper/706:55:007
18889309998,1cyclictest0-21swapper/608:27:266
18889309998,1cyclictest0-21swapper/607:34:566
18889299998,1cyclictest0-21swapper/507:55:015
18889299998,1cyclictest0-21swapper/506:33:125
18889299998,1cyclictest0-21swapper/506:25:575
18889279998,1cyclictest0-21swapper/408:32:144
18889279998,1cyclictest0-21swapper/408:19:284
18889279998,1cyclictest0-21swapper/408:10:014
18889279998,1cyclictest0-21swapper/407:27:174
18889279998,1cyclictest0-21swapper/405:40:344
18889279998,1cyclictest0-21swapper/405:17:044
18889279998,1cyclictest0-21swapper/404:15:364
18889279998,1cyclictest0-21swapper/404:11:504
18889279998,1cyclictest0-21swapper/404:11:504
18889279996,3cyclictest2202309-21kworker/u16:1+flush-8:006:30:384
18889309987,1cyclictest0-21swapper/608:29:106
18889309987,1cyclictest0-21swapper/606:26:406
18889309987,1cyclictest0-21swapper/604:41:416
18889309987,1cyclictest0-21swapper/604:31:266
18889309985,1cyclictest201rcu_preempt09:02:376
18889309980,1cyclictest0-21swapper/606:19:336
18889299987,1cyclictest0-21swapper/509:06:265
18889299987,1cyclictest0-21swapper/508:43:155
18889299987,1cyclictest0-21swapper/508:25:175
18889279987,1cyclictest0-21swapper/406:45:404
18889279980,7cyclictest0-21swapper/407:18:304
18889309976,1cyclictest0-21swapper/605:57:126
18889309975,2cyclictest2224081-21kworker/u16:4+flush-8:006:44:256
18889309972,1cyclictest681rcuog/404:12:326
18889309972,1cyclictest681rcuog/404:12:326
18889309972,1cyclictest0-21swapper/604:59:446
18889309970,4cyclictest0-21swapper/605:31:306
18889309970,4cyclictest0-21swapper/604:44:376
18889299976,1cyclictest0-21swapper/506:14:455
18889299976,1cyclictest0-21swapper/504:31:205
18889279976,1cyclictest0-21swapper/407:43:484
18889279976,1cyclictest0-21swapper/406:24:144
18889279976,1cyclictest0-21swapper/405:00:414
18889309964,2cyclictest2320781-21kworker/u16:0+flush-8:008:19:076
18889309963,2cyclictest22350542sleep606:30:306
18889309960,3cyclictest0-21swapper/607:52:286
18889309960,3cyclictest0-21swapper/605:04:416
18889309960,3cyclictest0-21swapper/604:00:386
18889309960,2cyclictest0-21swapper/607:47:376
18889309960,1cyclictest0-21swapper/607:41:416
18889299965,1cyclictest0-21swapper/505:54:355
18889299960,1cyclictest0-21swapper/508:14:315
18889299960,1cyclictest0-21swapper/507:31:455
18889299960,1cyclictest0-21swapper/507:08:325
18889299960,1cyclictest0-21swapper/506:44:145
18889299960,1cyclictest0-21swapper/506:04:175
18889299960,1cyclictest0-21swapper/505:17:025
18889299960,1cyclictest0-21swapper/504:59:155
18889299960,1cyclictest0-21swapper/504:11:155
18889299960,1cyclictest0-21swapper/504:11:155
18889299960,1cyclictest0-21swapper/504:04:075
18889299960,1cyclictest0-21swapper/503:55:075
18889279965,1cyclictest0-21swapper/408:59:314
18889279965,1cyclictest0-21swapper/406:14:564
18889279965,1cyclictest0-21swapper/405:35:124
18889279964,1cyclictest22672862sleep406:53:184
18889279962,2cyclictest0-21swapper/407:22:094
18889279962,2cyclictest0-21swapper/407:22:094
18889309953,2cyclictest2141072-21kworker/u16:0+flush-8:006:04:256
18889309950,4cyclictest1977322-21chrt04:19:286
18889299950,4cyclictest1974828-21chrt04:14:395
18889279950,5cyclictest0-21swapper/408:54:154
18889279950,5cyclictest0-21swapper/408:54:154
18889279950,4cyclictest2299029-21chrt07:09:454
18889279950,4cyclictest2268145-21taskset06:55:024
18889279950,4cyclictest2099781-21chrt05:19:204
18889279950,4cyclictest1923425-21chrt03:50:024
18889279950,1cyclictest0-21swapper/404:05:004
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional