You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-13 - 17:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Mar 13, 2025 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2085017993529,4cyclictest2289136-21cron00:05:002
2085013993523,10cyclictest2296837-21latency_hist00:15:011
2085017993424,8cyclictest0-21swapper/219:19:592
2085013993316,2cyclictest0-21swapper/100:20:011
2085013993315,2cyclictest0-21swapper/122:35:001
2085013993315,16cyclictest0-21swapper/121:40:291
2085013993313,7cyclictest2169542-21tr21:55:131
2085017993218,12cyclictest0-21swapper/222:20:002
2085017993216,2cyclictest0-21swapper/220:25:002
208501799321,18cyclictest2227014-21taskset22:56:112
2085017993116,2cyclictest0-21swapper/219:40:002
2085017993115,2cyclictest0-21swapper/221:30:002
2085013993110,8cyclictest2269818-21seq23:43:151
2085013993110,8cyclictest2269818-21seq23:43:141
2085010993117,10cyclictest2284986-21cut00:00:150
2085017993022,5cyclictest699-21dbus-daemon00:00:012
2085017993020,9cyclictest703-21polkitd22:34:592
2085013993017,11cyclictest2088896-21awk19:19:591
2085013993016,12cyclictest2139843-21threads21:20:241
2085013993015,3cyclictest2093753-21cron19:34:591
2085013993014,2cyclictest0-21swapper/122:45:001
208501399301,16cyclictest2185726-21taskset22:12:051
208501399294,4cyclictest2181205-21sh22:08:131
208501399291,17cyclictest2218372-21sh22:47:541
208501399290,18cyclictest2133933-21sh21:15:091
2085020992822,2cyclictest191rcu_preempt23:44:513
2085020992822,2cyclictest191rcu_preempt23:44:503
2085017992815,2cyclictest703-21polkitd23:20:222
2085017992810,6cyclictest699-21dbus-daemon20:29:592
208501399289,6cyclictest2104687-21grep20:00:191
208501399288,6cyclictest699-21dbus-daemon21:50:181
2085013992823,3cyclictest2234330-21latency_hist23:05:001
208501399281,9cyclictest2151230-21apt21:34:591
208501399281,6cyclictest2144732-21taskset21:27:101
2085013992815,2cyclictest2122934-21turbostat.cron20:55:001
2085013992814,3cyclictest2127259-21cpuspeed_turbos21:05:441
2085013992814,1cyclictest0-21swapper/122:15:151
208501399281,16cyclictest2123839-21head20:55:161
2085013992810,7cyclictest940-21runrttasks23:49:001
2085013992810,6cyclictest2249796-21ssh23:20:381
2085020992712,6cyclictest191rcu_preempt21:40:173
208501799278,8cyclictest2257443-21ntp_192.168.11423:30:122
2085017992715,2cyclictest0-21swapper/221:00:012
208501399279,6cyclictest2243510-21idleruntime23:15:141
2085013992710,5cyclictest262-21systemd-journal23:36:061
208502099263,18cyclictest52-21ksoftirqd/323:08:323
2085020992622,2cyclictest703-21polkitd19:45:013
2085020992621,3cyclictest712-21tpm2-abrmd20:30:013
2085020992621,3cyclictest2266295-21munin-plugin-st23:40:003
208502099260,8cyclictest2130039-21ntpq21:10:203
208501799263,7cyclictest2186154-21sh22:12:432
2085017992616,2cyclictest1-21systemd00:15:022
2085017992615,7cyclictest2158890-21sshd21:41:512
2085017992613,3cyclictest2125536-21gltestperf21:00:132
208501799261,17cyclictest2181426-21ssh22:08:322
2085017992611,6cyclictest2193080-21ssh22:20:082
208501799260,17cyclictest2245365-21ssh23:16:342
2085013992616,8cyclictest0-21swapper/121:15:001
2085013992612,3cyclictest2125306-21cpuspeed_turbos21:00:121
2085013992610,6cyclictest2162738-21sh21:47:211
2085010992622,2cyclictest0-21swapper/023:20:000
2085020992520,1cyclictest191rcu_preempt23:01:013
2085020992516,2cyclictest191rcu_preempt20:40:293
208501799258,7cyclictest2271177-21grep23:45:012
208501799258,7cyclictest2271177-21grep23:45:002
208501799250,17cyclictest2212426-21ssh22:40:202
208501399259,5cyclictest2206950-21ls22:35:171
208501399259,5cyclictest2206950-21ls22:35:161
2085013992513,3cyclictest22821102sleep123:56:481
2085013992512,3cyclictest942-21snmpd22:24:231
208501399251,16cyclictest2100157-21/usr/sbin/munin19:50:151
2085013992510,5cyclictest2109183-21kworker/u16:320:15:181
208501399250,6cyclictest2286251-21ssh00:00:351
208501399250,13cyclictest2311729-21/usr/sbin/munin00:35:281
208501399250,13cyclictest2311729-21/usr/sbin/munin00:35:271
208502099248,10cyclictest52-21ksoftirqd/322:35:003
2085020992419,3cyclictest0-21swapper/323:00:003
2085020992419,1cyclictest191rcu_preempt21:31:153
208502099241,8cyclictest52-21ksoftirqd/323:10:123
208501399244,9cyclictest726-21gdbus20:40:001
208501399240,16cyclictest2236645-21ssh23:08:091
2085020992320,2cyclictest628-21rpcbind22:16:173
2085020992320,2cyclictest0-21swapper/322:40:003
2085020992320,2cyclictest0-21swapper/322:39:593
2085020992313,7cyclictest699-21dbus-daemon23:20:003
208502099231,18cyclictest2282623-21runrttasks23:57:373
208502099230,16cyclictest2134565-21sed21:15:183
2085017992311,5cyclictest2133506-21sshd21:14:502
2085013992319,2cyclictest2116107-21turbostat.cron20:35:001
208501399231,8cyclictest2258400-21ssh23:30:391
2085013992313,3cyclictest331ktimers/122:26:461
208501399231,15cyclictest2128849-21diskmemload22:56:121
208501399231,13cyclictest2089874-21if_enp1s019:20:161
2085010992318,3cyclictest2218327-21sh22:47:500
2085010992312,7cyclictest2311904-21df00:35:160
2085010992312,7cyclictest2311904-21df00:35:150
208502099229,3cyclictest0-21swapper/323:20:223
208502099223,15cyclictest2128849-21diskmemload22:48:283
2085020992217,1cyclictest191rcu_preempt20:10:123
2085020992217,1cyclictest191rcu_preempt19:55:263
2085020992217,1cyclictest191rcu_preempt19:10:163
2085020992216,2cyclictest191rcu_preempt23:25:293
2085020992215,1cyclictest191rcu_preempt00:15:293
208502099221,18cyclictest2183756-21ntpq22:10:133
2085020992211,5cyclictest52-21ksoftirqd/300:21:023
2085020992211,2cyclictest191rcu_preempt20:15:183
208501799221,7cyclictest2235269-21taskset23:06:022
208501799221,18cyclictest2164701-21munin-node21:50:072
2085017992211,8cyclictest2107213-21ntpq20:10:142
2085017992210,8cyclictest2086214-21mailstats19:10:252
2085013992217,3cyclictest2220518-21gltestperf22:50:101
208501399221,11cyclictest940-21runrttasks23:26:101
2085010992213,6cyclictest2138911-21cut21:20:140
208501099221,19cyclictest2249025-21ntpq23:20:200
2085010992211,7cyclictest2203450-21runrttasks22:31:040
208501099220,6cyclictest2098789-21/usr/sbin/munin19:45:200
2085020992118,2cyclictest2296526-21apt-key00:15:003
208502099211,7cyclictest2201245-21grep22:29:593
2085020992115,4cyclictest0-21swapper/319:55:003
2085020992114,2cyclictest191rcu_preempt21:00:143
2085020992112,2cyclictest191rcu_preempt21:47:103
208502099211,17cyclictest2128849-21diskmemload23:32:253
2085020992111,6cyclictest52-21ksoftirqd/321:35:143
2085020992111,4cyclictest52-21ksoftirqd/300:05:003
208502099210,15cyclictest2179771-21taskset22:06:093
2085017992117,2cyclictest0-21swapper/219:30:102
2085017992116,3cyclictest2151911-21fschecks_time21:35:152
208501799211,18cyclictest2162413-21taskset21:46:532
208501799210,18cyclictest2266026-21taskset23:39:352
2085013992118,2cyclictest2305644-21ssh00:26:381
208501399211,7cyclictest2154517-21ssh21:37:401
2085013992117,2cyclictest0-21swapper/100:10:011
208501099211,9cyclictest2138265-21apt21:20:010
2085010992118,2cyclictest262-21systemd-journal00:00:010
2085010992115,4cyclictest0-21swapper/021:35:000
2085010992112,7cyclictest2298560-21ssh00:15:280
208501099210,6cyclictest20913841latency_hist19:25:000
208502099208,9cyclictest2170271-21ntp_states21:55:213
208502099208,2cyclictest0-21swapper/300:30:253
208502099204,8cyclictest52-21ksoftirqd/319:30:203
208502099202,8cyclictest52-21ksoftirqd/322:00:513
208502099201,6cyclictest2090189-21ntpq19:20:193
2085020992015,1cyclictest191rcu_preempt21:26:213
2085020992015,1cyclictest191rcu_preempt21:05:463
2085020992013,2cyclictest191rcu_preempt23:50:073
2085020992013,2cyclictest191rcu_preempt21:55:013
208502099200,13cyclictest2194197-21ntpq22:20:493
208502099200,11cyclictest2124929-21readlink21:00:003
208501799201,8cyclictest2229864-21ssh23:00:052
2085017992016,2cyclictest712-21tpm2-abrmd19:50:012
2085017992016,2cyclictest0-21swapper/220:50:002
2085017992016,2cyclictest0-21swapper/219:40:222
2085017992012,5cyclictest2293984-21ssh00:10:292
2085017992011,7cyclictest2141524-21seq21:22:412
208501799201,17cyclictest2128849-21diskmemload22:49:392
2085017992011,6cyclictest2277611-21sh23:52:092
2085017992010,6cyclictest2090580-21timerandwakeup19:20:252
2085013992010,6cyclictest262-21systemd-journal19:48:011
208501399200,7cyclictest2178563-21ssh22:04:501
208501399200,17cyclictest2276522-21ssh23:50:301
208501099209,8cyclictest262-21systemd-journal23:35:110
208501099208,8cyclictest2155259-21sshd21:38:500
208501099205,5cyclictest2253801-21sh23:26:140
208501099202,8cyclictest940-21runrttasks22:36:060
208501099202,8cyclictest940-21runrttasks22:36:050
2085010992017,2cyclictest2183789-21collect222:10:120
2085010992016,2cyclictest0-21swapper/020:35:000
2085010992015,3cyclictest1-21systemd21:47:510
2085010992015,3cyclictest0-21swapper/020:39:590
2085010992011,8cyclictest2120575-21kworker/u16:3+rpciod21:42:190
208501099201,17cyclictest2274521-21ssh23:49:530
208501099201,17cyclictest2168537-21ssh21:54:120
208501099201,16cyclictest2290189-21ssh00:06:240
2085010992011,6cyclictest2259774-21runrttasks23:32:450
2085010992011,6cyclictest2220110-21/usr/sbin/munin22:50:150
2085010992011,6cyclictest2104581-21ntp_192.168.11420:00:170
2085010992010,7cyclictest2242427-21ssh23:14:290
2085010992010,7cyclictest2233488-21ssh23:03:360
2085010992010,7cyclictest2170524-21sed21:55:250
208501099200,17cyclictest2127947-21ntpq21:05:510
208502099199,6cyclictest726-21gdbus20:35:533
208502099198,7cyclictest191rcu_preempt22:40:163
2085020991915,1cyclictest191rcu_preempt20:06:213
2085020991914,1cyclictest191rcu_preempt19:35:153
2085020991914,1cyclictest191rcu_preempt00:10:013
2085020991913,2cyclictest191rcu_preempt21:20:143
2085020991912,2cyclictest191rcu_preempt23:49:533
2085020991912,2cyclictest191rcu_preempt22:50:083
2085020991911,6cyclictest2104876-21mailstats20:00:223
208502099190,9cyclictest52-21ksoftirqd/300:25:023
208501799199,7cyclictest2174033-21ssh22:00:102
2085017991915,2cyclictest2242738-21sh23:15:002
2085017991915,2cyclictest0-21swapper/220:20:002
2085017991912,5cyclictest2302521-21sh00:21:242
2085017991912,5cyclictest2171178-21ssh21:56:102
2085017991911,5cyclictest2307616-21sshd00:30:032
208501799191,15cyclictest2115251-21/usr/sbin/munin20:30:532
208501799191,10cyclictest2151384-21cat21:35:002
2085017991910,6cyclictest2207974-21sh22:35:332
2085017991910,6cyclictest2207974-21sh22:35:322
208501799190,7cyclictest2311795-21grep00:35:162
208501799190,7cyclictest2311795-21grep00:35:162
2085013991910,7cyclictest2310327-21taskset00:33:051
2085013991910,6cyclictest2302220-21grep00:20:401
208501399190,7cyclictest2239046-21ld23:10:141
208501099198,7cyclictest709-21systemd-logind19:38:520
208501099197,8cyclictest2212006-21awk22:40:150
208501099194,9cyclictest2271138-21grep23:45:010
208501099194,9cyclictest2271138-21grep23:45:000
2085010991914,3cyclictest2087871-21ntp_kernel_err19:15:180
2085010991912,5cyclictest2201424-21latency_hist22:30:010
208501099191,16cyclictest2132348-21ssh21:13:030
2085010991910,7cyclictest2194896-21ssh22:21:030
2085010991910,6cyclictest2093593-21sshd19:33:010
208501099190,17cyclictest2225932-21ntp_states22:55:200
208502099182,8cyclictest52-21ksoftirqd/320:25:003
2085020991814,2cyclictest0-21swapper/319:50:013
2085020991813,3cyclictest2114567-21cpuspeed_turbos20:30:463
2085020991813,1cyclictest191rcu_preempt19:15:143
208502099180,15cyclictest2121001-21awk20:50:003
208501799189,6cyclictest262-21systemd-journal20:52:382
208501799187,8cyclictest2127194-21apt-get21:05:442
2085017991815,2cyclictest488-21hwrng19:59:522
2085017991814,2cyclictest0-21swapper/200:10:012
208501799180,6cyclictest2305756-21ssh00:26:522
208501799180,5cyclictest2136958-21sh21:17:512
208501799180,5cyclictest2105105-21grep20:00:242
2085013991813,3cyclictest0-21swapper/120:45:001
208501099189,7cyclictest2175694-21sh22:00:410
208501099189,6cyclictest2178986-21ssh22:05:040
208501099188,7cyclictest2107288-21ntpq20:10:170
208501099186,8cyclictest262-21systemd-journal20:17:110
208501099186,2cyclictest191rcu_preempt00:22:370
2085010991815,2cyclictest3690845-21kworker/0:1+events_freezable00:27:500
2085010991814,2cyclictest0-21swapper/020:55:000
2085010991813,3cyclictest2191298-21taskset22:17:560
2085010991812,4cyclictest2235089-21sh23:05:460
208501099181,15cyclictest2128849-21diskmemload23:50:140
2085010991811,5cyclictest2128849-21diskmemload00:12:150
208501099180,16cyclictest2143412-21sh21:25:070
2085020991714,2cyclictest488-21hwrng20:52:063
2085020991712,1cyclictest191rcu_preempt19:30:003
208501799179,6cyclictest2106081-21sshd20:08:012
208501799171,5cyclictest989-21cron19:29:592
2085017991713,3cyclictest421ktimers/223:46:022
208501799171,15cyclictest2224016-21ssh22:54:052
208501399179,6cyclictest262-21systemd-journal19:40:001
2085013991714,2cyclictest488-21hwrng19:26:051
2085013991714,2cyclictest488-21hwrng19:11:451
2085013991713,2cyclictest2107440-21/usr/sbin/munin20:10:181
208501099178,5cyclictest2112481-21apt20:25:430
208501099177,7cyclictest2102071-21df19:55:140
2085010991715,1cyclictest488-21hwrng00:33:170
2085010991713,3cyclictest3690845-21kworker/0:1+events_freezable21:01:570
2085010991713,3cyclictest2084974-21kworker/u16:1-ext4-rsv-conversion19:44:540
2085010991713,2cyclictest0-21swapper/019:55:000
208501099171,13cyclictest2111686-21unixbench-2d20:20:250
2085010991710,5cyclictest2119684-21/usr/sbin/munin20:40:390
208501799169,5cyclictest0-21swapper/220:35:542
2085017991614,1cyclictest488-21hwrng19:51:402
208501799161,12cyclictest2198175-21ssh22:25:312
2085017991610,4cyclictest2128849-21diskmemload23:26:212
208501799160,14cyclictest2119285-21/usr/sbin/munin20:40:362
208501399167,7cyclictest2096464-21irqstats19:40:151
2085013991614,1cyclictest488-21hwrng20:45:571
2085013991614,1cyclictest488-21hwrng20:29:331
2085013991613,2cyclictest2103613-21cron20:00:011
208501399161,11cyclictest940-21runrttasks20:20:131
208501099167,6cyclictest2120571-21dpkg20:45:030
208501099166,6cyclictest722-21gdbus19:10:230
2085010991612,2cyclictest2123920-21ntpq20:55:180
208501399156,6cyclictest2105898-21seq20:05:421
208501099157,5cyclictest699-21dbus-daemon19:30:000
208501099131,8cyclictest940-21runrttasks20:06:490
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional