You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-21 - 08:44
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8.osadl.org (updated Mon Jan 20, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1943074993728,8cyclictest501-21dbus-daemon10:40:002
1943074992824,2cyclictest0-21swapper/209:55:002
19629372250,1chrt1962936-21tr07:20:230
1943074992521,3cyclictest500-21avahi-daemon08:10:002
1943074992521,3cyclictest3564018-21systemctl10:00:002
1943070992522,2cyclictest2350760-21systemd-run12:25:001
20081152240,4sleep22008121-21latency_hist08:00:002
1943074992421,2cyclictest2157731-21apt-get10:05:002
1943074992419,3cyclictest2205542-21tail10:40:162
1943070992319,3cyclictest522-21in:imuxsock12:35:001
1943070992318,3cyclictest3564018-21systemctl10:25:001
1943074992218,2cyclictest0-21swapper/210:20:002
1943074992116,3cyclictest3004-21cron11:00:002
1943070992118,2cyclictest2141207-21cron09:55:001
1943070992117,3cyclictest0-21swapper/111:05:001
194306899219,5cyclictest3561480-21systemd-journal08:35:000
1943068992115,5cyclictest500-21avahi-daemon09:40:000
1943068992115,5cyclictest2169126-21apt-key10:15:010
1943068992115,5cyclictest2101236-21kworker/u8:1-rpciod11:00:000
1943068992115,5cyclictest2052787-21irqcore08:40:160
1943068992115,5cyclictest2052067-21idleruntime-cro08:40:000
1943068992115,5cyclictest0-21swapper/011:50:000
1943068992114,6cyclictest3561480-21systemd-journal12:05:000
1943068992114,6cyclictest3561480-21systemd-journal12:05:000
1943068992114,5cyclictest0-21swapper/012:40:000
1943074992017,2cyclictest0-21swapper/212:20:002
194306899202,1cyclictest23617312cat12:35:000
1943068992014,5cyclictest1989511-21latency_hist07:45:000
1943068992011,8cyclictest2177967-21fschecks_time10:20:130
1943068992011,7cyclictest3561480-21systemd-journal10:10:000
194306899200,5cyclictest2001682-21idleruntime-cro07:55:000
1943076991916,1cyclictest0-21swapper/312:40:013
1943070991916,2cyclictest2279803-21irqcore11:35:181
1943070991916,2cyclictest2243345-21latency_hist11:10:011
194307099191,16cyclictest2259800-21systemctl11:20:011
1943068991914,4cyclictest0-21swapper/010:45:000
1943068991913,5cyclictest508-21polkitd07:10:230
1943068991913,5cyclictest2236995-21kthreadcore11:00:170
1943068991913,5cyclictest0-21swapper/009:30:000
1943068991912,5cyclictest0-21swapper/010:50:010
194307699184,11cyclictest0-21swapper/311:34:553
1943076991816,1cyclictest0-21swapper/310:50:013
1943074991815,2cyclictest2369922-21sendmail12:40:002
1943074991815,2cyclictest2071221-21sendmail09:00:012
1943074991815,1cyclictest0-21swapper/210:15:002
1943074991814,3cyclictest0-21swapper/212:35:002
1943074991814,2cyclictest2180218-21kthreadcore10:20:162
1943074991814,2cyclictest0-21swapper/212:05:002
1943074991814,2cyclictest0-21swapper/212:05:002
1943070991815,2cyclictest2331778-21threads12:10:261
1943070991815,2cyclictest2331778-21threads12:10:261
1943070991815,2cyclictest2295735-21idleruntime-cro11:45:001
1943070991815,2cyclictest2007866-21cut08:00:001
1943070991814,2cyclictest0-21swapper/107:25:021
1943070991813,3cyclictest7550irq/9-acpi10:10:081
194306899189,7cyclictest1-21systemd09:00:020
194306899189,4cyclictest2305482-21kworker/u8:2-xprtiod12:26:310
194306899189,4cyclictest1-21systemd11:10:010
194306899188,4cyclictest3561480-21systemd-journal10:35:300
194306899187,5cyclictest501-21dbus-daemon07:40:000
194306899187,5cyclictest3561480-21systemd-journal07:20:010
194306899187,4cyclictest2086637-21/usr/sbin/munin09:15:250
194306899187,10cyclictest2102897-21systemd-run09:25:010
194306899186,6cyclictest3561480-21systemd-journal10:05:000
1943068991813,4cyclictest2196591-21latency_hist10:35:010
1943068991813,4cyclictest2021105-21sh08:15:000
1943068991813,4cyclictest0-21swapper/010:30:000
1943068991812,5cyclictest2303807-21ntp_kernel_pll_11:50:170
1943068991812,5cyclictest2254836-21kthreadcore11:15:190
1943068991812,5cyclictest2219149-21kthreadcore10:50:130
1943068991812,5cyclictest2146148-21kthreadcore09:55:190
1943068991812,5cyclictest2014498-21latency_hist08:05:000
1943068991811,6cyclictest2308209-21irqrtprio11:55:140
1943068991811,6cyclictest2122963-21irqcore09:40:160
1943068991811,6cyclictest2007438-21unixbench_singl07:55:240
1943068991811,5cyclictest0-21swapper/011:35:210
1943076991714,1cyclictest0-21swapper/308:05:003
1943076991714,1cyclictest0-21swapper/307:30:013
1943074991714,2cyclictest2130422-21latency_hist09:45:002
1943070991714,2cyclictest2323386-21apt-key12:05:001
1943070991714,2cyclictest2323386-21apt-key12:05:001
1943070991714,2cyclictest2020165-21grep08:10:161
1943070991713,3cyclictest514-21systemd-logind11:25:001
1943070991713,3cyclictest2123801-21kthreadcore09:40:171
1943070991713,3cyclictest2117315-21kthreadcore09:35:171
1943070991713,3cyclictest1-21systemd09:48:161
194307099171,14cyclictest2086670-21apt-get09:15:131
194306899178,3cyclictest2078371-21apt-get09:10:110
194306899178,3cyclictest2039221-21unixbench_multi08:25:240
194306899177,4cyclictest2334618-21apt-get12:15:120
194306899176,5cyclictest1970098-21basename07:30:010
1943068991712,4cyclictest500-21avahi-daemon08:20:000
1943068991712,4cyclictest1989320-21turbostat.cron07:49:550
1943068991711,5cyclictest2294210-21mandb11:42:180
1943068991711,5cyclictest2172039-21kthreadcore10:15:170
1943068991711,5cyclictest0-21swapper/011:30:000
1943068991711,4cyclictest0-21swapper/007:30:240
194307699168,6cyclictest2188531-21chrt10:30:093
1943076991613,1cyclictest0-21swapper/311:25:003
1943076991613,1cyclictest0-21swapper/310:40:003
1943076991613,1cyclictest0-21swapper/307:15:013
1943074991614,1cyclictest2270732-21latency_hist11:30:002
1943074991614,1cyclictest2027305-21readlink08:20:002
1943074991613,2cyclictest1975658-21awk07:30:262
1943074991613,2cyclictest0-21swapper/211:35:252
1943074991613,1cyclictest0-21swapper/207:20:012
1943074991612,3cyclictest277-21plymouthd11:00:112
1943074991612,3cyclictest2057123-21latency08:40:192
1943074991611,3cyclictest1989859-21fschecks_time07:45:132
1943074991610,4cyclictest2084006-21sshd09:10:232
1943074991610,3cyclictest0-21swapper/207:30:012
1943070991614,1cyclictest0-21swapper/107:30:011
1943070991613,2cyclictest1978181-21kthreadcore07:35:181
1943070991613,2cyclictest0-21swapper/111:55:161
1943070991612,3cyclictest1948825-21unixbench_singl07:10:251
1943070991612,2cyclictest0-21swapper/112:10:001
1943070991610,4cyclictest501-21dbus-daemon12:20:001
194306899169,2cyclictest2277219-21sh11:31:280
194306899167,4cyclictest2260166-21apt-get11:20:120
1943068991611,4cyclictest2321339-21kworker/0:1+kacpid12:09:310
1943068991611,4cyclictest2060638-21kthreadcore08:50:130
1943068991611,4cyclictest2027709-21tr08:20:120
1943068991611,4cyclictest2027709-21tr08:20:120
194306899161,10cyclictest2342873-21cpu12:20:130
1943068991610,5cyclictest2327788-21kthreadcore12:10:200
1943068991610,5cyclictest2327788-21kthreadcore12:10:200
1943068991610,5cyclictest2243682-21cpuspeed_turbos11:10:090
1943068991610,5cyclictest2138294-21needreboot09:50:160
1943068991610,5cyclictest2108879-21kthreadcore09:30:140
1943068991610,5cyclictest2078016-21cron09:10:000
1943068991610,5cyclictest2077383-21cron09:05:000
1943076991512,1cyclictest0-21swapper/312:35:013
1943076991512,1cyclictest0-21swapper/312:30:003
1943076991512,1cyclictest0-21swapper/312:00:163
1943076991512,1cyclictest0-21swapper/312:00:163
1943076991512,1cyclictest0-21swapper/311:50:143
1943076991512,1cyclictest0-21swapper/311:35:263
1943076991512,1cyclictest0-21swapper/311:30:013
1943076991512,1cyclictest0-21swapper/311:10:123
1943076991512,1cyclictest0-21swapper/309:57:163
1943076991512,1cyclictest0-21swapper/309:30:183
1943076991512,1cyclictest0-21swapper/309:00:213
1943076991512,1cyclictest0-21swapper/308:35:123
1943076991512,1cyclictest0-21swapper/307:50:003
1943076991512,1cyclictest0-21swapper/307:32:363
194307499158,5cyclictest3561480-21systemd-journal09:32:122
1943074991513,1cyclictest0-21swapper/208:15:002
1943074991512,2cyclictest2307266-21gltestperf11:55:132
1943074991512,2cyclictest2255964-21kthreadcore11:15:202
1943074991512,2cyclictest1949244-21sh07:15:012
1943074991512,2cyclictest0-21swapper/212:30:002
1943074991512,2cyclictest0-21swapper/207:40:002
1943074991511,3cyclictest1-21systemd11:51:162
1943074991511,2cyclictest2274644-21kthreadcore11:30:142
194307099159,4cyclictest3561480-21systemd-journal09:00:001
1943070991512,2cyclictest2272215-21irqrtprio11:30:121
1943070991512,2cyclictest2232392-21cut11:00:011
1943070991512,2cyclictest2151239-21tail10:00:181
1943070991512,2cyclictest1943060-21cyclictest10:40:191
1943070991512,2cyclictest0-21swapper/110:31:021
1943070991512,2cyclictest0-21swapper/108:15:161
1943070991511,3cyclictest2244189-21irqcore11:10:121
1943070991511,3cyclictest2142405-21irqcore09:55:171
1943070991511,2cyclictest0-21swapper/110:50:011
194307099150,2cyclictest2074464-21kthreadcore09:00:161
194307099150,13cyclictest1995504-21awk07:50:001
194306899156,3cyclictest2014928-21sh08:10:000
1943068991510,4cyclictest2064828-21kworker/0:0+kacpid09:48:060
21694542140,1sleep30-21swapper/310:15:133
1943076991412,1cyclictest0-21swapper/311:15:183
1943076991412,1cyclictest0-21swapper/307:40:403
1943076991411,1cyclictest0-21swapper/311:55:163
1943076991411,1cyclictest0-21swapper/311:40:183
1943076991411,1cyclictest0-21swapper/310:40:223
1943076991411,1cyclictest0-21swapper/309:50:113
1943076991411,1cyclictest0-21swapper/309:10:143
1943076991411,1cyclictest0-21swapper/308:15:163
1943076991411,1cyclictest0-21swapper/308:00:003
1943076991411,1cyclictest0-21swapper/307:50:213
1943076991411,1cyclictest0-21swapper/307:10:183
194307699141,10cyclictest21139512sleep309:35:013
194307499149,4cyclictest2052018-21cron08:40:002
194307499149,4cyclictest0-21swapper/210:50:082
1943074991412,1cyclictest0-21swapper/211:15:012
1943074991411,2cyclictest3561480-21systemd-journal08:50:192
1943074991411,2cyclictest2101236-21kworker/u8:1+xprtiod09:37:312
1943074991411,2cyclictest2088779-21kthreadcore09:15:182
1943074991411,2cyclictest2078023-21diskmemload11:40:192
1943074991411,2cyclictest2077381-21cron09:05:002
1943074991410,3cyclictest2265753-21unixbench_multi11:20:232
194307099149,4cyclictest2033533-21systemd-run08:25:001
194307099149,4cyclictest2033533-21systemd-run08:25:001
1943070991411,2cyclictest2221777-21unixbench_singl10:50:191
1943070991411,2cyclictest2176475-21runrttasks10:18:151
194307099141,12cyclictest2052609-21expr08:40:151
1943070991411,2cyclictest2049814-21kthreadcore08:35:181
1943070991411,2cyclictest1970822-21fschecks_count07:30:181
1943070991411,2cyclictest1-21systemd07:53:171
1943070991411,2cyclictest0-21swapper/109:10:141
1943070991411,2cyclictest0-21swapper/108:00:231
1943070991411,2cyclictest0-21swapper/107:15:171
1943070991410,3cyclictest2364569-21kthreadcore12:35:181
1943070991410,2cyclictest0-21swapper/108:50:121
194306899148,5cyclictest0-21swapper/008:50:000
1943076991311,1cyclictest0-21swapper/308:50:103
1943076991310,1cyclictest0-21swapper/312:15:183
1943076991310,1cyclictest0-21swapper/312:10:203
1943076991310,1cyclictest0-21swapper/312:10:203
1943076991310,1cyclictest0-21swapper/311:47:583
1943076991310,1cyclictest0-21swapper/310:55:153
1943076991310,1cyclictest0-21swapper/310:20:153
1943076991310,1cyclictest0-21swapper/310:10:103
1943076991310,1cyclictest0-21swapper/310:00:183
1943076991310,1cyclictest0-21swapper/309:49:163
1943076991310,1cyclictest0-21swapper/308:44:393
1943076991310,1cyclictest0-21swapper/308:30:183
1943076991310,1cyclictest0-21swapper/308:25:193
1943076991310,1cyclictest0-21swapper/308:10:153
194307499139,3cyclictest2343107-21fschecks_time12:20:152
194307499139,2cyclictest2326475-21cpuspeed_turbos12:10:162
194307499139,2cyclictest2326475-21cpuspeed_turbos12:10:162
194307499139,2cyclictest0-21swapper/208:30:182
194307499139,2cyclictest0-21swapper/207:45:002
194307499137,4cyclictest547-21gdbus08:30:002
194307499137,4cyclictest3561480-21systemd-journal08:24:562
194307499137,4cyclictest3561480-21systemd-journal08:24:552
1943074991310,2cyclictest2216023-21apt-key10:50:012
1943074991310,2cyclictest2095902-21grep09:20:162
1943074991310,2cyclictest2078189-21latency_hist09:10:012
1943074991310,2cyclictest2058910-21latency_hist08:50:012
1943074991310,2cyclictest0-21swapper/208:05:002
194307099139,2cyclictest0-21swapper/110:10:001
194307099138,2cyclictest0-21swapper/109:25:011
194307099137,5cyclictest1-21systemd07:45:001
194307099137,4cyclictest3561480-21systemd-journal08:10:001
194307099131,9cyclictest2353389-21latency_hist12:30:011
1943070991310,2cyclictest744-21snmpd09:25:371
1943070991310,2cyclictest2042818-21kthreadcore08:30:201
1943070991310,2cyclictest0-21swapper/111:50:131
194307699129,1cyclictest0-21swapper/312:20:163
194307699129,1cyclictest0-21swapper/309:40:163
194307699129,1cyclictest0-21swapper/309:27:303
194307699129,1cyclictest0-21swapper/309:15:173
194307699129,1cyclictest0-21swapper/309:10:013
194307699129,1cyclictest0-21swapper/308:55:183
194307699129,1cyclictest0-21swapper/308:20:173
194307699129,1cyclictest0-21swapper/308:20:173
194307699129,1cyclictest0-21swapper/307:35:163
1943076991210,1cyclictest0-21swapper/311:00:153
1943076991210,1cyclictest0-21swapper/310:50:133
1943076991210,1cyclictest0-21swapper/307:20:213
194307499129,2cyclictest2243297-21cron11:10:002
194307499129,2cyclictest2189934-21kthreadcore10:30:122
194307499129,2cyclictest2160731-21latency_hist10:10:002
194307499129,2cyclictest2078023-21diskmemload11:48:342
194307499129,2cyclictest1960073-21kthreadcore07:20:202
194307499129,2cyclictest0-21swapper/210:30:002
194307499129,2cyclictest0-21swapper/207:50:202
194307099129,2cyclictest3004-21cron11:50:001
194307099129,2cyclictest2113870-21latency_hist09:35:001
194307099129,2cyclictest2058920-21latency_hist08:50:011
194307099128,2cyclictest2078017-21cron09:10:001
194307099126,4cyclictest2200721-21tr10:35:201
194307099121,3cyclictest1942986-21latency_hist07:10:011
194307099120,11cyclictest2187700-21ssh10:28:571
19371242121,5sleep30-21swapper/307:05:123
194307699119,1cyclictest0-21swapper/312:05:013
194307699119,1cyclictest0-21swapper/310:10:003
1943076991110,0cyclictest0-21swapper/309:20:163
194307499119,1cyclictest2325941-21cat12:09:552
194307499110,4cyclictest2105520-21cron09:30:002
194307099116,4cyclictest2033921-21grep08:25:141
194307099110,10cyclictest0-21swapper/111:25:121
194307699108,1cyclictest0-21swapper/311:06:453
194307499106,2cyclictest0-21swapper/209:50:002
19430769997,1cyclictest0-21swapper/310:27:563
19430769997,1cyclictest0-21swapper/308:50:003
19430769992,5cyclictest20148202sleep308:08:443
19430689981,4cyclictest1942986-21latency_hist07:10:010
19430749974,1cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional