You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 06:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Wed Apr 02, 2025 00:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
55121050,1sleep10-21swapper/118:58:051
133262940,1sleep113327-21bash00:11:501
133262940,1sleep113327-21bash00:11:501
324842910,1sleep60-21swapper/618:58:026
170625226,9sleep70-21swapper/718:58:127
3192525026,8sleep00-21swapper/018:57:490
263372500,0sleep50-21swapper/521:43:465
205125027,8sleep50-21swapper/519:02:355
167762470,0sleep70-21swapper/700:15:007
62202450,0sleep1241ktimersoftd/123:45:151
300232450,0sleep20-21swapper/200:13:112
321352440,0sleep40-21swapper/421:55:144
22732440,1sleep52268-21bash23:01:225
176032440,0sleep40-21swapper/422:43:194
68632430,0sleep60-21swapper/600:26:056
249962430,1sleep21-21systemd22:13:462
170392430,0sleep70-21swapper/722:24:087
170392430,0sleep70-21swapper/722:24:087
148892430,0sleep20-21swapper/200:30:412
113512430,2sleep1225799cyclictest21:03:371
89972420,0sleep00-21swapper/023:10:060
272342420,0sleep70-21swapper/723:11:417
179432420,0sleep40-21swapper/422:24:134
179432420,0sleep40-21swapper/422:24:134
5092410,0sleep20-21swapper/223:03:512
191424129,9sleep30-21swapper/319:00:493
174892410,0sleep20-21swapper/221:18:282
294072400,0sleep00-21swapper/000:13:090
259302400,0sleep10-21swapper/122:08:241
181724029,8sleep20-21swapper/218:59:282
301582390,0sleep20-21swapper/221:49:352
173123926,9sleep40-21swapper/418:58:174
41602380,0sleep00-21swapper/023:31:290
125872380,0sleep30-21swapper/323:02:163
99752360,0sleep20-21swapper/220:13:092
242672360,0sleep70-21swapper/723:25:037
241512360,0sleep50-21swapper/521:16:145
29402320,0sleep40-21swapper/423:53:034
57752250,1sleep599-21kswapd023:47:595
220392210,0sleep00-21swapper/000:18:300
2277992017,1cyclictest0-21swapper/521:05:275
226499202,17cyclictest16100-21bash23:51:292
229099198,5cyclictest0-21swapper/720:57:567
229099198,0cyclictest0-21swapper/719:06:027
229099198,0cyclictest0-21swapper/719:06:027
227799198,5cyclictest0-21swapper/520:51:525
227799190,7cyclictest0-21swapper/500:02:585
227799190,14cyclictest0-21swapper/522:07:525
2264991917,1cyclictest33-21ksoftirqd/223:15:562
2264991914,3cyclictest33-21ksoftirqd/200:00:202
2264991913,6cyclictest33-21ksoftirqd/221:44:192
229099187,5cyclictest3088-21sshd22:03:467
229099187,5cyclictest18530-21id23:51:427
229099187,5cyclictest0-21swapper/720:55:307
229099181,5cyclictest0-21swapper/721:28:247
229099180,6cyclictest0-21swapper/721:20:287
227799187,5cyclictest0-21swapper/519:28:005
227799187,0cyclictest0-21swapper/520:46:205
227799187,0cyclictest0-21swapper/520:46:205
227799180,6cyclictest0-21swapper/520:09:145
229099176,5cyclictest0-21swapper/722:20:107
229099176,5cyclictest0-21swapper/719:49:427
229099176,0cyclictest0-21swapper/719:14:007
229099175,5cyclictest0-21swapper/719:41:467
227799176,5cyclictest0-21swapper/523:35:165
2277991713,0cyclictest0-21swapper/500:00:045
226699170,13cyclictest29038-21sshd23:03:313
2264991715,2cyclictest33-21ksoftirqd/223:52:502
2264991715,1cyclictest33-21ksoftirqd/222:05:062
2264991712,3cyclictest33-21ksoftirqd/222:39:052
2264991712,3cyclictest33-21ksoftirqd/221:57:562
2264991711,3cyclictest33-21ksoftirqd/223:47:232
229099169,6cyclictest16069-21sshd21:34:447
229099168,8cyclictest0-21swapper/722:15:227
229099168,5cyclictest11052-21sshd23:21:187
229099164,1cyclictest19363-21sshd23:05:327
229099164,1cyclictest11386-21cstates22:42:567
2290991614,0cyclictest721ktimersoftd/700:24:547
2290991611,1cyclictest14010-21grepconf.sh22:32:087
2290991611,0cyclictest0-21swapper/720:50:387
227799165,5cyclictest0-21swapper/521:59:305
227799165,11cyclictest28105-21nfsd22:55:445
227799164,0cyclictest0-21swapper/520:01:225
227799160,13cyclictest18364-21sshd22:35:125
226699160,12cyclictest99-21kswapd020:38:353
226499164,2cyclictest33-21ksoftirqd/223:40:292
2264991614,1cyclictest33-21ksoftirqd/222:58:362
2264991614,1cyclictest33-21ksoftirqd/222:36:052
2264991614,1cyclictest33-21ksoftirqd/221:57:402
2264991614,1cyclictest33-21ksoftirqd/221:27:292
2264991614,1cyclictest33-21ksoftirqd/221:03:072
2264991611,3cyclictest33-21ksoftirqd/200:19:232
2264991610,4cyclictest33-21ksoftirqd/222:30:182
229099154,5cyclictest0-21swapper/720:46:527
229099154,5cyclictest0-21swapper/720:46:527
229099154,5cyclictest0-21swapper/720:36:087
229099154,0cyclictest0-21swapper/720:21:567
229099150,14cyclictest20697-21kworker/u16:221:24:137
2283991512,2cyclictest11342-21bash23:56:426
227799154,5cyclictest0-21swapper/520:38:505
227799154,5cyclictest0-21swapper/500:30:545
227799150,9cyclictest0-21swapper/523:07:105
227799150,9cyclictest0-21swapper/521:48:025
227799150,9cyclictest0-21swapper/520:26:045
227799150,4cyclictest13580-21id22:12:565
2264991514,1cyclictest33-21ksoftirqd/223:25:312
2264991514,1cyclictest33-21ksoftirqd/221:38:252
2264991514,1cyclictest33-21ksoftirqd/221:38:252
2264991513,1cyclictest33-21ksoftirqd/222:46:362
2264991513,1cyclictest33-21ksoftirqd/222:17:552
2264991513,1cyclictest33-21ksoftirqd/221:13:382
2264991513,1cyclictest33-21ksoftirqd/200:10:262
2264991513,1cyclictest33-21ksoftirqd/200:10:262
2264991513,1cyclictest33-21ksoftirqd/200:05:272
2264991512,3cyclictest33-21ksoftirqd/222:53:532
2264991512,3cyclictest33-21ksoftirqd/222:50:052
2264991511,1cyclictest33-21ksoftirqd/223:07:532
225799152,12cyclictest5119-21bash23:28:471
225799150,13cyclictest21039-21sshd23:24:461
207852150,1sleep09-21ksoftirqd/022:10:480
96982140,0sleep60-21swapper/622:28:566
278632140,0sleep10-21swapper/100:28:241
229099148,5cyclictest0-21swapper/722:39:127
229099148,5cyclictest0-21swapper/722:33:247
229099148,5cyclictest0-21swapper/700:04:447
229099142,11cyclictest0-21swapper/723:14:527
229099140,8cyclictest0-21swapper/720:27:527
229099140,8cyclictest0-21swapper/700:08:147
229099140,8cyclictest0-21swapper/700:08:147
227799149,1cyclictest948-21sshd23:31:125
227799149,1cyclictest27953-21sshd21:38:305
227799149,1cyclictest27953-21sshd21:38:305
227799148,6cyclictest0-21swapper/523:15:145
227799148,6cyclictest0-21swapper/519:20:425
227799148,5cyclictest630-21sshd22:25:325
227799148,5cyclictest630-21sshd22:25:325
227799148,5cyclictest0-21swapper/523:08:265
227799148,0cyclictest0-21swapper/519:51:345
227799143,0cyclictest0-21swapper/519:24:265
227799142,6cyclictest18088-21grep23:57:225
227799140,8cyclictest0-21swapper/522:50:165
227799140,8cyclictest0-21swapper/522:03:125
227799140,8cyclictest0-21swapper/500:12:285
227799140,8cyclictest0-21swapper/500:12:285
227799140,2cyclictest0-21swapper/521:09:445
2264991412,2cyclictest33-21ksoftirqd/223:19:262
2264991412,1cyclictest33-21ksoftirqd/223:33:052
2264991412,1cyclictest33-21ksoftirqd/223:32:492
2264991412,1cyclictest33-21ksoftirqd/222:24:332
2264991412,1cyclictest33-21ksoftirqd/222:24:332
2264991412,1cyclictest33-21ksoftirqd/221:35:162
2264991412,1cyclictest33-21ksoftirqd/221:29:072
2264991412,1cyclictest33-21ksoftirqd/200:24:202
2264991411,1cyclictest33-21ksoftirqd/221:07:522
2255991413,1cyclictest9-21ksoftirqd/023:24:210
2255991410,3cyclictest9-21ksoftirqd/023:50:540
225599140,1cyclictest131rcu_sched22:42:580
19862140,2sleep1225799cyclictest21:49:581
59942130,0sleep10-21swapper/122:45:081
229099138,5cyclictest21330-21sshd21:10:347
229099138,5cyclictest0-21swapper/723:57:587
229099138,5cyclictest0-21swapper/723:55:387
229099138,5cyclictest0-21swapper/723:44:287
229099138,5cyclictest0-21swapper/723:38:527
229099138,5cyclictest0-21swapper/723:34:407
229099138,5cyclictest0-21swapper/723:00:507
229099138,5cyclictest0-21swapper/722:47:527
229099138,5cyclictest0-21swapper/721:53:307
229099138,5cyclictest0-21swapper/721:49:007
229099138,5cyclictest0-21swapper/721:15:367
229099138,5cyclictest0-21swapper/719:34:147
229099138,5cyclictest0-21swapper/719:23:367
229099138,5cyclictest0-21swapper/719:08:447
229099138,1cyclictest19889-21grep21:43:127
229099138,1cyclictest10831-21sshd22:53:407
229099138,0cyclictest0-21swapper/722:00:067
229099138,0cyclictest0-21swapper/720:39:467
229099138,0cyclictest0-21swapper/720:02:547
229099132,5cyclictest0-21swapper/719:46:187
229099132,5cyclictest0-21swapper/719:29:167
229099132,5cyclictest0-21swapper/719:18:427
229099132,0cyclictest0-21swapper/721:06:407
229099132,0cyclictest0-21swapper/720:07:527
229099132,0cyclictest0-21swapper/719:53:047
229099132,0cyclictest0-21swapper/719:53:047
229099132,0cyclictest0-21swapper/700:28:307
2283991312,1cyclictest65-21ksoftirqd/622:02:266
2283991312,1cyclictest65-21ksoftirqd/621:43:046
2283991311,1cyclictest65-21ksoftirqd/621:41:426
2283991311,1cyclictest65-21ksoftirqd/621:41:426
227799138,5cyclictest0-21swapper/523:46:365
227799138,5cyclictest0-21swapper/523:41:225
227799138,5cyclictest0-21swapper/522:47:085
227799138,5cyclictest0-21swapper/522:30:365
227799138,5cyclictest0-21swapper/522:22:305
227799138,5cyclictest0-21swapper/520:58:185
227799138,5cyclictest0-21swapper/520:18:045
227799138,5cyclictest0-21swapper/520:13:025
227799138,5cyclictest0-21swapper/519:44:485
227799138,5cyclictest0-21swapper/519:38:325
227799138,5cyclictest0-21swapper/500:14:325
227799138,1cyclictest732-21NetworkManager23:18:245
227799138,0cyclictest0-21swapper/520:28:005
227799138,0cyclictest0-21swapper/519:33:145
227799138,0cyclictest0-21swapper/519:07:565
227799137,6cyclictest2366-21sshd22:39:245
227799137,5cyclictest30684-21sshd23:25:365
227799136,5cyclictest28105-21nfsd21:28:045
227799132,5cyclictest0-21swapper/520:35:405
227799132,5cyclictest0-21swapper/500:23:365
227799132,0cyclictest0-21swapper/520:53:185
227799131,6cyclictest0-21swapper/519:13:205
227799131,1cyclictest8893-21sshd21:20:325
227799130,8cyclictest22829-21sshd00:18:345
227499130,12cyclictest23934-21kworker/4:521:24:524
2264991311,1cyclictest33-21ksoftirqd/222:08:132
2255991312,1cyclictest9-21ksoftirqd/023:41:540
2255991312,1cyclictest9-21ksoftirqd/022:22:400
2255991312,1cyclictest9-21ksoftirqd/021:46:070
2255991312,1cyclictest9-21ksoftirqd/021:21:450
2255991311,1cyclictest9-21ksoftirqd/021:52:480
21692130,0sleep32168-21systemd-cgroups00:22:413
109712130,0sleep30-21swapper/321:09:403
314502120,0sleep10-21swapper/123:33:381
229099127,5cyclictest0-21swapper/723:28:187
229099127,5cyclictest0-21swapper/722:08:327
229099127,5cyclictest0-21swapper/700:20:247
229099121,6cyclictest0-21swapper/721:38:167
229099121,6cyclictest0-21swapper/721:38:167
229099121,0cyclictest0-21swapper/720:12:587
228399129,1cyclictest85350irq/25-em1-rx-023:15:286
228399129,1cyclictest65-21ksoftirqd/623:02:066
2283991210,1cyclictest65-21ksoftirqd/622:12:486
2283991210,1cyclictest65-21ksoftirqd/621:55:406
2283991210,1cyclictest65-21ksoftirqd/600:17:306
228399120,12cyclictest27913-21kworker/6:323:30:566
228399120,11cyclictest0-21swapper/619:22:486
227799127,5cyclictest0-21swapper/521:54:325
227799127,5cyclictest0-21swapper/521:33:285
227799126,5cyclictest1-21systemd21:22:545
227799121,5cyclictest0-21swapper/519:56:065
227799121,5cyclictest0-21swapper/519:56:065
227799120,0cyclictest0-21swapper/520:05:145
227799120,0cyclictest0-21swapper/519:07:445
227799120,0cyclictest0-21swapper/519:07:445
227499120,11cyclictest23070-21bash00:09:384
227499120,11cyclictest23070-21bash00:09:384
227499120,11cyclictest18980-21bash21:21:254
227499120,11cyclictest16253-21sshd23:43:184
2266991210,1cyclictest711-21dbus-daemon00:13:203
226699120,2cyclictest26932-21sshd23:25:173
226699120,2cyclictest18001-21bash22:43:213
225799120,11cyclictest859-21polkitd22:56:481
2255991211,1cyclictest9-21ksoftirqd/021:09:210
2255991210,1cyclictest9-21ksoftirqd/023:36:360
2255991210,1cyclictest9-21ksoftirqd/023:13:050
2255991210,1cyclictest9-21ksoftirqd/022:28:040
2255991210,1cyclictest9-21ksoftirqd/000:09:260
2255991210,1cyclictest9-21ksoftirqd/000:09:260
2255991210,1cyclictest9-21ksoftirqd/000:03:550
225599120,11cyclictest9584-21sshd22:34:250
229099116,5cyclictest0-21swapper/719:58:207
228399119,1cyclictest65-21ksoftirqd/623:38:006
228399119,1cyclictest65-21ksoftirqd/623:08:056
228399119,1cyclictest65-21ksoftirqd/621:28:036
228399114,5cyclictest32108-21sshd21:25:146
227499119,1cyclictest49-21ksoftirqd/423:58:244
227499117,1cyclictest49-21ksoftirqd/423:14:164
227499116,2cyclictest711-21dbus-daemon23:36:244
227499114,4cyclictest740-21gdbus22:02:054
2274991110,1cyclictest49-21ksoftirqd/421:07:394
227499110,11cyclictest0-21swapper/423:24:524
227499110,1cyclictest0-21swapper/423:49:274
226699119,1cyclictest30652-21sshd22:19:523
226699114,6cyclictest7625-21kworker/3:523:33:523
226699112,6cyclictest30106-21sshd21:13:553
226699110,11cyclictest0-21swapper/321:57:383
226699110,1cyclictest0-21swapper/323:10:193
225799118,3cyclictest20304-21kworker/1:021:28:121
225799113,2cyclictest7747-21grepconf.sh22:48:021
225799112,4cyclictest30340-21sshd22:36:141
225799111,4cyclictest20246-21/usr/sbin/munin23:02:561
225599119,1cyclictest9-21ksoftirqd/022:51:320
225599119,1cyclictest9-21ksoftirqd/021:24:160
225599119,1cyclictest9-21ksoftirqd/021:03:080
225599119,1cyclictest859-21polkitd00:30:270
225599117,3cyclictest9-21ksoftirqd/023:19:520
2255991110,1cyclictest9-21ksoftirqd/023:43:410
2255991110,1cyclictest9-21ksoftirqd/022:16:230
2255991110,1cyclictest9-21ksoftirqd/021:42:200
2255991110,1cyclictest9-21ksoftirqd/021:42:200
2255991110,1cyclictest9-21ksoftirqd/021:34:450
2255991110,1cyclictest9-21ksoftirqd/021:15:420
2255991110,1cyclictest9-21ksoftirqd/000:23:430
225599110,10cyclictest0-21swapper/022:25:060
225599110,10cyclictest0-21swapper/022:25:060
229099105,5cyclictest0-21swapper/720:24:387
228399109,1cyclictest65-21ksoftirqd/622:39:266
228399109,1cyclictest65-21ksoftirqd/621:14:106
228399109,1cyclictest65-21ksoftirqd/600:28:276
228399108,2cyclictest65-21ksoftirqd/622:55:266
228399108,1cyclictest65-21ksoftirqd/622:43:066
228399108,1cyclictest65-21ksoftirqd/622:34:336
228399108,1cyclictest65-21ksoftirqd/622:22:346
228399108,1cyclictest65-21ksoftirqd/621:49:426
228399104,3cyclictest1740-21sshd23:23:066
228399100,10cyclictest0-21swapper/622:23:136
228399100,10cyclictest0-21swapper/622:23:136
228399100,0cyclictest0-21swapper/623:44:246
227499108,1cyclictest49-21ksoftirqd/422:19:414
227499108,1cyclictest49-21ksoftirqd/419:27:494
227499108,1cyclictest0-21swapper/423:05:284
227499104,4cyclictest12148-21sshd22:48:174
227499102,4cyclictest0-21swapper/422:30:294
227499101,6cyclictest4125-21sshd22:17:364
227499101,3cyclictest25040-21sshd22:41:214
227499100,10cyclictest0-21swapper/421:16:014
227499100,10cyclictest0-21swapper/400:18:474
226699108,1cyclictest41-21ksoftirqd/323:32:483
226699108,1cyclictest41-21ksoftirqd/321:48:573
226699103,6cyclictest0-21swapper/323:14:163
226699103,4cyclictest1686-21perf21:02:483
226699101,4cyclictest30706-21grep23:55:273
226699100,4cyclictest26617-21sshd22:08:283
226699100,1cyclictest0-21swapper/300:02:393
226499109,1cyclictest33-21ksoftirqd/219:52:482
226499109,1cyclictest33-21ksoftirqd/219:18:102
225799109,1cyclictest0-21swapper/100:23:221
225799108,1cyclictest25-21ksoftirqd/121:37:091
225799104,2cyclictest22476-21sshd22:38:211
225799102,4cyclictest17028-21sshd23:10:481
225799101,4cyclictest21968-21sshd21:07:571
225799101,4cyclictest18197-21sshd21:18:311
225799101,3cyclictest11589-21sshd23:53:421
225799100,5cyclictest22536-21bash22:05:271
225799100,4cyclictest861-21gdbus21:57:061
225799100,4cyclictest859-21polkitd23:02:021
225799100,4cyclictest29326-21sshd00:22:121
225799100,10cyclictest0-21swapper/122:16:081
225799100,1cyclictest0-21swapper/122:31:351
225599109,1cyclictest9-21ksoftirqd/023:53:380
225599109,1cyclictest9-21ksoftirqd/023:02:560
225599109,1cyclictest9-21ksoftirqd/022:38:400
225599108,1cyclictest9-21ksoftirqd/022:58:090
225599108,1cyclictest9-21ksoftirqd/021:53:000
225599108,1cyclictest9-21ksoftirqd/021:30:350
225599107,3cyclictest9-21ksoftirqd/000:00:480
225599103,4cyclictest740-21gdbus22:55:470
173722100,0sleep60-21swapper/622:05:016
3253290,1sleep33255-21grepconf.sh22:17:313
28192290,0sleep10-21swapper/100:16:071
22839999,0cyclictest0-21swapper/623:49:086
22839998,1cyclictest65-21ksoftirqd/623:59:336
22839997,1cyclictest65-21ksoftirqd/623:19:406
22839997,1cyclictest65-21ksoftirqd/622:50:436
22839997,1cyclictest65-21ksoftirqd/600:08:086
22839997,1cyclictest65-21ksoftirqd/600:08:086
22839997,1cyclictest65-21ksoftirqd/600:02:516
22839994,3cyclictest26379-21sshd21:35:376
22839993,3cyclictest732-21NetworkManager21:08:116
22839992,4cyclictest711-21dbus-daemon00:22:416
22839990,9cyclictest27913-21kworker/6:323:35:436
22839990,1cyclictest0-21swapper/622:14:326
22839990,0cyclictest0-21swapper/623:04:226
22749999,0cyclictest14462-21sshd21:29:094
22749998,1cyclictest49-21ksoftirqd/421:50:394
22749995,3cyclictest4330-21sshd00:25:474
22749995,1cyclictest14399-21bash21:40:084
22749995,1cyclictest14399-21bash21:40:084
22749993,3cyclictest25529-21bash23:08:434
22749993,3cyclictest24769-21sshd21:32:504
22749992,4cyclictest711-21dbus-daemon23:18:214
22749992,4cyclictest25994-21bash22:35:524
22749990,9cyclictest0-21swapper/423:01:034
22749990,9cyclictest0-21swapper/422:56:384
22749990,9cyclictest0-21swapper/422:07:184
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional