You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:17
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack2slot0.osadl.org (updated Thu Apr 03, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
308932730,0sleep10-21swapper/123:56:041
136762610,1sleep013671-21sshd00:01:240
108872610,0sleep010873-21sshd21:40:380
327052560,0sleep00-21swapper/021:22:230
98282550,0sleep30-21swapper/323:57:263
65512540,0sleep00-21swapper/021:36:530
269532540,0sleep026951-21lspci22:12:380
112472530,0sleep10-21swapper/123:01:331
278872510,1sleep027890-21bash00:17:470
227182510,0sleep20-21swapper/223:55:132
50772500,1sleep10-21swapper/123:32:051
313362500,0sleep10-21swapper/100:10:381
295892500,0sleep1231rcuc/121:59:331
282742500,0sleep30-21swapper/321:21:553
188202500,0sleep00-21swapper/021:58:240
164232500,0sleep30-21swapper/322:01:273
4962490,0sleep20-21swapper/222:23:052
34232490,0sleep10-21swapper/120:40:161
86692480,0sleep20-21swapper/223:15:162
298552480,0sleep20-21swapper/222:12:562
133222480,0sleep30-21swapper/323:54:143
111182450,2sleep33025599cyclictest23:22:313
2992123727,6sleep20-21swapper/219:07:242
2979522715,7sleep10-21swapper/119:05:471
298522236,4sleep30-21swapper/319:06:303
3005322210,7sleep00-21swapper/019:09:020
293252190,1sleep20-21swapper/222:29:312
293252190,1sleep20-21swapper/222:29:302
279292180,0sleep30-21swapper/322:22:343
146022170,0sleep10-21swapper/121:54:401
130872170,1sleep30-21swapper/300:04:593
70572160,0sleep30-21swapper/323:39:163
153802160,1sleep1646-21dbus-daemon21:41:071
55792150,0sleep20-21swapper/200:07:512
37912150,0sleep20-21swapper/221:22:462
30250991514,0cyclictest6864-21lspci23:25:332
30250991514,0cyclictest21311-21lspci23:51:262
127312150,1sleep012729-21lspci22:44:340
10552150,0sleep10-21swapper/121:49:561
30243991412,1cyclictest27882-21lspci00:37:481
297472140,0sleep2692-21NetworkManager21:52:532
23812140,1sleep12378-21sshd22:03:231
202452140,0sleep00-21swapper/021:31:320
201052140,0sleep30-21swapper/321:10:093
3025599131,1cyclictest75550irq/24-eno1-rx-23:27:303
3025599130,11cyclictest24384-21sshd21:59:003
3025099134,1cyclictest33-21ksoftirqd/221:30:392
3025099130,12cyclictest0-21swapper/223:43:002
3024399130,12cyclictest1183-21bash21:29:371
3025599120,1cyclictest14558-21sshd21:54:403
3025599120,1cyclictest0-21swapper/323:31:203
3025099122,1cyclictest33-21ksoftirqd/223:06:572
3025099121,1cyclictest33-21ksoftirqd/223:03:502
30250991211,0cyclictest321ktimersoftd/221:47:152
30250991210,1cyclictest8596-21sshd00:23:202
3025099120,1cyclictest5061-21lspci22:54:092
3025099120,11cyclictest0-21swapper/222:30:592
3025099120,11cyclictest0-21swapper/221:14:402
3025099120,11cyclictest0-21swapper/200:31:402
3025099120,0cyclictest646-21dbus-daemon00:30:012
30243991210,1cyclictest6785-21lspci21:19:371
30243991210,1cyclictest1659-21sshd22:16:371
3024399120,1cyclictest0-21swapper/100:24:281
3024399120,11cyclictest0-21swapper/100:03:591
30238991210,1cyclictest16325-21sshd22:24:460
3023899120,11cyclictest5032-21sshd21:19:250
289962120,0sleep20-21swapper/222:46:212
3025599119,1cyclictest1092-21nfsd23:18:103
3025599119,1cyclictest0-21swapper/322:14:513
3025599110,10cyclictest5942-21bash22:07:103
3025599110,10cyclictest0-21swapper/323:47:513
3025599110,10cyclictest0-21swapper/323:07:103
3025599110,10cyclictest0-21swapper/320:25:203
3025599110,0cyclictest0-21swapper/323:40:403
3025099119,1cyclictest19129-21lspci22:59:052
3025099119,1cyclictest13464-21cat00:20:002
3025099113,2cyclictest121rcu_preempt23:45:272
3025099111,1cyclictest33-21ksoftirqd/222:40:442
30250991110,0cyclictest0-21swapper/200:36:402
3025099110,1cyclictest33-21ksoftirqd/222:15:452
3025099110,1cyclictest121rcu_preempt21:36:502
3025099110,10cyclictest0-21swapper/222:01:202
3025099110,10cyclictest0-21swapper/220:45:202
3025099110,0cyclictest0-21swapper/221:40:492
3024399119,1cyclictest5583-21ntp_states19:25:171
30243991110,0cyclictest0-21swapper/100:34:281
3024399110,10cyclictest17774-21lspci21:31:171
3024399110,10cyclictest17336-21lspci22:51:581
3024399110,10cyclictest0-21swapper/100:16:081
3024399110,0cyclictest0-21swapper/123:35:191
3024399110,0cyclictest0-21swapper/121:10:481
3023899119,1cyclictest2963-21sshd00:11:060
30238991110,0cyclictest0-21swapper/023:11:360
293702110,1sleep11-21systemd23:45:111
144462110,0sleep3391rcuc/322:31:123
121872110,1sleep21095-21nfsd23:22:382
3025599109,0cyclictest0-21swapper/322:45:503
3025599109,0cyclictest0-21swapper/321:36:103
3025599108,1cyclictest21226-21sshd21:45:113
3025599100,9cyclictest18855-21sshd21:28:003
3025099108,1cyclictest33-21ksoftirqd/223:40:012
3025099108,1cyclictest33-21ksoftirqd/223:30:232
3025099102,1cyclictest33-21ksoftirqd/222:39:032
3025099100,1cyclictest7242-21sshd00:00:372
3025099100,1cyclictest131rcu_sched22:07:322
3025099100,1cyclictest131rcu_sched00:13:252
3025099100,0cyclictest0-21swapper/219:35:192
3024399109,0cyclictest7142-21sshd22:23:481
3024399109,0cyclictest0-21swapper/123:12:491
3024399109,0cyclictest0-21swapper/123:07:181
3024399108,1cyclictest10773-21lspci23:25:581
3024399108,1cyclictest0-21swapper/122:11:071
3024399100,0cyclictest0-21swapper/122:25:291
3024399100,0cyclictest0-21swapper/122:25:281
3024399100,0cyclictest0-21swapper/122:05:071
3023899109,0cyclictest0-21swapper/023:35:460
3023899109,0cyclictest0-21swapper/022:56:460
3023899109,0cyclictest0-21swapper/022:17:370
3023899109,0cyclictest0-21swapper/022:08:160
3023899109,0cyclictest0-21swapper/020:50:150
3023899109,0cyclictest0-21swapper/000:21:060
3023899108,1cyclictest646-21dbus-daemon23:17:060
3023899108,1cyclictest0-21swapper/023:26:270
3023899102,4cyclictest24855-21awk00:10:000
3023899100,0cyclictest0-21swapper/023:51:470
3023899100,0cyclictest0-21swapper/023:42:560
30821290,0sleep30-21swapper/323:14:173
302559998,0cyclictest0-21swapper/321:00:003
302509998,0cyclictest0-21swapper/219:30:192
302509997,1cyclictest33-21ksoftirqd/221:15:392
302509992,1cyclictest121rcu_preempt21:55:022
302509990,1cyclictest33-21ksoftirqd/223:11:472
302509990,1cyclictest33-21ksoftirqd/221:25:162
302439998,0cyclictest0-21swapper/123:53:281
302439998,0cyclictest0-21swapper/123:18:181
302389998,0cyclictest0-21swapper/023:22:360
302389998,0cyclictest0-21swapper/023:06:060
302389998,0cyclictest0-21swapper/022:46:560
302389998,0cyclictest0-21swapper/021:51:560
302389991,4cyclictest30053-21sshd21:15:000
29317290,4sleep00-21swapper/023:31:130
302559986,1cyclictest28959-21lspci00:17:533
302559986,1cyclictest0-21swapper/323:01:403
302559985,2cyclictest649-21avahi-daemon00:36:133
302559984,3cyclictest11506-21sshd22:51:213
302559983,2cyclictest646-21dbus-daemon21:33:493
302559983,2cyclictest646-21dbus-daemon21:16:593
302439987,0cyclictest0-21swapper/119:37:271
302439986,1cyclictest8562-21lspci00:08:111
302439986,1cyclictest7471-21sshd23:22:081
302439986,1cyclictest30685-21lspci23:41:421
302439986,1cyclictest2841-21lspci00:26:551
302389987,0cyclictest0-21swapper/023:00:370
302389987,0cyclictest0-21swapper/022:04:250
302389986,1cyclictest9-21ksoftirqd/000:39:050
302389986,1cyclictest4178-21lspci23:56:450
302389985,2cyclictest700-21gdbus23:46:240
302389985,2cyclictest649-21avahi-daemon00:25:180
302389980,4cyclictest499-21basename20:34:590
302559976,0cyclictest0-21swapper/300:05:303
302559975,1cyclictest649-21avahi-daemon00:31:133
302559975,1cyclictest19362-21sshd22:35:093
302559975,1cyclictest18562-21lspci00:20:303
302559975,1cyclictest12282-21lspci22:41:023
302559974,2cyclictest649-21avahi-daemon22:26:153
302559974,2cyclictest649-21avahi-daemon22:26:143
302559974,2cyclictest649-21avahi-daemon00:26:093
302559974,2cyclictest646-21dbus-daemon21:41:123
302559974,2cyclictest646-21dbus-daemon21:09:223
302559974,2cyclictest646-21dbus-daemon00:10:203
302559972,2cyclictest646-21dbus-daemon22:16:443
302509976,0cyclictest0-21swapper/219:43:292
302439976,0cyclictest0-21swapper/122:57:581
302439976,0cyclictest0-21swapper/121:09:071
302439976,0cyclictest0-21swapper/119:55:271
302439976,0cyclictest0-21swapper/119:55:271
302439975,1cyclictest4822-21lspci22:30:121
302439975,1cyclictest4563-21sshd22:40:151
302439975,1cyclictest15427-21lspci22:48:211
302439974,2cyclictest649-21avahi-daemon22:38:291
302439973,2cyclictest684-21polkitd21:24:271
302439972,2cyclictest646-21dbus-daemon21:35:531
302389975,1cyclictest649-21avahi-daemon21:28:270
302389975,1cyclictest29801-21lspci22:36:100
302389975,1cyclictest28077-21lspci22:32:390
302389975,1cyclictest21932-21lspci21:45:140
302389974,2cyclictest649-21avahi-daemon22:25:110
302389974,2cyclictest649-21avahi-daemon22:25:110
302389974,2cyclictest1-21systemd22:50:310
25896270,1sleep341-21ksoftirqd/322:56:153
302559965,0cyclictest0-21swapper/319:46:303
302559964,1cyclictest6225-21grep20:45:213
302559963,2cyclictest17299-21snmpd20:42:023
302559963,2cyclictest17299-21snmpd20:20:223
302559960,5cyclictest0-21swapper/319:22:303
302509964,1cyclictest33-21ksoftirqd/221:08:482
302509964,1cyclictest33-21ksoftirqd/219:15:002
302509964,1cyclictest28857-21ls20:24:592
302509960,5cyclictest0-21swapper/221:01:492
302509960,5cyclictest0-21swapper/219:25:002
302439965,0cyclictest0-21swapper/120:46:281
302389964,1cyclictest700-21gdbus19:40:010
302389964,1cyclictest646-21dbus-daemon00:30:040
302389963,2cyclictest17299-21snmpd21:06:390
302389963,2cyclictest17299-21snmpd20:44:170
302389963,2cyclictest17299-21snmpd19:54:450
302389960,5cyclictest0-21swapper/020:58:150
302389960,5cyclictest0-21swapper/019:47:150
302559954,0cyclictest0-21swapper/320:05:313
302559953,1cyclictest31624-21cat20:30:183
302559953,1cyclictest23495-21cat20:10:173
302559953,1cyclictest17299-21snmpd20:51:503
302559953,1cyclictest17299-21snmpd20:15:103
302559953,1cyclictest17299-21snmpd19:57:283
302559953,1cyclictest17299-21snmpd19:57:273
302559953,1cyclictest17299-21snmpd19:51:093
302559953,1cyclictest17299-21snmpd19:40:093
302559953,1cyclictest17299-21snmpd19:35:093
302559953,1cyclictest17299-21snmpd19:31:073
302559953,1cyclictest17299-21snmpd19:25:263
302559953,1cyclictest17299-21snmpd19:15:593
302559953,1cyclictest1264-21cat20:35:143
302559952,2cyclictest19941-21gunzip20:00:173
302559952,2cyclictest17299-21snmpd20:58:343
302559950,0cyclictest0-21swapper/319:11:203
302509953,1cyclictest33-21ksoftirqd/220:00:002
302509953,1cyclictest21459-21cat20:05:162
302509953,1cyclictest17299-21snmpd20:57:242
302509953,1cyclictest17299-21snmpd20:36:472
302509953,1cyclictest17299-21snmpd20:30:032
302509953,1cyclictest17299-21snmpd20:26:412
302509953,1cyclictest17299-21snmpd20:16:192
302509953,1cyclictest17299-21snmpd19:55:362
302509953,1cyclictest17299-21snmpd19:55:362
302509953,1cyclictest17299-21snmpd19:51:362
302509953,1cyclictest17299-21snmpd19:45:552
302509953,1cyclictest17299-21snmpd19:29:212
302509953,1cyclictest1130-21cat19:15:172
302509952,2cyclictest17299-21snmpd20:11:132
302509950,4cyclictest3511-21sed20:40:162
302509950,4cyclictest0-21swapper/220:52:092
302439954,0cyclictest0-21swapper/120:57:171
302439954,0cyclictest0-21swapper/120:21:581
302439953,1cyclictest23494-21cat20:10:171
302439953,1cyclictest17299-21snmpd21:00:211
302439953,1cyclictest17299-21snmpd20:51:261
302439953,1cyclictest17299-21snmpd20:07:111
302439953,1cyclictest17299-21snmpd19:50:331
302439953,1cyclictest17299-21snmpd19:47:131
302439953,1cyclictest17299-21snmpd19:40:511
302439953,1cyclictest17299-21snmpd19:32:541
302439953,1cyclictest17299-21snmpd19:22:091
302439953,1cyclictest17299-21snmpd19:15:561
302439953,1cyclictest17299-21snmpd19:10:441
302439952,2cyclictest29761-21cat20:25:161
302439950,0cyclictest0-21swapper/120:00:571
302389954,0cyclictest0-21swapper/020:29:350
302389954,0cyclictest0-21swapper/019:41:350
302389953,1cyclictest682-21lsmd20:08:540
302389953,1cyclictest649-21avahi-daemon19:34:040
302389953,1cyclictest5246-21gltestperf20:45:140
302389953,1cyclictest5206-21cat19:25:150
302389953,1cyclictest3184-21cat19:20:140
302389953,1cyclictest17299-21snmpd21:00:070
302389953,1cyclictest17299-21snmpd20:16:280
302389953,1cyclictest17299-21snmpd20:10:190
302389953,1cyclictest17299-21snmpd19:56:190
302389953,1cyclictest17299-21snmpd19:56:180
302389953,1cyclictest17299-21snmpd19:18:470
302389953,1cyclictest17299-21snmpd19:12:570
302389953,1cyclictest1334-21irqstats20:35:150
302439942,1cyclictest31135-21/usr/sbin/munin20:30:141
302439942,1cyclictest25141-21grep20:15:131
302439942,1cyclictest1306-21if_err_eno220:35:141
302389944,0cyclictest18676-21kworker/0:220:00:040
302389942,1cyclictest48-21kauditd20:20:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional