You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 08:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot6.osadl.org (updated Wed Apr 02, 2025 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
362568699146139,5cyclictest3676891-21kworker/u8:0+events_unbound22:25:020
362568699116106,7cyclictest3608112-21kworker/u8:0+events_unbound19:10:020
362569199114101,10cyclictest3666582-21latency_hist21:30:001
36256919911198,5cyclictest3697089-21kworker/u8:0+events_unbound23:40:241
362569199110104,4cyclictest3632115-21kworker/u8:2+flush-179:019:35:241
36256919910996,11cyclictest3705783-40logrotate00:00:011
36256869910698,7cyclictest3656562-21kworker/u8:3+events_unbound21:20:260
36256869910698,7cyclictest3656562-21kworker/u8:3+events_unbound21:20:250
36256869910694,4cyclictest3638476-21kworker/u8:0+flush-179:019:50:240
36256869910599,4cyclictest3635290-21kworker/u8:1+events_unbound19:45:230
36256919910497,6cyclictest3682190-21kworker/u8:2+flush-179:022:40:251
36256869910488,6cyclictest3656562-21kworker/u8:3+events_unbound21:00:150
36256869910488,6cyclictest3656562-21kworker/u8:3+events_unbound21:00:150
36256919910290,6cyclictest3673506-21kworker/u8:3+flush-179:022:00:001
36256869910290,4cyclictest3651997-21kworker/u8:1+events_unbound20:40:160
36256919910193,6cyclictest3656562-21kworker/u8:3+events_unbound21:30:021
36256919910193,5cyclictest3608112-21kworker/u8:0+events_unbound19:30:251
36256919910092,6cyclictest3656562-21kworker/u8:3+events_unbound21:20:131
36256919910092,6cyclictest3656562-21kworker/u8:3+events_unbound21:20:121
3625691999991,5cyclictest3643259-21kworker/u8:0+events_unbound20:20:251
3625686999994,4cyclictest3682190-21kworker/u8:2+events_unbound00:10:200
3625691999887,8cyclictest3625482-21latency_hist19:10:011
3625686999791,4cyclictest3682190-21kworker/u8:2+flush-179:023:10:160
3625686999785,5cyclictest3656562-21kworker/u8:3+events_unbound21:05:230
3625691999688,5cyclictest3697089-21kworker/u8:0+flush-179:023:30:111
3625686999685,8cyclictest3682190-21kworker/u8:2+events_unbound22:35:120
3625686999685,8cyclictest3682190-21kworker/u8:2+events_unbound22:35:120
3625686999591,3cyclictest3638476-21kworker/u8:0+events_unbound19:55:250
3625691999285,5cyclictest3661271-21kworker/u8:0+flush-179:021:10:241
3625691999279,4cyclictest3638731-21kworker/u8:2+events_unbound20:00:001
3625686999287,4cyclictest3692312-21kworker/u8:3+events_unbound23:20:240
3625691999180,9cyclictest3702389-21gpgv23:50:011
3625691999079,7cyclictest3676891-21kworker/u8:0+flush-179:022:10:171
362568699900,88cyclictest1-21systemd21:40:010
3625691998880,6cyclictest3692516-21/usr/sbin/munin23:10:191
3625686998882,4cyclictest3697089-21kworker/u8:0+events_unbound00:00:000
3625691998780,5cyclictest3635290-21kworker/u8:1+events_unbound19:45:121
3625686998779,6cyclictest3661271-21kworker/u8:0+events_unbound21:15:020
3625686998777,8cyclictest3697089-21kworker/u8:0+events_unbound23:50:140
3625686998777,8cyclictest3697089-21kworker/u8:0+events_unbound23:50:140
3625691998680,4cyclictest3638731-21kworker/u8:2+events_unbound21:05:161
3625691998678,6cyclictest3668410-21kworker/u8:0+flush-179:021:40:001
3625691998678,5cyclictest3668410-21kworker/u8:0+events_unbound21:50:091
3625691998577,5cyclictest3682190-21kworker/u8:2+flush-179:000:10:161
3625686998580,4cyclictest3710446-21kworker/u8:0+events_unbound00:20:250
3625691998479,4cyclictest3712724-21kworker/u8:2+flush-179:000:35:231
3625691998478,4cyclictest3692312-21kworker/u8:3+events_unbound23:20:171
3625691998476,6cyclictest3682190-21kworker/u8:2+flush-179:022:55:251
3625686998480,3cyclictest3608112-21kworker/u8:0+events_unbound19:20:240
3625686998475,7cyclictest3656562-21kworker/u8:3+events_unbound21:10:190
3625691998375,6cyclictest3694100-21kworker/u8:1+events_unbound23:15:221
3625691998374,7cyclictest502-21rs:main1
3625686998376,6cyclictest3692312-21kworker/u8:3+events_unbound23:30:210
3625686998376,5cyclictest3638731-21kworker/u8:2+flush-179:020:35:240
3625686998376,5cyclictest3638731-21kworker/u8:2+flush-179:020:35:240
3625686998373,7cyclictest3671619-21latency_hist21:45:010
3625691998269,10cyclictest3712994-21apt-get00:30:131
3625691998175,5cyclictest3661271-21kworker/u8:0+events_unbound21:15:241
3625686998175,4cyclictest3676891-21kworker/u8:0+events_unbound22:10:120
3625686998170,9cyclictest3675305-21/usr/sbin/munin22:00:220
3625691998072,5cyclictest3682190-21kworker/u8:2+events_unbound00:00:231
3625686998072,5cyclictest3651997-21kworker/u8:1+events_unbound20:45:140
3625686998071,6cyclictest3641650-21kworker/u8:3+events_unbound20:00:140
3625691997974,4cyclictest3638731-21kworker/u8:2+events_unbound20:10:231
3625691997972,5cyclictest3651997-21kworker/u8:1+flush-179:020:45:211
3625691997869,6cyclictest3673506-21kworker/u8:3+events_unbound22:20:191
3625691997867,8cyclictest3627354-21latency_hist19:15:011
3625691997867,4cyclictest3682190-21kworker/u8:2+events_unbound22:30:201
3625686997867,3cyclictest3651997-21kworker/u8:1+events_unbound20:50:300
3625691997772,3cyclictest3608112-21kworker/u8:0+events_unbound19:25:241
3625691997770,6cyclictest3638476-21kworker/u8:0+events_unbound19:55:011
3625686997774,2cyclictest3682190-21kworker/u8:2+events_unbound00:00:250
3625691997670,4cyclictest3710446-21kworker/u8:0+flush-179:000:20:221
3625691997669,5cyclictest3676947-21kworker/u8:1+flush-179:022:15:201
3625691997663,5cyclictest3656562-21kworker/u8:3+events_unbound20:55:211
3625686997665,8cyclictest3636911-21latency_hist19:45:000
3625691997570,4cyclictest3638731-21kworker/u8:2+events_unbound20:35:241
3625691997570,4cyclictest3638731-21kworker/u8:2+events_unbound20:35:241
3625691997566,4cyclictest3635290-21kworker/u8:1+events_unbound19:40:271
3625691997564,8cyclictest3645369-21apt-get20:15:131
3625686997571,3cyclictest3680063-21kworker/u8:2+events_unbound22:20:210
3625686997568,5cyclictest3682259-21kworker/u8:3+events_unbound22:40:130
3625691997467,5cyclictest3638731-21kworker/u8:2+flush-179:020:35:001
3625686997468,4cyclictest3666358-21kworker/u8:2+flush-179:021:55:230
3625686997467,5cyclictest3712724-21kworker/u8:2+events_unbound00:35:140
3625691997366,5cyclictest3704158-21kworker/u8:1+flush-179:000:15:241
3625691997366,5cyclictest3651997-21kworker/u8:1+flush-179:020:50:291
3625686997369,3cyclictest3630529-21kworker/u8:1+events_unbound19:25:130
3625686997367,5cyclictest3688872-21kworker/u8:0+events_unbound23:00:200
3625691997266,5cyclictest3697089-21kworker/u8:0+flush-179:023:35:211
3625686997266,5cyclictest3694100-21kworker/u8:1+flush-179:023:15:220
3625686997165,5cyclictest3656562-21kworker/u8:3+events_unbound21:30:190
3625686997165,4cyclictest3697089-21kworker/u8:0+flush-179:023:35:160
3625691996963,5cyclictest3666358-21kworker/u8:2+events_unbound22:00:251
3625686996962,5cyclictest3643259-21kworker/u8:0+events_unbound20:15:130
3625691996859,4cyclictest3638731-21kworker/u8:2+events_unbound20:45:001
3625691996760,6cyclictest3697089-21kworker/u8:0+events_unbound23:50:021
3625691996760,6cyclictest3697089-21kworker/u8:0+events_unbound23:50:011
3625691996758,7cyclictest3692022-21latency_hist23:05:011
3625686996759,6cyclictest3608112-21kworker/u8:0+events_unbound19:35:130
3625691996662,3cyclictest3638731-21kworker/u8:2+flush-179:020:00:281
3625686996661,4cyclictest3608112-21kworker/u8:0+events_unbound19:18:230
3625686996561,3cyclictest3676947-21kworker/u8:1+events_unbound22:15:210
3625691996459,3cyclictest3692312-21kworker/u8:3+events_unbound23:27:231
3625686996460,3cyclictest3676891-21kworker/u8:0+events_unbound22:30:100
3625686996459,4cyclictest3682190-21kworker/u8:2+flush-179:022:55:210
3625686996456,7cyclictest3704158-21kworker/u8:1+events_unbound00:30:130
3625686996352,5cyclictest3643259-21kworker/u8:0+events_unbound20:10:200
3625691996257,4cyclictest3682190-21kworker/u8:2+flush-179:022:35:241
3625691996257,4cyclictest3682190-21kworker/u8:2+flush-179:022:35:241
3625691996256,4cyclictest3664704-21kworker/u8:1+events_unbound21:45:001
3625686996257,4cyclictest3710446-21kworker/u8:0+events_unbound00:17:000
3625686996257,4cyclictest3682259-21kworker/u8:3+flush-179:022:49:550
3625686996257,4cyclictest3668410-21kworker/u8:0+flush-179:021:50:160
3625686996255,5cyclictest3666620-21gpgv21:30:000
3625686996155,5cyclictest3643259-21kworker/u8:0+events_unbound20:20:220
3625686996151,8cyclictest3702348-21latency_hist23:50:010
3625686996149,10cyclictest3687498-21apt-get22:50:080
3625686996055,4cyclictest3673506-21kworker/u8:3+flush-179:022:09:560
3625686996048,9cyclictest3632115-21kworker/u8:2+flush-179:019:35:000
3625693995855,2cyclictest0-21swapper/222:45:592
3625686995853,4cyclictest3643259-21kworker/u8:0+flush-179:020:29:560
3625697995654,1cyclictest0-21swapper/322:45:593
3625691995651,4cyclictest3627008-21kworker/u8:1+flush-179:019:15:251
3625691995650,4cyclictest3676891-21kworker/u8:0+events_unbound22:05:241
3625691995649,4cyclictest3105085-21systemd-journal00:25:231
3625693995449,3cyclictest0-21swapper/200:26:592
3625691995449,4cyclictest3630529-21kworker/u8:1+events_unbound19:23:221
3625686995449,4cyclictest3671909-21systemd-run21:50:000
3625686995449,4cyclictest3671909-21systemd-run21:50:000
3625691995348,4cyclictest3656562-21kworker/u8:3+events_unbound21:01:241
3625691995348,4cyclictest3656562-21kworker/u8:3+events_unbound21:01:231
3625697995250,1cyclictest0-21swapper/300:26:593
3625691995247,4cyclictest3704158-21kworker/u8:1+events_unbound00:07:251
4505291490,14ptp4l61790irq/134-eno1-TxRx-019:09:563
3625686994942,5cyclictest3643530-21latency_hist20:10:020
3625691994842,4cyclictest3682259-21kworker/u8:3+flush-179:022:48:251
3625686994843,4cyclictest3682190-21kworker/u8:2+events_unbound23:26:240
3625691994640,5cyclictest3668410-21kworker/u8:0+events_unbound21:49:221
3625691994640,5cyclictest3668410-21kworker/u8:0+events_unbound21:49:221
3625686994540,4cyclictest3685436-21kworker/u8:1+events_unbound23:07:240
3625686994540,4cyclictest3656562-21kworker/u8:3+events_unbound20:55:200
3625686994439,4cyclictest3643259-21kworker/u8:0+flush-179:020:30:220
3625697994235,5cyclictest3702360-21turbostat23:50:003
3625691994137,3cyclictest3682190-21kworker/u8:2+events_unbound22:28:231
3625686994136,4cyclictest3697089-21kworker/u8:0+flush-179:023:40:250
3625686994133,6cyclictest3105085-21systemd-journal00:30:010
3625697994034,5cyclictest3715916-21turbostat00:40:013
3625697994034,4cyclictest3705794-21turbostat00:04:553
362569199400,38cyclictest3648320-21turbostat20:29:551
3625686994036,3cyclictest3704158-21kworker/u8:1+events_unbound00:09:240
362569399390,36cyclictest0-21swapper/219:14:002
362569799380,37cyclictest0-21swapper/300:16:583
362569799380,1cyclictest0-21swapper/322:58:003
3625693993834,2cyclictest0-21swapper/200:04:542
362569399380,35cyclictest0-21swapper/222:23:592
362569399380,35cyclictest0-21swapper/219:16:592
362569399380,35cyclictest0-21swapper/200:16:592
3625691993835,2cyclictest0-21swapper/120:05:591
3625693993733,2cyclictest0-21swapper/220:58:592
3625693993733,2cyclictest0-21swapper/220:53:592
3625697993634,1cyclictest0-21swapper/322:24:553
3625697993634,1cyclictest0-21swapper/320:53:593
3625697993633,2cyclictest0-21swapper/323:28:593
3625697993633,2cyclictest0-21swapper/321:51:593
362569799360,35cyclictest0-21swapper/319:39:593
362569799360,35cyclictest0-21swapper/319:17:003
362569799360,35cyclictest0-21swapper/319:13:593
3625693993634,1cyclictest0-21swapper/200:40:002
3625693993633,2cyclictest0-21swapper/223:28:592
3625693993633,2cyclictest0-21swapper/221:51:592
3625697993532,2cyclictest0-21swapper/320:28:003
3625693993531,2cyclictest0-21swapper/222:18:592
3625693993531,2cyclictest0-21swapper/221:09:552
3625693993531,2cyclictest0-21swapper/220:27:592
362569399350,33cyclictest0-21swapper/219:39:582
362569399350,33cyclictest0-21swapper/200:20:582
362569399350,1cyclictest0-21swapper/222:58:002
3625697993432,1cyclictest0-21swapper/322:29:593
3625697993432,1cyclictest0-21swapper/322:18:593
3625697993432,1cyclictest0-21swapper/320:59:003
362569799340,32cyclictest0-21swapper/300:20:593
3625693993431,2cyclictest0-21swapper/220:06:582
3625693993431,2cyclictest0-21swapper/219:44:552
3625693993430,2cyclictest0-21swapper/222:25:592
3625691993430,3cyclictest3682190-21kworker/u8:2+flush-179:023:09:561
3625697993331,1cyclictest0-21swapper/320:06:583
3625697993331,1cyclictest0-21swapper/319:44:553
3625693993329,2cyclictest0-21swapper/223:57:002
3625697993230,1cyclictest0-21swapper/321:09:553
3625693993229,2cyclictest0-21swapper/222:38:592
3625693993229,2cyclictest0-21swapper/222:38:592
3625693993229,2cyclictest0-21swapper/220:13:592
3625693993228,2cyclictest0-21swapper/221:01:592
3625693993228,2cyclictest0-21swapper/221:01:582
362569399320,30cyclictest0-21swapper/223:15:132
362569399320,30cyclictest0-21swapper/221:21:592
362569399320,30cyclictest0-21swapper/221:21:592
362569399320,30cyclictest0-21swapper/220:20:132
3625697993129,1cyclictest0-21swapper/323:56:593
3625697993129,1cyclictest0-21swapper/322:38:593
3625697993129,1cyclictest0-21swapper/322:38:593
362569799310,30cyclictest0-21swapper/321:22:003
362569799310,30cyclictest0-21swapper/321:21:593
362569799310,29cyclictest0-21swapper/323:30:093
362569799310,29cyclictest0-21swapper/323:15:143
362569799310,29cyclictest0-21swapper/319:20:143
362569799310,23cyclictest0-21swapper/320:20:133
3625693993128,2cyclictest0-21swapper/223:34:562
3625693993128,2cyclictest0-21swapper/220:39:562
3625693993128,2cyclictest0-21swapper/220:39:552
3625693993127,2cyclictest0-21swapper/221:10:592
362569399310,30cyclictest0-21swapper/219:20:142
3625697993028,1cyclictest0-21swapper/321:01:583
3625697993028,1cyclictest0-21swapper/321:01:583
3625697993028,1cyclictest0-21swapper/320:39:553
3625697993028,1cyclictest0-21swapper/320:39:553
3625697993028,1cyclictest0-21swapper/320:14:003
362569799300,29cyclictest0-21swapper/323:40:123
362569399300,27cyclictest0-21swapper/221:40:142
362569399300,27cyclictest0-21swapper/221:36:592
4505291290,6ptp4l441ktimers/319:45:013
3625697992927,1cyclictest0-21swapper/321:11:003
3625697992927,1cyclictest0-21swapper/300:34:583
3625697992926,2cyclictest0-21swapper/319:55:143
3625697992925,2cyclictest0-21swapper/320:30:103
3625693992926,2cyclictest0-21swapper/223:10:092
3625693992926,2cyclictest0-21swapper/222:40:152
3625693992926,2cyclictest0-21swapper/219:50:192
3625693992925,2cyclictest0-21swapper/222:50:092
3625693992925,2cyclictest0-21swapper/220:30:102
3625693992925,2cyclictest0-21swapper/220:17:592
362569399290,2cyclictest0-21swapper/200:30:142
362569399290,27cyclictest0-21swapper/223:09:592
362569399290,27cyclictest0-21swapper/220:45:152
362569399290,27cyclictest0-21swapper/219:55:142
362569399290,22cyclictest0-21swapper/223:40:122
362569399290,21cyclictest0-21swapper/223:50:022
362569399290,21cyclictest0-21swapper/223:50:012
3625697992826,1cyclictest0-21swapper/322:40:143
3625697992826,1cyclictest0-21swapper/320:17:583
3625697992826,1cyclictest0-21swapper/319:50:203
3625697992825,2cyclictest0-21swapper/323:10:093
362569799280,27cyclictest0-21swapper/323:00:133
362569799280,27cyclictest0-21swapper/322:50:093
362569799280,27cyclictest0-21swapper/300:10:003
362569799280,26cyclictest0-21swapper/321:40:133
362569799280,26cyclictest0-21swapper/320:45:143
362569799280,21cyclictest0-21swapper/323:50:033
362569799280,21cyclictest0-21swapper/323:50:033
3625693992825,2cyclictest0-21swapper/223:00:132
362569399280,26cyclictest0-21swapper/200:10:002
362569399280,25cyclictest0-21swapper/223:23:592
362569399280,25cyclictest0-21swapper/219:30:162
4505291270,7ptp4l441ktimers/321:19:553
4505291270,6ptp4l61790irq/134-eno1-TxRx-000:15:013
4505291270,6ptp4l441ktimers/322:10:023
4505291270,6ptp4l441ktimers/322:00:013
4505291270,5ptp4l441ktimers/319:30:003
3625697992725,1cyclictest0-21swapper/319:34:583
362569799270,26cyclictest0-21swapper/323:09:593
362569799270,1cyclictest0-21swapper/321:37:003
3625693992724,2cyclictest0-21swapper/222:11:592
3625693992724,2cyclictest0-21swapper/221:30:582
3625693992724,2cyclictest0-21swapper/219:48:002
3625693992723,2cyclictest0-21swapper/219:29:562
4505291260,6ptp4l61790irq/134-eno1-TxRx-022:35:013
4505291260,6ptp4l61790irq/134-eno1-TxRx-021:30:003
4505291260,6ptp4l61790irq/134-eno1-TxRx-020:44:573
4505291260,5ptp4l441ktimers/320:04:553
3625697992624,1cyclictest0-21swapper/322:11:593
3625693992622,2cyclictest0-21swapper/221:16:002
362569399260,24cyclictest0-21swapper/220:02:002
4505291250,5ptp4l61790irq/134-eno1-TxRx-021:49:563
4505291250,5ptp4l61790irq/134-eno1-TxRx-021:49:563
4505291250,5ptp4l441ktimers/323:40:003
3625697992523,1cyclictest0-21swapper/321:30:583
362569799250,24cyclictest0-21swapper/323:23:593
3625693992522,2cyclictest0-21swapper/200:12:592
3625693992521,2cyclictest0-21swapper/220:41:592
4505291240,5ptp4l441ktimers/322:04:563
3625693992420,2cyclictest0-21swapper/221:55:162
3625693992320,2cyclictest0-21swapper/222:32:592
3625686992219,2cyclictest3625768-21latency_hist19:10:020
3625693992118,2cyclictest0-21swapper/221:46:582
3625693992118,2cyclictest0-21swapper/221:46:582
3625693992017,2cyclictest0-21swapper/223:35:592
3625693992016,2cyclictest0-21swapper/222:07:592
3625693991815,2cyclictest0-21swapper/223:46:592
3625693991714,2cyclictest0-21swapper/222:04:552
3625693991613,2cyclictest0-21swapper/221:30:012
36256939941,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional