You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-23 - 17:21
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6.osadl.org (updated Sat Nov 23, 2024 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36169549910797,8cyclictest3683144-21/usr/sbin/munin10:35:251
36169549910597,6cyclictest3672567-21kworker/u8:1+events_unbound10:10:001
36169549910594,8cyclictest3676524-21kworker/u8:0+flush-179:010:15:221
36169549910492,9cyclictest3628264-21latency_hist07:45:011
36169539910291,9cyclictest3669908-21latency_hist09:55:010
36169549910196,4cyclictest3663533-21kworker/u8:2+events_unbound09:35:011
36169549910194,5cyclictest3603870-21kworker/u8:0+events_unbound07:25:001
36169549910192,6cyclictest3646086-21kworker/u8:0+flush-179:009:00:211
36169539910192,6cyclictest3705985-21latency_hist11:50:010
36169549910092,5cyclictest3681355-21kworker/u8:3+flush-179:010:35:001
3616953999987,9cyclictest3712610-21apt-get12:10:130
3616953999890,6cyclictest3641295-21kworker/u8:3+flush-179:008:35:210
3616953999885,9cyclictest3678160-21kworker/u8:2+flush-179:010:20:240
3616954999791,4cyclictest3652370-21kworker/u8:1+flush-179:009:15:211
3616954999783,5cyclictest3603870-21kworker/u8:0+events_unbound07:15:241
3616954999783,5cyclictest3603870-21kworker/u8:0+events_unbound07:15:241
3616954999688,6cyclictest3684552-21kworker/u8:2+flush-179:011:00:001
3616954999688,6cyclictest3632098-21kworker/u8:2+events_unbound08:35:001
3616953999686,8cyclictest3699104-21latency_hist11:25:010
3616954999585,8cyclictest3699394-21kworker/u8:3+events_unbound11:35:231
3616953999572,21cyclictest3634642-21apt-get08:05:010
3616953999572,21cyclictest3634642-21apt-get08:05:010
3616954999488,4cyclictest3705971-21kworker/u8:2+flush-179:012:00:231
3616954999481,10cyclictest3721903-21latency_hist12:40:011
3616953999473,19cyclictest3674688-21latency_hist10:10:020
3616953999384,6cyclictest3617083-21apt-get07:10:110
3616953999381,9cyclictest3684499-21latency_hist10:40:000
3616953999380,9cyclictest3666713-21kworker/u8:0+flush-179:009:45:210
3616953999280,10cyclictest3625326-21apt-get07:35:110
3616954999182,6cyclictest3697555-21kworker/u8:0+flush-179:012:05:211
3616954999182,6cyclictest3676524-21kworker/u8:0+flush-179:010:40:201
3616953999180,8cyclictest3613851-21kworker/u8:1+flush-179:007:50:190
3616953999165,23cyclictest3686104-21latency_hist10:45:010
3616954999081,6cyclictest3632098-21kworker/u8:2+events_unbound08:15:221
3616954998876,9cyclictest3715530-21latency_hist12:20:011
3616954998778,5cyclictest3613851-21kworker/u8:1+flush-179:007:25:201
3616953998780,5cyclictest3697555-21kworker/u8:0+flush-179:011:55:230
3616954998579,4cyclictest3697555-21kworker/u8:0+flush-179:011:50:211
3616954998476,5cyclictest3684552-21kworker/u8:2+flush-179:011:05:251
3616954998474,8cyclictest3652370-21kworker/u8:1+flush-179:009:09:591
3616953998472,9cyclictest3613851-21kworker/u8:1+flush-179:007:40:210
3616954998377,5cyclictest3613851-21kworker/u8:1+flush-179:007:45:281
3616954998377,4cyclictest3689568-21kworker/u8:1+events_unbound11:10:161
3616954998276,4cyclictest3710815-21kworker/u8:1+events_unbound12:30:181
3616954998275,4cyclictest3666713-21kworker/u8:0+events_unbound10:00:291
3616954998275,4cyclictest3603870-21kworker/u8:0+events_unbound07:10:131
3616954998272,8cyclictest3637018-21diskstats08:10:161
3616954998176,3cyclictest3668986-21kworker/u8:2+events_unbound09:55:231
3616954998174,4cyclictest3652370-21kworker/u8:1+events_unbound09:25:201
3616954998163,5cyclictest3699394-21kworker/u8:3+events_unbound11:40:151
3616953998170,9cyclictest3631690-21apt-get07:55:110
3616953998170,8cyclictest3635562-21kworker/u8:3+flush-179:008:05:200
3616953998170,8cyclictest3635562-21kworker/u8:3+flush-179:008:05:200
3616954998073,4cyclictest3705971-21kworker/u8:2+events_unbound11:55:151
3616953998074,4cyclictest191rcuc/011:05:000
3616953998054,7cyclictest3623437-21/usr/sbin/munin07:30:260
3616954997974,3cyclictest3699394-21kworker/u8:3+flush-179:011:30:181
3616954997973,4cyclictest3641295-21kworker/u8:3+flush-179:008:25:011
3616953997867,9cyclictest3711026-21apt-get12:05:130
3616954997771,5cyclictest3676524-21kworker/u8:0+flush-179:010:25:011
3616954997771,4cyclictest3684552-21kworker/u8:2+flush-179:011:15:211
3616954997771,4cyclictest3613851-21kworker/u8:1+flush-179:008:05:001
3616954997771,4cyclictest3613851-21kworker/u8:1+flush-179:008:05:001
3616954997769,5cyclictest3641295-21kworker/u8:3+events_unbound08:35:131
3616954997767,7cyclictest3684552-21kworker/u8:2+events_unbound11:20:201
3616954997760,5cyclictest3717118-21kworker/u8:0+events_unbound12:25:211
3616953997770,5cyclictest3632098-21kworker/u8:2+flush-179:008:44:540
3616954997670,4cyclictest3652370-21kworker/u8:1+events_unbound09:20:191
3616953997665,8cyclictest3105085-21systemd-journal09:14:590
3616954997569,4cyclictest3641295-21kworker/u8:3+flush-179:008:55:201
3616954997569,4cyclictest3641295-21kworker/u8:3+flush-179:008:55:201
3616953997567,6cyclictest3646086-21kworker/u8:0+flush-179:008:50:010
3616953997464,8cyclictest3652370-21kworker/u8:1+flush-179:009:25:200
3616954997360,10cyclictest3641498-21/usr/sbin/munin08:25:231
3616953997362,8cyclictest3613851-21kworker/u8:1+flush-179:007:25:200
3616954997266,4cyclictest3652370-21kworker/u8:1+flush-179:009:35:211
3616954997264,6cyclictest3705971-21kworker/u8:2+flush-179:012:20:221
3616953997263,6cyclictest3699394-21kworker/u8:3+flush-179:011:40:150
3616953997263,6cyclictest3684552-21kworker/u8:2+events_unbound11:20:010
3616954997166,4cyclictest3684552-21kworker/u8:2+flush-179:011:00:261
3616954997160,8cyclictest3713940-21latency_hist12:15:011
3616953997164,5cyclictest3632098-21kworker/u8:2+flush-179:008:30:230
3616954997066,3cyclictest3668986-21kworker/u8:2+flush-179:010:10:201
3616953997058,10cyclictest3666713-21kworker/u8:0+flush-179:009:45:000
3616954996959,7cyclictest3105085-21systemd-journal08:05:031
3616954996959,7cyclictest3105085-21systemd-journal08:05:021
3616954996763,3cyclictest3663533-21kworker/u8:2+flush-179:009:40:251
3616953996762,4cyclictest3672567-21kworker/u8:1+flush-179:010:04:550
3616953996761,4cyclictest3652370-21kworker/u8:1+flush-179:009:20:220
3616954996660,4cyclictest3632098-21kworker/u8:2+events_unbound07:55:231
3616953996654,9cyclictest3689533-21latency_hist10:55:010
3616954996558,5cyclictest3668986-21kworker/u8:2+flush-179:009:50:301
3616954996557,5cyclictest3613851-21kworker/u8:1+events_unbound07:30:251
3616954996554,9cyclictest3632098-21kworker/u8:2+flush-179:008:50:251
3616953996557,6cyclictest3714177-21/usr/sbin/munin12:15:180
3616954996457,6cyclictest3646086-21kworker/u8:0+flush-179:009:15:001
3616953996455,7cyclictest3105085-21systemd-journal07:20:000
3616953996455,7cyclictest3105085-21systemd-journal07:20:000
3616954996359,3cyclictest3603870-21kworker/u8:0+flush-179:007:35:251
3616954996358,3cyclictest3676524-21kworker/u8:0+flush-179:010:25:201
3616954996357,5cyclictest3684552-21kworker/u8:2+flush-179:010:45:231
3616953996352,8cyclictest3702807-21latency_hist11:40:010
3616953996352,7cyclictest3705971-21kworker/u8:2+events_unbound12:20:190
3616959996259,2cyclictest0-21swapper/209:18:592
3616953996252,7cyclictest3689568-21kworker/u8:1+events_unbound11:10:230
3616954996156,3cyclictest3613851-21kworker/u8:1+events_unbound07:50:121
3616953996154,5cyclictest3632098-21kworker/u8:2+flush-179:008:25:160
3616953996154,5cyclictest3632098-21kworker/u8:2+events_unbound08:10:260
3616953996148,10cyclictest3658759-21latency_hist09:20:010
3616954996055,3cyclictest3702782-21kworker/u8:1+flush-179:011:45:151
3616953996051,7cyclictest3684552-21kworker/u8:2+flush-179:010:45:250
3616953996051,7cyclictest3681574-21apt-get10:30:140
3616953996049,9cyclictest3105085-21systemd-journal10:20:000
3616960995855,2cyclictest0-21swapper/309:19:003
3616953995852,5cyclictest3613851-21kworker/u8:1+flush-179:007:45:230
3616954995751,4cyclictest3684552-21kworker/u8:2+flush-179:010:50:241
3616953995751,4cyclictest3652370-21kworker/u8:1+events_unbound09:30:280
3616953995748,7cyclictest3650803-21latency_hist08:55:010
3616954995649,5cyclictest3641295-21kworker/u8:3+flush-179:008:45:211
3616953995550,4cyclictest3613851-21kworker/u8:1+flush-179:007:24:560
3616953995445,7cyclictest3655583-21latency_hist09:10:010
3616953995346,6cyclictest3646086-21kworker/u8:0+flush-179:009:00:250
3616953995345,7cyclictest3684552-21kworker/u8:2+events_unbound11:05:170
3616954995243,7cyclictest3616800-21latency_hist07:10:021
3616953995246,5cyclictest3669944-21kworker/u8:3+flush-179:009:59:290
3616953995245,5cyclictest3676524-21kworker/u8:0+flush-179:010:25:180
3616953995148,2cyclictest0-21swapper/008:20:590
3616953995142,7cyclictest3700947-21latency_hist11:35:010
4505291490,14ptp4l61790irq/134-eno1-TxRx-007:10:003
3616953994943,4cyclictest3710815-21kworker/u8:1+flush-179:012:30:240
3616953994842,5cyclictest3652370-21kworker/u8:1+events_unbound09:35:250
3616954994642,3cyclictest3641295-21kworker/u8:3+flush-179:008:40:251
3616953994640,5cyclictest3717118-21kworker/u8:0+flush-179:012:25:220
3616954994540,3cyclictest3668327-21munin-run09:50:001
3616953994540,4cyclictest3632098-21kworker/u8:2+events_unbound08:15:250
361695399451,42cyclictest3699362-21idleruntime-cro11:29:590
3616954994439,4cyclictest3699398-21systemd-run11:30:011
3616953994439,4cyclictest3705971-21kworker/u8:2+flush-179:012:00:150
3616953994433,8cyclictest3691141-21sh11:00:010
361696099430,4cyclictest3618437-21turbostat07:15:003
3616953994337,5cyclictest3717679-21kworker/u8:3+flush-179:012:39:300
361695999420,39cyclictest0-21swapper/207:39:552
361695399420,40cyclictest722-21Xorg10:14:590
3616953994136,4cyclictest3646086-21kworker/u8:0+flush-179:008:55:210
3616953994136,4cyclictest3646086-21kworker/u8:0+flush-179:008:55:200
3616960994038,1cyclictest0-21swapper/307:39:553
361696099400,4cyclictest3623210-21turbostat07:30:003
361696099400,38cyclictest3681335-21turbostat10:30:013
3616959994036,2cyclictest0-21swapper/210:52:582
3616960993932,5cyclictest3665166-21turbostat09:40:013
361696099390,37cyclictest3713960-21turbostat12:15:003
361696099380,37cyclictest3620025-21turbostat07:20:023
361696099380,37cyclictest3620025-21turbostat07:20:013
3616959993835,2cyclictest0-21swapper/209:44:592
3616959993834,2cyclictest0-21swapper/210:00:592
361695999380,36cyclictest0-21swapper/211:52:592
361695999380,36cyclictest0-21swapper/211:35:592
361695999380,36cyclictest0-21swapper/210:20:592
361695999380,36cyclictest0-21swapper/209:34:592
3616960993734,2cyclictest0-21swapper/308:30:593
3616960993733,2cyclictest0-21swapper/311:40:593
3616960993733,2cyclictest0-21swapper/310:21:003
3616960993732,4cyclictest3663546-21turbostat09:35:003
361696099370,36cyclictest0-21swapper/310:08:593
3616959993735,1cyclictest0-21swapper/211:15:002
3616959993734,2cyclictest0-21swapper/210:37:582
3616959993734,2cyclictest0-21swapper/209:39:002
3616959993733,2cyclictest0-21swapper/207:58:592
361695999370,1cyclictest0-21swapper/211:57:592
361695399370,36cyclictest0-21swapper/011:51:590
3616960993634,1cyclictest0-21swapper/311:35:583
3616960993633,2cyclictest0-21swapper/311:52:593
3616960993631,4cyclictest3717155-21turbostat12:25:003
361696099360,35cyclictest0-21swapper/311:14:553
361696099360,35cyclictest0-21swapper/309:29:553
3616959993632,2cyclictest0-21swapper/207:44:552
361695999360,1cyclictest0-21swapper/212:21:592
361695999360,1cyclictest0-21swapper/211:15:592
361695999360,1cyclictest0-21swapper/208:30:582
4505291350,3ptp4l441ktimers/311:20:133
3616959993532,2cyclictest0-21swapper/208:10:002
3616959993532,2cyclictest0-21swapper/208:10:002
3616959993531,2cyclictest0-21swapper/211:05:582
3616960993432,1cyclictest0-21swapper/310:52:583
3616960993431,2cyclictest0-21swapper/309:59:543
3616959993430,2cyclictest0-21swapper/210:43:592
3616960993330,2cyclictest0-21swapper/309:44:583
3616960993330,2cyclictest0-21swapper/307:30:593
3616960993328,4cyclictest3636485-21turbostat08:10:003
3616960993328,4cyclictest3636485-21turbostat08:10:003
3616959993331,1cyclictest0-21swapper/212:35:002
3616959993330,2cyclictest0-21swapper/209:58:592
3616959993329,2cyclictest0-21swapper/209:10:592
3616960993231,1cyclictest0-21swapper/312:19:553
3616960993230,1cyclictest0-21swapper/310:37:583
3616960993227,4cyclictest3720326-21turbostat12:35:003
3616959993229,2cyclictest0-21swapper/212:08:592
3616959993229,2cyclictest0-21swapper/212:00:592
3616959993229,2cyclictest0-21swapper/210:56:592
3616959993229,2cyclictest0-21swapper/208:15:002
3616959993228,2cyclictest0-21swapper/208:00:592
3616959993228,2cyclictest0-21swapper/208:00:582
3616960993129,1cyclictest0-21swapper/310:00:593
3616960993125,4cyclictest3638077-21turbostat08:15:003
3616959993129,1cyclictest0-21swapper/210:15:002
3616959993129,1cyclictest0-21swapper/209:10:002
3616959993127,2cyclictest0-21swapper/208:54:552
361695999310,29cyclictest0-21swapper/211:45:132
4505291300,6ptp4l441ktimers/308:49:563
3616960993028,1cyclictest0-21swapper/307:58:593
3616960993024,5cyclictest3686122-21turbostat10:45:013
3616959993027,2cyclictest0-21swapper/212:10:152
3616959993027,2cyclictest0-21swapper/210:15:152
3616959993027,2cyclictest0-21swapper/208:20:592
3616959993027,2cyclictest0-21swapper/207:52:592
3616959993026,2cyclictest0-21swapper/212:36:592
4505291290,7ptp4l441ktimers/312:05:003
3616960992927,1cyclictest0-21swapper/311:45:133
3616960992927,1cyclictest0-21swapper/307:44:553
3616960992926,2cyclictest0-21swapper/311:05:583
3616960992926,2cyclictest0-21swapper/310:55:143
3616960992926,2cyclictest0-21swapper/310:45:123
3616960992924,4cyclictest3655588-21turbostat09:10:003
3616960992921,5cyclictest3699372-21turbostat11:29:593
361696099290,28cyclictest0-21swapper/307:45:153
361696099290,27cyclictest0-21swapper/312:35:163
361696099290,1cyclictest0-21swapper/310:15:143
3616959992926,2cyclictest0-21swapper/209:00:142
3616959992925,2cyclictest0-21swapper/212:30:002
3616959992925,2cyclictest0-21swapper/211:33:592
361695999290,2cyclictest0-21swapper/209:50:172
361695999290,27cyclictest0-21swapper/211:40:122
361695999290,27cyclictest0-21swapper/210:45:132
361695999290,27cyclictest0-21swapper/208:35:122
361695999290,27cyclictest0-21swapper/207:45:152
4505291280,7ptp4l441ktimers/311:05:003
3616960992826,1cyclictest0-21swapper/309:23:003
3616960992825,2cyclictest0-21swapper/312:07:003
3616960992825,2cyclictest0-21swapper/311:58:593
361696099280,27cyclictest0-21swapper/310:30:163
361696099280,27cyclictest0-21swapper/310:10:023
361696099280,27cyclictest0-21swapper/309:00:143
361696099280,26cyclictest0-21swapper/309:50:183
3616959992825,2cyclictest0-21swapper/210:30:172
3616959992825,2cyclictest0-21swapper/207:10:122
3616959992823,2cyclictest0-21swapper/208:26:002
361695999280,26cyclictest0-21swapper/208:55:592
361695999280,26cyclictest0-21swapper/208:55:592
4505291270,6ptp4l61790irq/134-eno1-TxRx-008:20:013
4505291270,6ptp4l441ktimers/307:55:013
3616960992725,1cyclictest0-21swapper/312:29:593
3616960992724,2cyclictest3647645-21turbostat08:44:593
3616960992724,2cyclictest0-21swapper/309:10:583
3616960992724,2cyclictest0-21swapper/308:55:593
3616960992724,2cyclictest0-21swapper/308:55:583
361696099270,26cyclictest0-21swapper/308:20:593
361696099270,25cyclictest0-21swapper/308:35:593
361696099270,1cyclictest0-21swapper/308:00:593
361696099270,1cyclictest0-21swapper/308:00:583
3616959992725,1cyclictest0-21swapper/211:29:592
3616959992723,2cyclictest0-21swapper/211:22:592
361695999270,25cyclictest0-21swapper/208:15:592
361695999270,24cyclictest0-21swapper/208:49:582
361695999270,1cyclictest0-21swapper/209:23:002
4505291260,6ptp4l61790irq/134-eno1-TxRx-011:35:013
4505291260,6ptp4l61790irq/134-eno1-TxRx-007:25:003
3616959992624,1cyclictest0-21swapper/207:24:592
361695999260,24cyclictest0-21swapper/207:31:592
4505291250,5ptp4l441ktimers/308:29:563
3616960992523,1cyclictest0-21swapper/308:52:003
3616959992522,2cyclictest0-21swapper/207:27:592
3616959992521,2cyclictest0-21swapper/207:15:102
3616959992521,2cyclictest0-21swapper/207:15:102
201699250,12rtkit-daemon3615800-21df_inode07:05:170
3616959992421,2cyclictest0-21swapper/212:15:142
3616959992421,2cyclictest0-21swapper/210:29:002
361695999240,1cyclictest0-21swapper/208:45:002
4505291230,4ptp4l61790irq/134-eno1-TxRx-011:19:563
3616959992219,2cyclictest0-21swapper/209:46:592
3616959992017,2cyclictest0-21swapper/210:06:592
3616959991916,2cyclictest0-21swapper/209:26:592
3616959991714,2cyclictest0-21swapper/211:04:002
4505291160,9ptp4l61790irq/134-eno1-TxRx-009:46:023
36169599930,2cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional