You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-16 - 08:26
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Mon Jul 15, 2024 12:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
98997599152133,5cyclictest1025720-21kworker/u8:1+events_unbound09:12:141
9899759910395,6cyclictest1009411-21kworker/u8:2+flush-179:008:40:001
9899759910390,9cyclictest1015375-21latency_hist08:30:021
9899759910295,4cyclictest1056619-21kworker/u8:1+events_unbound11:00:011
9899719910185,12cyclictest2791555-21systemd-journal12:05:000
989975999689,5cyclictest1025720-21kworker/u8:1+flush-179:009:05:011
989975999587,5cyclictest1042690-21kworker/u8:0+flush-179:010:00:001
989975999384,6cyclictest1074791-21kworker/u8:2+flush-179:012:15:011
989975999284,6cyclictest1092297-21kworker/u8:1+events_unbound12:30:021
989975999283,7cyclictest1082748-21kworker/u8:3+events_unbound12:10:011
989975998681,4cyclictest1013834-21kworker/u8:3+flush-179:008:45:321
989971998678,6cyclictest1007424-21latency_hist08:05:020
989975998577,6cyclictest1032555-21kworker/u8:3+events_unbound09:35:021
989975998479,4cyclictest1056619-21kworker/u8:1+flush-179:011:04:551
989975998477,5cyclictest1045286-21kworker/u8:2+events_unbound10:05:011
989975998477,4cyclictest1074791-21kworker/u8:2+events_unbound11:50:011
989975998378,4cyclictest1018861-21kworker/u8:0+events_unbound08:40:301
989975998373,8cyclictest1058260-21latency_hist10:40:011
989975998174,5cyclictest997830-21kworker/u8:3+flush-179:007:35:161
989975998174,5cyclictest993901-21kworker/u8:1+events_unbound07:25:311
989971998174,4cyclictest989912-21kworker/u8:2+flush-179:007:25:010
989975998071,7cyclictest1012441-21/usr/sbin/munin08:20:291
98997199790,59cyclictest1-21systemd09:13:350
989975997666,7cyclictest1074791-21kworker/u8:2+events_unbound11:40:241
989975997570,4cyclictest1071530-21kworker/u8:0+events_unbound11:21:501
989971997570,3cyclictest1072887-21kworker/u8:1+events_unbound11:40:000
989975997468,4cyclictest1074791-21kworker/u8:2+events_unbound12:00:001
989975997465,6cyclictest1032555-21kworker/u8:3+flush-179:009:25:241
989975997465,6cyclictest1022033-21kworker/u8:2+events_unbound09:05:281
989975997464,6cyclictest1045286-21kworker/u8:2+flush-179:010:15:161
128899740,10rtkit-daemon1287-21rtkit-daemon07:06:241
989975997368,4cyclictest1032555-21kworker/u8:3+flush-179:009:40:011
989975997366,5cyclictest999512-21kworker/u8:1+events_unbound08:15:221
989975997366,4cyclictest1056619-21kworker/u8:1+flush-179:010:45:341
989975997364,7cyclictest993901-21kworker/u8:1+flush-179:007:30:211
989971997362,8cyclictest1092502-21/usr/sbin/munin12:25:320
989975997064,4cyclictest1053232-21kworker/u8:0+events_unbound10:30:351
989975996963,5cyclictest1067874-21kworker/u8:2+flush-179:011:15:001
989975996962,5cyclictest1046870-21kworker/u8:3+flush-179:010:30:011
989975996958,5cyclictest1013834-21kworker/u8:3+events_unbound08:54:591
989975996958,5cyclictest1013834-21kworker/u8:3+events_unbound08:54:591
989971996862,4cyclictest1046870-21kworker/u8:3+flush-179:010:15:310
989975996757,7cyclictest1035734-21kworker/u8:2+events_unbound09:45:281
989971996759,5cyclictest1056631-21latency_hist10:35:010
989975996656,5cyclictest1045286-21kworker/u8:2+events_unbound10:05:291
989975996560,4cyclictest999512-21kworker/u8:1+flush-179:007:44:551
989975996557,5cyclictest1035734-21kworker/u8:2+events_unbound09:40:251
989975996459,3cyclictest1074791-21kworker/u8:2+flush-179:011:30:211
989975996458,4cyclictest999512-21kworker/u8:1+flush-179:007:45:311
989975996457,5cyclictest1056619-21kworker/u8:1+events_unbound11:05:241
989971996459,4cyclictest1074791-21kworker/u8:2+flush-179:011:50:310
989975996357,4cyclictest1082748-21kworker/u8:3+events_unbound12:25:181
989971996353,8cyclictest1034360-21apt-get09:25:110
989984996058,1cyclictest0-21swapper/312:27:583
989975996053,5cyclictest1042690-21kworker/u8:0+events_unbound10:15:001
989971995953,4cyclictest1025720-21kworker/u8:1+flush-179:009:20:320
989971995952,4cyclictest983867-21kworker/u8:3+events_unbound07:10:020
989975995853,4cyclictest1009411-21kworker/u8:2+flush-179:008:10:301
989975995846,9cyclictest2791555-21systemd-journal10:25:001
989971995849,7cyclictest1004231-21latency_hist07:55:000
989971995848,7cyclictest1059859-21latency_hist10:45:010
989975995650,5cyclictest1074791-21kworker/u8:2+events_unbound11:50:271
989975995645,4cyclictest1082748-21kworker/u8:3+events_unbound12:15:131
989971995621,6cyclictest1096005-21apt-get12:35:120
989975995547,6cyclictest1076580-21/usr/sbin/munin11:35:181
989971995547,6cyclictest1045286-21kworker/u8:2+flush-179:010:10:330
989975995448,4cyclictest996239-21kworker/u8:0+events_unbound08:05:291
989971995449,3cyclictest1043671-21kworker/u8:1+flush-179:009:55:320
989971995446,6cyclictest1013834-21kworker/u8:3+flush-179:008:50:000
989971995442,9cyclictest993901-21kworker/u8:1+flush-179:007:30:000
989975995340,11cyclictest991734-21apt-get07:15:151
98997199530,51cyclictest0-21swapper/010:21:590
989975995244,6cyclictest1061479-21kworker/u8:3+events_unbound11:15:231
989971995246,4cyclictest1009411-21kworker/u8:2+events_unbound08:30:020
989971995227,6cyclictest1007713-21grep08:05:140
989975995147,3cyclictest1056619-21kworker/u8:1+events_unbound10:40:461
989975995145,4cyclictest983867-21kworker/u8:3+flush-179:007:10:311
989971995147,3cyclictest2791555-21systemd-journal10:01:450
989971995143,7cyclictest998101-21apt-get07:35:130
4521791510,14ptp4l53390irq/134-eno1-TxRx-007:10:013
989971995043,5cyclictest1056619-21kworker/u8:1+events_unbound11:05:300
989971995041,7cyclictest1018861-21kworker/u8:0+events_unbound08:40:010
989971995040,8cyclictest989912-21kworker/u8:2+flush-179:007:20:000
989975994944,4cyclictest1082748-21kworker/u8:3+flush-179:012:20:331
989975994944,4cyclictest1074791-21kworker/u8:2+flush-179:012:00:301
989975994942,4cyclictest999512-21kworker/u8:1+events_unbound07:50:131
989971994943,4cyclictest1082748-21kworker/u8:3+flush-179:012:25:010
989975994836,10cyclictest1031166-21apt-get09:15:121
989975994836,10cyclictest1031166-21apt-get09:15:121
989971994844,3cyclictest1009411-21kworker/u8:2+flush-179:008:20:310
989971994837,8cyclictest999512-21kworker/u8:1+flush-179:008:20:010
989971994826,21cyclictest0-21swapper/011:00:010
989975994743,3cyclictest1061479-21kworker/u8:3+flush-179:011:25:211
989975994743,3cyclictest1018861-21kworker/u8:0+events_unbound08:59:461
989975994743,3cyclictest1013754-21kworker/u8:0+events_unbound08:30:371
989971994739,6cyclictest1025195-21latency_hist09:00:000
989971994727,18cyclictest0-21swapper/008:25:130
98997199470,28cyclictest0-21swapper/011:20:020
989975994642,3cyclictest999512-21kworker/u8:1+events_unbound08:04:451
989975994642,3cyclictest1095675-21kworker/u8:2+events_unbound12:35:301
989971994642,3cyclictest1059893-21kworker/u8:2+flush-179:010:50:460
989971994637,6cyclictest1038869-21latency_hist09:40:020
989971994624,4cyclictest1070391-21needreboot11:15:250
989975994540,4cyclictest996239-21kworker/u8:0+events_unbound07:55:241
989971994539,4cyclictest1013834-21kworker/u8:3+events_unbound08:45:010
989971994537,6cyclictest2791555-21systemd-journal08:52:460
989971994537,6cyclictest2791555-21systemd-journal08:52:450
989971994535,7cyclictest1068095-21apt-get11:10:120
989971994524,4cyclictest990183-21cpu07:10:150
989971994524,3cyclictest1053872-21gltestperf10:25:170
98998099440,42cyclictest0-21swapper/211:16:582
989975994433,8cyclictest1263-21rs:main1
989971994425,2cyclictest0-21swapper/010:39:060
98997199441,2cyclictest658-21Xorg08:13:590
989971994340,2cyclictest0-21swapper/007:34:580
989971994324,3cyclictest658-21Xorg09:34:590
98997199430,42cyclictest0-21swapper/007:47:580
989984994240,1cyclictest0-21swapper/310:36:593
989975994033,5cyclictest989912-21kworker/u8:2+events_unbound07:20:301
989971994035,4cyclictest1035734-21kworker/u8:2+flush-179:009:40:220
989971994034,4cyclictest1095781-21systemd-run12:35:010
989971994031,5cyclictest0-21swapper/012:13:590
989984993937,1cyclictest0-21swapper/312:01:003
989984993937,1cyclictest0-21swapper/310:08:583
98998499390,1cyclictest0-21swapper/309:01:593
989984993836,1cyclictest0-21swapper/310:45:593
98998099380,36cyclictest0-21swapper/212:23:592
98998099380,36cyclictest0-21swapper/211:23:592
989975993833,4cyclictest1025720-21kworker/u8:1+flush-179:009:20:231
4521791380,3ptp4l441ktimers/310:30:033
989984993735,1cyclictest0-21swapper/311:09:593
98998499370,1cyclictest0-21swapper/310:25:593
98998099370,35cyclictest0-21swapper/207:32:592
989971993733,2cyclictest658-21Xorg10:07:590
989971993732,3cyclictest658-21Xorg09:53:580
989971993729,6cyclictest1075011-21apt-get11:30:090
989984993634,1cyclictest0-21swapper/311:44:593
989984993633,2cyclictest0-21swapper/311:29:593
98998499360,35cyclictest0-21swapper/312:33:593
98998499360,35cyclictest0-21swapper/311:30:593
98998499360,1cyclictest0-21swapper/309:43:593
98998499360,1cyclictest0-21swapper/309:22:003
989980993633,2cyclictest0-21swapper/210:57:592
989980993633,2cyclictest0-21swapper/207:55:592
98998099360,1cyclictest0-21swapper/210:13:592
989975993632,3cyclictest1059893-21kworker/u8:2+flush-179:010:53:061
98997199361,7cyclictest1081119-21latency_hist11:50:000
98998499350,34cyclictest0-21swapper/310:24:593
98998499350,1cyclictest0-21swapper/311:55:593
989980993532,2cyclictest0-21swapper/211:26:592
989980993532,2cyclictest0-21swapper/211:03:592
989980993532,2cyclictest0-21swapper/210:40:592
989980993532,2cyclictest0-21swapper/209:34:592
989980993532,2cyclictest0-21swapper/209:26:592
98998099350,1cyclictest0-21swapper/212:04:592
98998099350,1cyclictest0-21swapper/211:49:592
989971993526,8cyclictest1025446-21apt-get09:00:120
989984993432,1cyclictest0-21swapper/309:09:003
989984993431,2cyclictest0-21swapper/310:04:593
989980993431,2cyclictest0-21swapper/211:40:592
989980993431,2cyclictest0-21swapper/208:57:592
989971993430,3cyclictest996239-21kworker/u8:0+flush-179:007:40:370
989971993430,3cyclictest1061479-21kworker/u8:3+events_unbound11:25:150
989984993331,1cyclictest0-21swapper/311:12:593
989984993331,1cyclictest0-21swapper/308:36:003
989984993331,1cyclictest0-21swapper/308:03:593
989980993331,1cyclictest0-21swapper/210:09:592
989980993331,1cyclictest0-21swapper/209:54:582
989980993330,2cyclictest0-21swapper/208:47:592
989980993330,2cyclictest0-21swapper/207:28:002
989971993326,5cyclictest658-21Xorg09:46:590
98997199332,23cyclictest1028587-21sed09:10:020
989984993230,1cyclictest0-21swapper/312:18:003
989984993230,1cyclictest0-21swapper/311:36:593
989980993229,2cyclictest0-21swapper/211:57:592
989980993229,2cyclictest0-21swapper/209:10:592
989971993228,3cyclictest996239-21kworker/u8:0+flush-179:007:55:260
989971993228,3cyclictest1032646-21latency_hist09:20:010
989971993228,3cyclictest1032646-21latency_hist09:20:010
989971993225,5cyclictest1064876-21/usr/sbin/munin11:00:210
989984993129,1cyclictest0-21swapper/311:00:593
989984993129,1cyclictest0-21swapper/308:24:593
989984993129,1cyclictest0-21swapper/307:45:143
989980993129,1cyclictest0-21swapper/212:39:592
989980993129,1cyclictest0-21swapper/208:34:592
989980993127,2cyclictest0-21swapper/212:15:002
989980993127,2cyclictest0-21swapper/211:05:132
98998099310,29cyclictest0-21swapper/209:35:172
989971993126,4cyclictest1084407-21cut12:00:010
989971993126,3cyclictest1074740-21kworker/u8:0+events_unbound11:40:150
98997199310,30cyclictest0-21swapper/010:45:160
98997199310,29cyclictest1086891-21ntp_192.168.11412:05:260
4521791310,7ptp4l441ktimers/310:15:003
989984993027,2cyclictest0-21swapper/310:40:123
989984993027,2cyclictest0-21swapper/309:30:133
989984993027,2cyclictest0-21swapper/307:30:013
98998499300,28cyclictest0-21swapper/310:55:013
98998499300,28cyclictest0-21swapper/309:00:003
989980993027,2cyclictest0-21swapper/212:05:142
989980993027,2cyclictest0-21swapper/211:13:592
989980993027,2cyclictest0-21swapper/210:54:592
989980993027,2cyclictest0-21swapper/210:30:032
989980993027,2cyclictest0-21swapper/210:15:132
989980993027,2cyclictest0-21swapper/209:20:122
989980993027,2cyclictest0-21swapper/209:00:132
989980993027,2cyclictest0-21swapper/208:40:132
989980993027,2cyclictest0-21swapper/208:20:142
989980993027,2cyclictest0-21swapper/208:01:592
98998099300,2cyclictest0-21swapper/208:55:012
98998099300,2cyclictest0-21swapper/208:55:002
98998099300,2cyclictest0-21swapper/207:50:102
98998099300,29cyclictest0-21swapper/209:15:132
98998099300,29cyclictest0-21swapper/209:15:132
98998099300,28cyclictest0-21swapper/211:40:002
98998099300,28cyclictest0-21swapper/211:30:102
98997199300,29cyclictest0-21swapper/012:15:110
989984992927,1cyclictest0-21swapper/311:15:133
989984992926,2cyclictest0-21swapper/309:56:003
989984992926,2cyclictest0-21swapper/307:40:023
98998499290,28cyclictest0-21swapper/312:05:123
98998499290,28cyclictest0-21swapper/311:45:113
98998499290,27cyclictest0-21swapper/309:10:183
98998499290,27cyclictest0-21swapper/308:30:023
98998499290,27cyclictest0-21swapper/308:25:143
98998499290,27cyclictest0-21swapper/308:10:143
98998499290,1cyclictest0-21swapper/309:45:143
98998499290,1cyclictest0-21swapper/309:15:123
98998499290,1cyclictest0-21swapper/309:15:123
98998499290,1cyclictest0-21swapper/308:40:113
989980992926,2cyclictest0-21swapper/210:22:592
989980992926,2cyclictest0-21swapper/208:25:122
989980992926,2cyclictest0-21swapper/207:45:012
98998099290,28cyclictest0-21swapper/210:00:132
98998099290,27cyclictest0-21swapper/212:35:002
98998099290,27cyclictest0-21swapper/212:25:152
98998099290,27cyclictest0-21swapper/211:50:112
98998099290,27cyclictest0-21swapper/210:00:012
98998099290,27cyclictest0-21swapper/209:45:152
98998099290,27cyclictest0-21swapper/209:40:112
98998099290,27cyclictest0-21swapper/207:35:272
4521791290,7ptp4l53390irq/134-eno1-TxRx-008:20:013
4521791290,6ptp4l441ktimers/312:25:013
989984992826,1cyclictest0-21swapper/311:20:033
989984992826,1cyclictest0-21swapper/309:35:593
989984992825,2cyclictest0-21swapper/307:20:013
989984992825,2cyclictest0-21swapper/307:15:013
98998499280,2cyclictest0-21swapper/307:40:023
989980992825,2cyclictest0-21swapper/210:25:012
989980992825,2cyclictest0-21swapper/208:05:132
989980992825,2cyclictest0-21swapper/207:45:142
989980992824,2cyclictest0-21swapper/208:40:002
98998099280,2cyclictest0-21swapper/207:20:002
98998099280,26cyclictest0-21swapper/208:15:132
98998099280,26cyclictest0-21swapper/207:10:152
98998099280,21cyclictest0-21swapper/208:10:142
4521791280,7ptp4l441ktimers/309:55:003
4521791280,6ptp4l53390irq/134-eno1-TxRx-009:30:003
4521791280,6ptp4l441ktimers/312:40:013
4521791280,6ptp4l441ktimers/308:09:593
4521791280,6ptp4l441ktimers/307:20:013
989984992725,1cyclictest0-21swapper/310:15:153
989984992725,1cyclictest0-21swapper/308:46:593
98998499270,26cyclictest0-21swapper/308:54:003
98998499270,26cyclictest0-21swapper/308:54:003
989980992725,1cyclictest0-21swapper/207:20:012
4521791270,6ptp4l53390irq/134-eno1-TxRx-007:55:003
4521791270,6ptp4l441ktimers/308:00:013
4521791270,6ptp4l441ktimers/307:35:003
4521791270,5ptp4l53390irq/134-eno1-TxRx-012:15:013
4521791270,5ptp4l53390irq/134-eno1-TxRx-011:55:003
98998499260,25cyclictest0-21swapper/310:55:153
98998099260,2cyclictest0-21swapper/212:15:122
98998099260,2cyclictest0-21swapper/210:45:172
98998099260,2cyclictest0-21swapper/210:35:142
98998099260,2cyclictest0-21swapper/209:05:122
9899809940,3cyclictest0-21swapper/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional