You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-29 - 13:02
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Fri Jun 28, 2024 12:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4186070999488,5cyclictest12823-21kworker/u8:2+flush-179:010:24:520
4186073998877,8cyclictest4832-21latency_hist07:50:011
4186073998676,7cyclictest274-21systemd-journal09:15:011
4186070998270,9cyclictest274-21systemd-journal10:09:490
4186073998164,14cyclictest11439-21/usr/sbin/munin08:10:251
4186073998074,4cyclictest274-21systemd-journal09:50:501
4186073998073,5cyclictest59033-21kworker/u8:3+flush-179:010:40:331
4186070998072,6cyclictest12823-21kworker/u8:2+flush-179:008:45:290
4186073997872,5cyclictest12823-21kworker/u8:2+flush-179:008:42:531
4186073997872,5cyclictest12823-21kworker/u8:2+flush-179:008:42:521
4186073997555,18cyclictest4192328-21kworker/u8:0+flush-179:007:45:011
4186070997561,11cyclictest91669-21apt-get12:15:130
4186070997362,8cyclictest274-21systemd-journal07:55:000
4186073997265,5cyclictest84566-21gpgv11:55:001
4186070997261,9cyclictest35016-21apt-get09:20:130
4186073997161,8cyclictest54436-21cat10:20:021
4186070997160,9cyclictest96455-21apt-get12:30:140
4186073997061,7cyclictest4187794-21/usr/sbin/munin07:15:311
4186070997061,7cyclictest30214-21/usr/sbin/munin09:05:190
4186073996961,6cyclictest12823-21kworker/u8:2+events_unbound08:45:251
4186070996957,10cyclictest46186-21apt-get09:55:150
418607899660,63cyclictest0-21swapper/307:55:593
4186073996657,6cyclictest89828-21latency_hist12:10:011
4186070996657,6cyclictest12823-21kworker/u8:2+flush-179:008:55:150
4186070996655,8cyclictest63832-21latency_hist10:50:010
4186070996462,2cyclictest0-21swapper/008:50:580
4186070996455,7cyclictest87981-21latency_hist12:05:000
4186070996455,6cyclictest53994-21apt-get10:20:000
4186078996159,1cyclictest0-21swapper/309:06:583
4186073996158,2cyclictest658-21Xorg11:34:591
4186073996152,7cyclictest79804-21kworker/u8:0+flush-179:012:20:271
4186073996051,7cyclictest86430-21kworker/u8:2+flush-179:012:15:001
4186073996051,6cyclictest12764-21kworker/u8:1+flush-179:008:40:011
4186073995949,7cyclictest94848-21/usr/sbin/munin12:25:301
4186070995848,8cyclictest85041-21apt-get11:55:110
4186070995848,7cyclictest65417-21latency_hist10:55:000
4186073995744,10cyclictest274-21systemd-journal11:05:361
4186073995650,4cyclictest49364-21kworker/u8:1+events_unbound10:20:011
4186073995646,8cyclictest4186226-21apt-get07:10:101
4186073995645,9cyclictest13045-21apt-get08:15:101
4186073995446,6cyclictest31564-21latency_hist09:10:011
4186073995444,8cyclictest76571-21latency_hist11:30:011
4186073995432,19cyclictest56129-21apt-get10:25:121
4186073995349,3cyclictest72667-21kworker/u8:0+flush-179:011:15:321
4186073995347,4cyclictest4189128-21kworker/u8:1+flush-179:007:40:001
4186073995344,7cyclictest68614-21latency_hist11:05:011
4186070995345,6cyclictest59033-21kworker/u8:3+flush-179:011:20:020
4186073995246,4cyclictest94588-21kworker/u8:2+events_unbound12:35:301
4186073995236,14cyclictest0-21swapper/107:25:011
4186070995242,8cyclictest70184-21kworker/u8:1+flush-179:011:10:000
4186073995139,9cyclictest12823-21kworker/u8:2+flush-179:009:00:011
418607099501,47cyclictest1-21systemd07:38:580
4521791490,14ptp4l53390irq/134-eno1-TxRx-007:09:563
4186070994942,5cyclictest274-21systemd-journal11:28:410
4186073994845,2cyclictest0-21swapper/108:07:591
4186073994839,6cyclictest12764-21kworker/u8:1+flush-179:008:55:001
4186073994838,8cyclictest19157-21latency_hist08:35:001
4186070994844,3cyclictest0-21swapper/011:50:010
4186070994840,6cyclictest12784-21latency_hist08:15:020
4186074994744,2cyclictest0-21swapper/208:36:592
4186073994740,4cyclictest37967-21kworker/u8:3+events_unbound09:45:131
4186073994737,7cyclictest70439-21/usr/sbin/munin11:10:301
4186073994626,2cyclictest34884-21latency_hist09:20:001
418607399461,28cyclictest0-21swapper/110:30:131
4186070994641,4cyclictest593-21systemd-logind08:44:170
4186070994641,4cyclictest593-21systemd-logind08:44:170
4186073994539,4cyclictest52617-21gdbus10:09:481
4186073994523,8cyclictest97191-21ntp_192.168.11412:30:271
4186074994440,2cyclictest0-21swapper/209:10:582
4186073994436,6cyclictest274-21systemd-journal10:54:411
4186073994436,6cyclictest16222-21/usr/sbin/munin08:25:291
4186070994432,10cyclictest90082-21apt-get12:10:110
4186073994340,2cyclictest658-21Xorg07:55:591
418607399430,29cyclictest658-21Xorg08:23:591
4186070994339,2cyclictest658-21Xorg08:00:590
4186070994338,4cyclictest4190739-21kworker/u8:2+flush-179:007:25:320
4186070994336,5cyclictest12823-21kworker/u8:2+flush-179:010:40:000
4186073994239,2cyclictest0-21swapper/112:04:591
418607399420,2cyclictest0-21swapper/109:05:011
4186070994240,2cyclictest0-21swapper/009:50:590
4186070994237,4cyclictest658-21Xorg09:01:000
418607899410,40cyclictest0-21swapper/311:53:593
418607899410,1cyclictest0-21swapper/308:19:593
4186073994128,10cyclictest82977-21kworker/u8:1+events_unbound11:55:021
418607399411,29cyclictest41383-21apt-get09:40:141
418607099416,32cyclictest89975-21cut12:10:010
418607399400,39cyclictest9730-21cat08:05:011
4186070994037,2cyclictest658-21Xorg12:30:000
4186070994036,3cyclictest0-21swapper/007:42:580
418607099401,37cyclictest658-21Xorg10:40:590
418607099401,37cyclictest658-21Xorg09:34:590
418607099401,37cyclictest658-21Xorg09:34:590
4186074993937,1cyclictest0-21swapper/207:35:002
418607499390,36cyclictest0-21swapper/209:08:592
418607399394,13cyclictest1240-21lightdm-gtk-gre11:40:001
4186073993927,10cyclictest73653-21apt-get11:20:101
418607399391,36cyclictest81610-21apt-get11:45:121
418607399390,2cyclictest658-21Xorg10:48:581
4186070993935,3cyclictest37967-21kworker/u8:3+flush-179:009:40:290
418607099391,36cyclictest31822-21apt-get09:10:140
4186074993835,2cyclictest0-21swapper/210:26:592
4186074993835,2cyclictest0-21swapper/208:56:582
418607499380,35cyclictest0-21swapper/210:55:002
4186073993834,3cyclictest658-21Xorg10:39:001
4186073993826,10cyclictest5088-21apt-get07:50:111
418607399380,37cyclictest0-21swapper/111:00:001
4186070993835,2cyclictest4869-21systemd-run07:50:010
418607099383,33cyclictest42974-21http09:45:110
4186070993832,4cyclictest74450-21kworker/u8:2+flush-179:011:45:000
418607099380,37cyclictest0-21swapper/008:29:590
4186078993734,2cyclictest0-21swapper/311:41:593
4186078993734,2cyclictest0-21swapper/309:36:583
418607899370,36cyclictest0-21swapper/309:03:003
4186074993734,2cyclictest0-21swapper/212:01:592
418607499372,1cyclictest0-21swapper/208:51:592
418607499370,35cyclictest0-21swapper/210:12:002
418607499370,1cyclictest0-21swapper/211:14:002
418607399370,1cyclictest0-21swapper/110:10:591
418607399370,10cyclictest0-21swapper/109:35:141
4186070993734,2cyclictest0-21swapper/011:15:590
4186070993729,7cyclictest83201-21cpu11:50:130
418607099370,35cyclictest4194081-21cat07:35:010
418607099370,1cyclictest0-21swapper/011:11:590
4186078993633,2cyclictest0-21swapper/307:19:593
4186078993630,5cyclictest67042-21turbostat11:00:003
418607899360,35cyclictest0-21swapper/311:32:583
418607899360,35cyclictest0-21swapper/309:47:003
418607899360,35cyclictest0-21swapper/309:40:593
418607899360,1cyclictest0-21swapper/312:02:593
418607899360,1cyclictest0-21swapper/309:13:593
4186074993633,2cyclictest0-21swapper/211:30:592
4186074993633,2cyclictest0-21swapper/209:54:592
4186074993633,2cyclictest0-21swapper/207:51:002
4186073993630,4cyclictest0-21swapper/109:30:251
4186073993630,4cyclictest0-21swapper/109:30:241
418607099364,31cyclictest56169-21grep10:25:140
418607099361,33cyclictest67555-21df11:00:120
418607099361,33cyclictest57749-21grep10:30:120
418607099361,33cyclictest13782-21ntp_192.168.11408:15:230
4186078993533,1cyclictest0-21swapper/310:36:593
4186078993533,1cyclictest0-21swapper/308:21:593
4186078993532,2cyclictest0-21swapper/312:33:593
4186074993532,2cyclictest0-21swapper/211:52:592
4186073993528,5cyclictest12823-21kworker/u8:2+flush-179:010:00:321
4186070993529,4cyclictest17831-21http08:30:150
4186070993527,6cyclictest41229-21sed09:40:000
4186070993525,8cyclictest33411-21http09:15:130
418607099350,3cyclictest658-21Xorg07:23:580
418607099350,33cyclictest77103-21sh11:30:140
4186078993432,2cyclictest0-21swapper/312:13:593
4186078993432,1cyclictest0-21swapper/310:30:593
4186078993431,2cyclictest0-21swapper/311:08:003
4186074993431,2cyclictest0-21swapper/211:00:592
418607399340,32cyclictest81461-21latency_hist11:45:001
418607099341,2cyclictest0-21swapper/010:55:150
418607099340,32cyclictest6415-21latency_hist07:55:010
4186078993331,1cyclictest0-21swapper/308:35:593
4186078993330,2cyclictest0-21swapper/307:22:583
4186078993330,2cyclictest0-21swapper/307:13:593
4186074993331,1cyclictest0-21swapper/207:48:592
4186074993330,2cyclictest0-21swapper/210:18:002
4186073993324,8cyclictest46478-21diskstats09:55:201
4186073993322,6cyclictest0-21swapper/107:25:591
4186070993327,5cyclictest0-21swapper/008:35:130
4186070993322,9cyclictest98837-21ntp_states12:35:270
418607099331,3cyclictest50985-21/usr/sbin/munin10:10:120
418607099331,30cyclictest47525-21latency_hist10:00:000
418607099330,31cyclictest93282-21http12:20:130
418607099330,31cyclictest10043-21expr08:05:130
4186078993230,1cyclictest0-21swapper/312:21:593
4186073993229,2cyclictest0-21swapper/109:30:001
418607399321,3cyclictest4192880-21sh07:30:151
4186070993227,3cyclictest4186230-21cpu07:10:120
4521791310,5ptp4l53390irq/134-eno1-TxRx-009:59:553
4186078993129,1cyclictest0-21swapper/310:47:593
4186078993129,1cyclictest0-21swapper/308:55:133
418607899310,30cyclictest0-21swapper/308:50:013
4186074993129,1cyclictest0-21swapper/209:30:002
4186074993129,1cyclictest0-21swapper/209:19:592
4186074993129,1cyclictest0-21swapper/209:00:112
4186074993128,2cyclictest0-21swapper/211:35:102
4186074993128,2cyclictest0-21swapper/210:55:182
418607499310,29cyclictest0-21swapper/210:20:032
418607499310,29cyclictest0-21swapper/210:05:132
418607499310,29cyclictest0-21swapper/208:00:012
4186073993127,2cyclictest0-21swapper/112:15:151
418607099310,29cyclictest37400-21ntp_states09:25:250
4186078993027,2cyclictest0-21swapper/312:05:013
4186078993027,2cyclictest0-21swapper/311:45:133
4186078993027,2cyclictest0-21swapper/311:10:143
4186078993027,2cyclictest0-21swapper/310:05:003
4186078993027,2cyclictest0-21swapper/307:30:123
4186078993027,2cyclictest0-21swapper/307:28:593
418607899300,28cyclictest0-21swapper/308:50:023
418607899300,28cyclictest0-21swapper/308:00:123
418607899300,28cyclictest0-21swapper/307:50:233
4186074993027,2cyclictest0-21swapper/212:25:152
4186074993027,2cyclictest0-21swapper/212:05:112
4186074993027,2cyclictest0-21swapper/211:27:592
4186074993027,2cyclictest0-21swapper/211:16:592
4186074993027,2cyclictest0-21swapper/210:40:142
4186074993027,2cyclictest0-21swapper/209:23:592
4186074993027,2cyclictest0-21swapper/208:18:002
418607499300,2cyclictest0-21swapper/208:25:112
418607499300,2cyclictest0-21swapper/207:15:142
418607499300,28cyclictest0-21swapper/212:20:122
418607499300,28cyclictest0-21swapper/211:55:132
418607499300,28cyclictest0-21swapper/210:35:122
418607499300,28cyclictest0-21swapper/209:45:002
418607499300,28cyclictest0-21swapper/209:35:002
418607499300,28cyclictest0-21swapper/209:35:002
418607499300,28cyclictest0-21swapper/208:10:132
418607499300,28cyclictest0-21swapper/207:10:112
4186070993025,4cyclictest79249-21phc2sys11:35:260
418607099300,29cyclictest0-21swapper/007:15:150
4521791290,5ptp4l441ktimers/308:44:563
4521791290,5ptp4l441ktimers/308:44:563
4186078992927,1cyclictest0-21swapper/312:40:013
4186078992927,1cyclictest0-21swapper/310:40:133
4186078992927,1cyclictest0-21swapper/309:50:143
4186078992927,1cyclictest0-21swapper/308:10:113
4186078992927,1cyclictest0-21swapper/307:45:103
4186078992926,2cyclictest0-21swapper/310:20:023
4186078992926,2cyclictest0-21swapper/310:05:173
4186078992926,2cyclictest0-21swapper/309:15:153
4186078992926,2cyclictest0-21swapper/308:06:593
418607899290,28cyclictest0-21swapper/309:25:113
418607899290,28cyclictest0-21swapper/307:40:113
418607899290,27cyclictest0-21swapper/311:55:033
418607899290,27cyclictest0-21swapper/311:15:153
418607899290,27cyclictest0-21swapper/307:35:143
418607899290,1cyclictest0-21swapper/309:30:143
418607899290,1cyclictest0-21swapper/309:30:143
418607899290,1cyclictest0-21swapper/309:20:143
4186074992926,2cyclictest0-21swapper/212:35:132
4186074992926,2cyclictest0-21swapper/210:45:272
4186074992926,2cyclictest0-21swapper/210:35:012
4186074992926,2cyclictest0-21swapper/209:55:172
4186074992926,2cyclictest0-21swapper/208:40:102
4186074992926,2cyclictest0-21swapper/208:40:092
4186074992926,2cyclictest0-21swapper/208:00:132
4186074992926,2cyclictest0-21swapper/207:38:592
418607499290,28cyclictest0-21swapper/208:05:102
418607499290,27cyclictest0-21swapper/210:00:262
418607499290,27cyclictest0-21swapper/207:20:162
418607499290,1cyclictest0-21swapper/212:35:012
4186070992927,1cyclictest14654-21apt-get08:20:100
4186078992826,1cyclictest0-21swapper/310:15:123
4186078992826,1cyclictest0-21swapper/310:10:103
4186078992825,2cyclictest0-21swapper/312:25:143
4186078992825,2cyclictest0-21swapper/311:00:103
418607899280,27cyclictest0-21swapper/310:50:113
418607899280,27cyclictest0-21swapper/310:29:003
418607899280,27cyclictest0-21swapper/308:25:133
418607899280,26cyclictest0-21swapper/312:15:143
4186074992825,2cyclictest0-21swapper/212:10:232
4186074992825,2cyclictest0-21swapper/211:45:262
4186074992825,2cyclictest0-21swapper/208:45:172
4186074992825,2cyclictest0-21swapper/208:20:112
418607499280,2cyclictest0-21swapper/207:40:122
418607499280,27cyclictest0-21swapper/211:05:122
418607499280,27cyclictest0-21swapper/209:45:122
418607499280,27cyclictest0-21swapper/208:30:012
418607499280,26cyclictest0-21swapper/207:30:012
4186073992822,4cyclictest30520-21kworker/u8:0+flush-179:009:25:001
4186074992725,1cyclictest0-21swapper/209:35:012
4186070992722,4cyclictest4186096-21latency_hist07:10:010
4521791260,6ptp4l441ktimers/311:29:553
4521791260,6ptp4l441ktimers/308:35:013
418607899260,25cyclictest0-21swapper/311:25:013
418607499260,2cyclictest0-21swapper/212:15:142
418607499260,25cyclictest0-21swapper/211:40:142
418607499260,24cyclictest0-21swapper/211:20:112
4521791250,6ptp4l53390irq/134-eno1-TxRx-011:40:003
4186073992116,3cyclictest0-21swapper/107:10:011
41860749930,2cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional