You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 08:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Fri Nov 22, 2024 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
368708099138131,6cyclictest3777691-21kworker/u8:2+events_unbound00:04:031
368707999117106,8cyclictest3732534-21/usr/sbin/munin21:30:300
368707999116111,4cyclictest3738898-21kworker/u8:2+events_unbound22:00:240
368707999112102,8cyclictest3774055-21kworker/u8:1+flush-179:000:00:000
368708099110105,4cyclictest3777691-21kworker/u8:2+events_unbound00:05:251
36870809910897,4cyclictest3749794-21kworker/u8:2+events_unbound22:25:191
36870809910799,6cyclictest3755694-21kworker/u8:1+flush-179:022:50:271
36870809910398,4cyclictest3693543-21kworker/u8:0+events_unbound20:00:221
36870799910187,11cyclictest3776122-21apt-get23:50:090
3687080999994,4cyclictest3682419-21kworker/u8:0+events_unbound19:15:251
3687080999991,6cyclictest3772395-21kworker/u8:0+flush-179:023:50:211
3687080999888,7cyclictest3753040-21kworker/u8:3+flush-179:022:40:001
3687080999486,6cyclictest3749794-21kworker/u8:2+flush-179:023:05:221
3687080999287,4cyclictest3751642-21kworker/u8:1+events_unbound22:30:311
3687080999287,4cyclictest3737124-21kworker/u8:1+events_unbound21:55:241
3687079999280,9cyclictest3692216-21/usr/sbin/munin19:25:230
3687080999187,3cyclictest3711068-21kworker/u8:1+events_unbound20:30:221
3687079999186,4cyclictest3722470-21kworker/u8:0+events_unbound21:05:230
3687080998979,4cyclictest3722470-21kworker/u8:0+events_unbound21:25:241
3687080998979,4cyclictest3722470-21kworker/u8:0+events_unbound21:25:241
3687080998884,3cyclictest3693543-21kworker/u8:0+events_unbound19:45:241
3687079998774,10cyclictest3759928-21apt-get22:55:090
3687080998675,8cyclictest3105085-21systemd-journal21:15:011
3687080998473,4cyclictest3772395-21kworker/u8:0+events_unbound00:10:321
3687079998479,4cyclictest3772395-21kworker/u8:0+events_unbound23:39:550
3687080998374,7cyclictest3726994-21latency_hist21:15:011
3687080998374,3cyclictest3693601-21kworker/u8:3+events_unbound19:55:231
3687079998378,4cyclictest3749794-21kworker/u8:2+events_unbound22:49:550
3687079998378,4cyclictest3741909-21kworker/u8:0+events_unbound22:29:560
3687079998371,10cyclictest3791016-21/usr/sbin/munin00:35:240
3687080998278,3cyclictest3706337-21kworker/u8:3+events_unbound20:24:571
3687080998277,3cyclictest3693543-21kworker/u8:0+events_unbound20:05:131
3687079998270,9cyclictest3712930-21apt-get20:30:110
3687080998176,4cyclictest3766021-21kworker/u8:0+events_unbound23:24:561
3687079998176,4cyclictest3755694-21kworker/u8:1+events_unbound22:54:560
3687080998076,3cyclictest3764436-21kworker/u8:1+events_unbound23:19:561
3687080998076,3cyclictest3686871-21kworker/u8:1+events_unbound19:25:211
3687080998076,3cyclictest3686871-21kworker/u8:1+events_unbound19:10:211
3687080998075,4cyclictest3764436-21kworker/u8:1+events_unbound23:10:111
3687080998070,4cyclictest3682419-21kworker/u8:0+events_unbound19:20:261
3687080997974,4cyclictest3755694-21kworker/u8:1+events_unbound23:00:171
3687080997974,4cyclictest3755694-21kworker/u8:1+events_unbound23:00:161
368707999790,76cyclictest688-21snmpd21:15:000
3687080997873,4cyclictest3693543-21kworker/u8:0+events_unbound20:17:011
3687079997873,4cyclictest3728898-21kworker/u8:3+events_unbound21:49:570
3687080997768,3cyclictest3711068-21kworker/u8:1+events_unbound20:55:111
3687079997672,3cyclictest3789160-21kworker/u8:2+events_unbound00:34:560
3687080997570,4cyclictest3772395-21kworker/u8:0+events_unbound23:38:431
3687080997570,4cyclictest3728898-21kworker/u8:3+events_unbound21:34:551
3687079997571,3cyclictest3728898-21kworker/u8:3+events_unbound21:24:560
3687080997470,3cyclictest3720593-21kworker/u8:2+events_unbound21:07:431
3687080997469,4cyclictest3746877-21kworker/u8:1+events_unbound22:20:191
3687080997467,6cyclictest3715823-21kworker/u8:0+flush-179:020:40:221
3687079997470,3cyclictest3691956-21kworker/u8:2+events_unbound19:37:380
3687079997466,6cyclictest3105085-21systemd-journal00:04:010
3687079997369,3cyclictest3711068-21kworker/u8:1+events_unbound20:51:430
3687079997368,4cyclictest3737124-21kworker/u8:1+events_unbound21:54:570
3687079997362,9cyclictest3771082-21apt-get23:30:160
3687080997267,4cyclictest3782287-21kworker/u8:1+events_unbound00:31:421
3687080997262,8cyclictest3787852-21apt-get00:25:121
3687079997267,4cyclictest3749794-21kworker/u8:2+events_unbound22:23:440
3687079997267,4cyclictest3693543-21kworker/u8:0+events_unbound20:02:440
3687080997167,3cyclictest3767888-21kworker/u8:2+events_unbound23:29:561
3687080997167,3cyclictest3745068-21kworker/u8:2+events_unbound22:10:281
3687080997159,10cyclictest3697017-21apt-get19:40:131
3687079997166,4cyclictest3741909-21kworker/u8:0+events_unbound22:36:440
3687079997166,4cyclictest3728898-21kworker/u8:3+events_unbound21:38:010
3687080997065,4cyclictest3746877-21kworker/u8:1+events_unbound22:18:431
3687080997065,4cyclictest3732330-21kworker/u8:2+events_unbound21:39:571
3687079997065,4cyclictest3749794-21kworker/u8:2+events_unbound23:04:420
3687079997065,4cyclictest3749794-21kworker/u8:2+events_unbound23:04:420
3687079997065,4cyclictest3720593-21kworker/u8:2+events_unbound21:17:580
3687079997065,4cyclictest3693543-21kworker/u8:0+events_unbound19:32:390
3687080996959,8cyclictest3743684-21/usr/sbin/munin22:05:251
3687079996965,3cyclictest3693543-21kworker/u8:0+events_unbound20:28:400
3687080996864,3cyclictest3715823-21kworker/u8:0+events_unbound20:48:411
3687080996863,4cyclictest3789160-21kworker/u8:2+events_unbound00:36:421
3687079996864,3cyclictest3693543-21kworker/u8:0+events_unbound19:52:410
3687080996763,3cyclictest3728898-21kworker/u8:3+events_unbound22:04:421
3687080996763,3cyclictest3720593-21kworker/u8:2+events_unbound20:50:291
3687080996762,4cyclictest3691956-21kworker/u8:2+events_unbound19:36:401
3687079996763,3cyclictest3722470-21kworker/u8:0+events_unbound21:02:570
3687080996662,3cyclictest3722470-21kworker/u8:0+events_unbound21:00:251
3687080996661,4cyclictest3772395-21kworker/u8:0+events_unbound00:17:411
3687080996658,6cyclictest3728898-21kworker/u8:3+flush-179:021:20:251
3687079996662,3cyclictest3749794-21kworker/u8:2+events_unbound23:07:430
3687079996662,3cyclictest3706337-21kworker/u8:3+events_unbound20:23:400
3687079996661,4cyclictest3741909-21kworker/u8:0+events_unbound22:44:410
3687080996561,3cyclictest3769288-21kworker/u8:3+events_unbound23:46:421
3687080996561,3cyclictest3693543-21kworker/u8:0+events_unbound20:27:401
3687079996455,6cyclictest3732295-21latency_hist21:30:010
3687079996455,6cyclictest3732295-21latency_hist21:30:000
3687080996359,3cyclictest3755694-21kworker/u8:1+events_unbound22:58:411
3687080996354,7cyclictest3779050-21latency_hist00:00:011
3687081996259,2cyclictest0-21swapper/220:25:592
3687080996258,3cyclictest3693543-21kworker/u8:0+events_unbound19:52:391
3687080996257,4cyclictest3711068-21kworker/u8:1+events_unbound20:35:041
3687080996252,8cyclictest3706539-21/usr/sbin/munin20:10:251
3687080996252,8cyclictest3706539-21/usr/sbin/munin20:10:241
3687080996157,3cyclictest3728898-21kworker/u8:3+events_unbound21:51:411
3687079996156,4cyclictest3682419-21kworker/u8:0+events_unbound19:19:380
3687080996049,9cyclictest3787349-21apt-get00:25:001
3687079995652,3cyclictest3764436-21kworker/u8:1+events_unbound23:15:270
3687079995645,9cyclictest3715863-21latency_hist20:40:010
3687079995348,4cyclictest3693543-21kworker/u8:0+events_unbound20:11:040
3687079995348,4cyclictest3693543-21kworker/u8:0+events_unbound20:11:040
3687079995245,5cyclictest3711068-21kworker/u8:1+flush-179:020:55:140
3687091995149,1cyclictest0-21swapper/320:54:593
3687079995147,3cyclictest3772395-21kworker/u8:0+events_unbound00:20:020
3687079995145,4cyclictest3772395-21kworker/u8:0+flush-179:000:15:220
3687080995045,4cyclictest3769288-21kworker/u8:3+events_unbound23:41:421
3687080994941,6cyclictest3737079-21latency_hist21:45:021
4505291470,13ptp4l441ktimers/319:10:013
3687079994743,3cyclictest3711068-21kworker/u8:1+events_unbound20:47:540
3687079994743,3cyclictest3704692-21kworker/u8:1+events_unbound20:06:390
3687079994742,4cyclictest3767888-21kworker/u8:2+events_unbound23:26:430
3687080994642,3cyclictest3769288-21kworker/u8:3+events_unbound23:30:021
3687079994641,4cyclictest3738898-21kworker/u8:2+events_unbound21:57:420
3687079994635,9cyclictest3751948-21diskstats22:30:190
3687080994541,3cyclictest3755694-21kworker/u8:1+events_unbound22:43:421
3687079994540,4cyclictest3766021-21kworker/u8:0+events_unbound23:12:420
3687079994540,4cyclictest3682419-21kworker/u8:0+events_unbound19:20:080
368708199440,41cyclictest0-21swapper/223:19:552
3687080994440,3cyclictest3691956-21kworker/u8:2+events_unbound19:33:201
3687079994439,4cyclictest3789160-21kworker/u8:2+events_unbound00:29:130
3687079994436,6cyclictest3769288-21kworker/u8:3+flush-179:023:40:250
368709199420,39cyclictest3730453-21turbostat21:25:003
3687079994237,4cyclictest3745068-21kworker/u8:2+events_unbound22:14:240
3687079994136,4cyclictest3693601-21kworker/u8:3+events_unbound19:45:070
368709199400,39cyclictest0-21swapper/322:43:593
3687079994033,5cyclictest722-21Xorg19:45:000
368709199390,4cyclictest3792396-21turbostat00:40:013
368709199390,36cyclictest0-21swapper/323:44:593
368708199390,36cyclictest0-21swapper/222:34:592
3687079993930,7cyclictest3105085-21systemd-journal00:10:000
3687079993930,3cyclictest3728898-21kworker/u8:3+events_unbound21:43:540
3687081993835,2cyclictest0-21swapper/219:48:592
368708199380,36cyclictest0-21swapper/219:26:592
3687080993834,3cyclictest3722470-21kworker/u8:0+events_unbound21:47:411
3687079993833,4cyclictest3693543-21kworker/u8:0+events_unbound19:56:510
3687079993832,5cyclictest3741909-21kworker/u8:0+flush-179:022:05:170
3687079993827,9cyclictest3782751-21cat00:10:020
3687091993735,1cyclictest0-21swapper/322:00:593
3687091993734,2cyclictest0-21swapper/321:29:593
3687091993734,2cyclictest0-21swapper/321:29:583
3687091993731,4cyclictest3764461-21turbostat23:10:013
368709199370,35cyclictest0-21swapper/321:50:593
3687081993734,2cyclictest0-21swapper/219:11:592
368708199370,35cyclictest0-21swapper/223:47:592
368708199370,35cyclictest0-21swapper/222:09:002
3687091993634,1cyclictest0-21swapper/322:29:543
3687091993632,2cyclictest0-21swapper/300:03:593
3687081993632,2cyclictest0-21swapper/220:24:542
368708199360,1cyclictest0-21swapper/220:43:592
368708199360,1cyclictest0-21swapper/200:35:592
3687079993632,3cyclictest3715823-21kworker/u8:0+events_unbound20:40:340
3687079993631,4cyclictest3686871-21kworker/u8:1+flush-179:019:10:270
368707999361,33cyclictest0-21swapper/023:20:140
3687091993533,1cyclictest0-21swapper/319:34:543
3687091993532,2cyclictest0-21swapper/323:58:583
3687081993532,2cyclictest0-21swapper/223:26:592
3687081993532,2cyclictest0-21swapper/220:12:002
3687081993532,2cyclictest0-21swapper/220:11:592
3687081993532,2cyclictest0-21swapper/219:17:002
3687080993531,3cyclictest3755694-21kworker/u8:1+events_unbound22:48:181
3687079993530,4cyclictest3741909-21kworker/u8:0+events_unbound22:15:070
3687079993529,4cyclictest3709662-21latency_hist20:20:000
3687091993432,1cyclictest0-21swapper/322:39:003
3687079993430,3cyclictest3772395-21kworker/u8:0+events_unbound23:49:000
3687091993331,1cyclictest0-21swapper/321:39:543
3687091993331,1cyclictest0-21swapper/319:39:553
3687091993330,2cyclictest0-21swapper/319:45:003
3687091993330,2cyclictest0-21swapper/319:12:593
3687081993329,2cyclictest0-21swapper/221:04:592
3687080993313,7cyclictest29-21ksoftirqd/119:10:031
3687091993230,1cyclictest0-21swapper/319:22:593
3687091993229,2cyclictest0-21swapper/320:21:583
3687091993229,2cyclictest0-21swapper/300:30:593
368709199320,29cyclictest0-21swapper/322:10:143
3687081993229,2cyclictest0-21swapper/221:43:582
3687091993125,4cyclictest3774014-21turbostat23:40:003
368709199310,27cyclictest0-21swapper/321:55:113
3687081993128,2cyclictest0-21swapper/223:06:592
3687081993128,2cyclictest0-21swapper/221:56:592
3687081993128,2cyclictest0-21swapper/200:31:592
368708199310,29cyclictest0-21swapper/222:45:132
3687091993028,1cyclictest0-21swapper/300:19:563
368709199300,27cyclictest0-21swapper/322:20:143
368709199300,26cyclictest0-21swapper/323:20:143
368709199300,26cyclictest0-21swapper/320:55:113
368709199300,26cyclictest0-21swapper/320:31:593
368709199300,26cyclictest0-21swapper/320:15:143
3687081993027,2cyclictest0-21swapper/219:20:122
368709199290,28cyclictest0-21swapper/323:26:593
368709199290,27cyclictest0-21swapper/322:49:593
368709199290,27cyclictest0-21swapper/322:05:143
368709199290,26cyclictest0-21swapper/320:05:123
3687081992926,2cyclictest0-21swapper/221:07:002
3687081992926,2cyclictest0-21swapper/200:12:592
368708199290,28cyclictest0-21swapper/219:50:142
368708199290,27cyclictest0-21swapper/220:30:132
368708199290,27cyclictest0-21swapper/200:25:022
368708199290,1cyclictest0-21swapper/200:00:122
4505291280,2ptp4l61790irq/134-eno1-TxRx-021:45:003
3687091992826,1cyclictest0-21swapper/320:00:003
3687091992825,2cyclictest0-21swapper/319:15:133
368709199280,26cyclictest0-21swapper/322:55:023
368709199280,26cyclictest0-21swapper/321:30:183
368709199280,26cyclictest0-21swapper/300:10:043
368709199280,24cyclictest0-21swapper/322:50:163
368709199280,1cyclictest0-21swapper/321:15:143
3687081992825,2cyclictest0-21swapper/220:18:592
3687081992825,2cyclictest0-21swapper/219:34:592
3687081992825,2cyclictest0-21swapper/200:21:592
368708199280,27cyclictest0-21swapper/222:00:122
368708199280,26cyclictest0-21swapper/223:30:172
368708199280,26cyclictest0-21swapper/222:50:162
368708199280,26cyclictest0-21swapper/222:41:582
4505291270,6ptp4l441ktimers/323:55:013
4505291270,6ptp4l441ktimers/323:15:003
4505291270,6ptp4l441ktimers/320:45:003
3687091992725,1cyclictest0-21swapper/322:16:593
368709199270,26cyclictest0-21swapper/319:53:583
368709199270,26cyclictest0-21swapper/300:28:593
3687081992725,1cyclictest0-21swapper/223:41:592
3687081992724,2cyclictest0-21swapper/222:17:592
3687081992724,2cyclictest0-21swapper/220:39:002
368708199270,2cyclictest0-21swapper/222:29:552
368708199270,26cyclictest0-21swapper/219:45:002
368708199270,1cyclictest0-21swapper/220:58:592
4505291260,6ptp4l441ktimers/320:30:013
4505291260,3ptp4l441ktimers/323:34:563
3687091992624,1cyclictest0-21swapper/321:04:593
368709199260,2cyclictest0-21swapper/323:02:593
368709199260,2cyclictest0-21swapper/323:02:593
368709199260,25cyclictest0-21swapper/321:07:593
368709199260,24cyclictest0-21swapper/320:10:143
368709199260,24cyclictest0-21swapper/320:10:143
3687081992623,2cyclictest0-21swapper/221:34:002
368708199260,2cyclictest0-21swapper/223:37:592
368708199260,2cyclictest0-21swapper/222:21:592
368708199260,2cyclictest0-21swapper/221:17:582
368708199260,24cyclictest0-21swapper/219:59:592
368708199260,24cyclictest0-21swapper/200:19:562
4505291250,3ptp4l441ktimers/320:05:013
3687091992523,1cyclictest0-21swapper/321:14:543
3687091992523,1cyclictest0-21swapper/319:25:593
3687091992522,2cyclictest0-21swapper/321:45:123
3687091992522,2cyclictest0-21swapper/320:38:593
3687091992521,3cyclictest3767640-21turbostat23:20:003
3687081992522,2cyclictest0-21swapper/220:47:592
368708199250,2cyclictest0-21swapper/223:11:002
368708199250,2cyclictest0-21swapper/221:25:002
4505291240,3ptp4l441ktimers/319:50:013
3687091992422,1cyclictest0-21swapper/300:06:593
3687081992422,1cyclictest0-21swapper/220:03:592
3687081992421,2cyclictest0-21swapper/200:00:012
3687079992421,2cyclictest3687181-21apt-key19:10:030
4505291230,4ptp4l441ktimers/300:24:563
4505291230,3ptp4l61790irq/134-eno1-TxRx-023:50:003
4505291230,3ptp4l61790irq/134-eno1-TxRx-022:35:003
4505291230,3ptp4l61790irq/134-eno1-TxRx-020:49:553
3687081992320,2cyclictest0-21swapper/223:22:592
3687081992320,2cyclictest0-21swapper/222:56:592
3687081992320,2cyclictest0-21swapper/220:54:002
3687081992219,2cyclictest0-21swapper/221:25:592
3687081992219,2cyclictest0-21swapper/221:25:592
3687081992219,2cyclictest0-21swapper/221:10:592
3687081992119,1cyclictest0-21swapper/200:06:002
3687081992018,1cyclictest0-21swapper/223:54:592
3687081992018,1cyclictest0-21swapper/221:49:592
3687081991715,1cyclictest0-21swapper/222:36:592
3687081991714,2cyclictest0-21swapper/223:00:112
3687081991714,2cyclictest0-21swapper/223:00:112
3687081991614,1cyclictest0-21swapper/222:10:152
3687081991412,1cyclictest0-21swapper/219:37:582
368708199110,10cyclictest0-21swapper/221:55:002
36870819985,2cyclictest0-21swapper/220:07:352
36870819980,7cyclictest0-21swapper/221:37:522
36870819950,4cyclictest0-21swapper/219:10:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional