You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3.osadl.org (updated Thu Apr 03, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2609950996143,17cyclictest2665637-21kworker/1:022:10:091
2609950996140,20cyclictest2323756-1kworker/1:2H21:10:001
2609950996045,14cyclictest2674359-21kworker/1:122:50:091
2609951995943,4cyclictest2677316-21kworker/2:100:09:592
2609950995944,14cyclictest2696544-21kworker/1:100:20:001
2609950995940,18cyclictest2627161-21kworker/1:020:05:161
2609950995940,17cyclictest2608474-21kworker/1:119:10:151
2609950995843,14cyclictest2678911-21kworker/1:223:18:401
2609950995843,14cyclictest2651174-21kworker/1:021:20:011
2609950995843,14cyclictest2629769-21kworker/1:120:10:271
2609950995843,14cyclictest2615196-21kworker/1:119:40:001
2609950995843,14cyclictest2615196-21kworker/1:119:25:151
2609950995841,16cyclictest2684511-21kworker/1:023:30:131
2609950995841,16cyclictest2633120-21kworker/1:020:35:151
2609950995841,16cyclictest2615196-21kworker/1:119:40:151
2609950995840,17cyclictest2641667-21kworker/1:121:10:181
2609950995840,17cyclictest2615196-21kworker/1:119:30:191
2609950995837,19cyclictest2633120-21kworker/1:020:30:181
2609950995830,27cyclictest2611401-21kworker/1:019:15:001
2609950995742,14cyclictest2696544-21kworker/1:100:10:011
2609950995742,14cyclictest2684511-21kworker/1:023:57:401
2609950995742,14cyclictest2684511-21kworker/1:023:53:411
2609950995742,14cyclictest2654887-21kworker/1:121:35:151
2609950995742,14cyclictest2654626-21kworker/1:221:25:011
2609950995742,14cyclictest2323756-1kworker/1:2H23:02:401
2609950995742,14cyclictest2323756-1kworker/1:2H00:15:151
2609950995742,14cyclictest2323756-1kworker/1:2H00:00:171
2609950995740,16cyclictest2702020-21kworker/1:000:30:111
2609950995739,17cyclictest2674359-21kworker/1:122:55:141
2609950995737,18cyclictest2658190-21kworker/1:221:40:151
2609950995734,21cyclictest2323756-1kworker/1:2H22:00:001
2609950995642,14cyclictest2658190-21kworker/1:222:00:331
2609950995642,14cyclictest2615196-21kworker/1:119:50:011
2609950995641,14cyclictest2684511-21kworker/1:023:40:141
2609950995641,14cyclictest2678911-21kworker/1:223:10:111
2609950995641,14cyclictest2639847-21kworker/1:220:45:011
2609950995641,14cyclictest2639847-21kworker/1:220:40:231
2609950995641,14cyclictest2611401-21kworker/1:019:20:181
2609950995641,14cyclictest2611401-21kworker/1:019:20:171
2609950995640,15cyclictest2658190-21kworker/1:221:45:001
2609950995639,16cyclictest2629769-21kworker/1:120:15:151
2609953995551,2cyclictest2611400-21kworker/3:119:20:493
2609953995551,2cyclictest2611400-21kworker/3:119:20:493
2609950995540,14cyclictest2674359-21kworker/1:122:43:411
2609950995539,15cyclictest2665637-21kworker/1:022:20:181
2609950995539,15cyclictest2654887-21kworker/1:121:30:131
2609950995538,16cyclictest2684511-21kworker/1:023:20:111
2609953995450,2cyclictest2109825-1kworker/3:1H00:26:073
2609953995446,3cyclictest2665133-21kworker/3:122:49:053
2609951995448,3cyclictest2677316-21kworker/2:100:15:362
2609951995447,4cyclictest367561-1kworker/2:1H23:11:152
2609951995446,4cyclictest367561-1kworker/2:1H19:50:172
2609950995440,14cyclictest2633120-21kworker/1:020:28:321
2609950995440,13cyclictest2615196-21kworker/1:119:45:151
2609950995439,14cyclictest2641667-21kworker/1:121:00:101
2609950995439,14cyclictest2641667-21kworker/1:120:52:571
2609950995438,15cyclictest2702020-21kworker/1:000:35:181
2609950995438,15cyclictest2665637-21kworker/1:022:25:011
2609950995438,15cyclictest2627161-21kworker/1:020:03:031
2609950995437,16cyclictest2658190-21kworker/1:221:50:111
2609950995437,16cyclictest2651174-21kworker/1:021:15:141
2609953995345,2cyclictest2109825-1kworker/3:1H21:29:543
2609951995346,4cyclictest367561-1kworker/2:1H00:03:412
2609950995338,14cyclictest2696544-21kworker/1:100:26:351
2609950995338,14cyclictest2684511-21kworker/1:023:45:071
2609950995338,14cyclictest2674359-21kworker/1:122:45:001
2609950995338,14cyclictest2665637-21kworker/1:022:30:111
2609950995338,14cyclictest2641667-21kworker/1:120:58:341
2609950995338,14cyclictest2615196-21kworker/1:119:55:131
2609945995350,2cyclictest2641410-21kworker/0:120:45:200
2609945995349,2cyclictest2666999-21kworker/0:122:25:400
2609953995249,2cyclictest2603653-21kworker/3:219:12:083
2609953995248,2cyclictest2651132-21kworker/3:121:18:403
2609953995247,3cyclictest2109825-1kworker/3:1H19:51:333
2609951995246,4cyclictest2677316-21kworker/2:123:34:112
2609951995244,5cyclictest367561-1kworker/2:1H21:30:132
2609950995237,14cyclictest2665637-21kworker/1:022:38:411
2609950995237,14cyclictest2633120-21kworker/1:020:20:061
2609950995236,15cyclictest2684511-21kworker/1:023:35:011
2609950995236,15cyclictest2684511-21kworker/1:023:28:411
2609945995248,2cyclictest2670511-21kworker/0:022:33:070
2609945995248,2cyclictest2660063-21kworker/0:322:07:460
2609945995247,3cyclictest3427659-1kworker/0:0H22:14:400
2609953995149,1cyclictest2633054-21kworker/3:120:24:373
2609953995147,3cyclictest2639776-21kworker/3:020:50:013
2609953995147,3cyclictest2109825-1kworker/3:1H23:20:213
2609953995147,3cyclictest2109825-1kworker/3:1H22:38:403
2609953995146,3cyclictest2696123-21kworker/3:100:15:153
2609953995146,3cyclictest2109825-1kworker/3:1H21:47:413
2609945995146,3cyclictest2620038-21kworker/0:219:37:580
2609953995046,3cyclictest2617062-21kworker/3:020:02:583
2609953995045,3cyclictest2109825-1kworker/3:1H23:06:413
2609950995035,14cyclictest2665637-21kworker/1:022:15:241
2609950995031,18cyclictest2684511-21kworker/1:000:05:011
2609945995046,3cyclictest2689355-21kworker/0:023:49:400
2609945995046,3cyclictest2677247-21kworker/0:123:20:270
2609945995046,2cyclictest2690990-21kworker/0:223:44:120
2609945995045,3cyclictest3427659-1kworker/0:0H19:25:190
2609945995045,3cyclictest2699877-21kworker/0:000:20:400
2609953994946,2cyclictest2703389-21kworker/3:200:37:063
2609953994944,3cyclictest2665133-21kworker/3:122:13:083
2609953994944,3cyclictest2109825-1kworker/3:1H00:24:543
2609951994943,3cyclictest367561-1kworker/2:1H21:05:152
2609950994935,14cyclictest2678911-21kworker/1:223:08:421
2609945994946,2cyclictest3427659-1kworker/0:0H19:11:580
2609945994946,2cyclictest2652738-21kworker/0:121:31:020
2609945994945,2cyclictest3427659-1kworker/0:0H22:59:410
2609945994944,3cyclictest3427659-1kworker/0:0H22:04:400
2609945994944,3cyclictest2677247-21kworker/0:123:03:400
2609945994936,11cyclictest2666999-21kworker/0:122:15:120
2609953994845,2cyclictest2639776-21kworker/3:021:10:563
2609953994845,2cyclictest2638156-21kworker/3:220:34:333
2609953994844,3cyclictest2109825-1kworker/3:1H19:44:563
2609953994844,2cyclictest2617062-21kworker/3:019:56:373
2609953994843,2cyclictest2109825-1kworker/3:1H00:07:083
2609951994842,4cyclictest367561-1kworker/2:1H23:54:412
2609945994845,2cyclictest2677247-21kworker/0:123:17:500
2609945994845,2cyclictest2677247-21kworker/0:122:54:460
2609945994845,2cyclictest2652738-21kworker/0:121:41:080
2609945994844,2cyclictest2699877-21kworker/0:000:18:170
2609953994744,2cyclictest2617062-21kworker/3:020:16:343
2609953994742,3cyclictest2109825-1kworker/3:1H22:09:053
2609953994739,6cyclictest2109825-1kworker/3:1H23:59:413
2609951994740,5cyclictest367561-1kworker/2:1H00:39:412
2609945994744,2cyclictest3427659-1kworker/0:0H21:18:060
2609945994743,3cyclictest3427659-1kworker/0:0H23:10:020
2609945994742,3cyclictest2629770-21kworker/0:120:15:100
2609945994741,3cyclictest3427659-1kworker/0:0H00:00:160
2609953994643,2cyclictest2684233-21kworker/3:123:18:413
2609953994643,2cyclictest2665133-21kworker/3:122:28:123
2609951994640,4cyclictest2677316-21kworker/2:100:20:352
2609945994643,2cyclictest2677247-21kworker/0:123:37:120
2609945994643,2cyclictest2608294-21kworker/0:019:20:560
2609945994643,2cyclictest2608294-21kworker/0:019:20:550
2609953994542,2cyclictest2691000-21kworker/3:200:04:563
2609953994541,3cyclictest2633054-21kworker/3:120:27:423
2609953994541,3cyclictest2109825-1kworker/3:1H21:23:073
2609953994541,3cyclictest2109825-1kworker/3:1H21:09:403
2609953994541,3cyclictest2109825-1kworker/3:1H21:00:193
2609953994541,3cyclictest2109825-1kworker/3:1H19:15:443
2609953994540,3cyclictest2109825-1kworker/3:1H20:40:153
2609945994542,2cyclictest3427659-1kworker/0:0H20:55:160
2609945994542,2cyclictest2608294-21kworker/0:019:47:430
2609945994541,3cyclictest2608294-21kworker/0:019:15:390
2609945994541,2cyclictest3427659-1kworker/0:0H20:10:000
2609945994541,2cyclictest2699877-21kworker/0:000:34:060
2609945994541,2cyclictest2652738-21kworker/0:121:50:470
2609945994541,2cyclictest2608294-21kworker/0:019:30:300
2609953994441,2cyclictest2661629-21kworker/3:222:00:383
2609953994440,3cyclictest2639776-21kworker/3:020:56:053
2609953994440,3cyclictest2109825-1kworker/3:1H22:34:053
2609951994438,4cyclictest367561-1kworker/2:1H22:30:412
2609951994436,4cyclictest367561-1kworker/2:1H19:15:162
2609950994441,2cyclictest0-21swapper/122:06:421
2609945994442,1cyclictest2692865-21kworker/0:200:07:410
2609945994441,2cyclictest3427659-1kworker/0:0H23:30:580
2609945994441,2cyclictest2623288-21kworker/0:219:57:440
2609945994440,3cyclictest2608294-21kworker/0:019:40:220
2609945994440,2cyclictest2647900-21kworker/0:221:10:460
2609945994439,3cyclictest3427659-1kworker/0:0H20:25:140
2609953994340,2cyclictest2691000-21kworker/3:223:50:053
2609953994340,2cyclictest2109825-1kworker/3:1H22:44:563
2609953994339,3cyclictest2617062-21kworker/3:019:45:433
2609953994339,3cyclictest2109825-1kworker/3:1H22:56:413
2609953994339,2cyclictest2665133-21kworker/3:122:23:403
2609953994337,3cyclictest2109825-1kworker/3:1H00:15:003
2609951994337,3cyclictest367561-1kworker/2:1H23:00:172
2609951994337,3cyclictest367561-1kworker/2:1H00:26:352
2609945994340,2cyclictest2623288-21kworker/0:220:32:320
2609945994339,2cyclictest2649764-21kworker/0:021:25:480
2609953994239,2cyclictest2654884-21kworker/3:021:34:163
2609953994239,2cyclictest2615197-21kworker/3:219:29:563
2609953994238,3cyclictest2680751-21kworker/3:223:14:543
2609953994237,3cyclictest2109825-1kworker/3:1H22:15:163
2609951994236,4cyclictest367561-1kworker/2:1H20:37:322
2609951994236,3cyclictest367561-1kworker/2:1H20:45:002
2609951994235,4cyclictest367561-1kworker/2:1H19:33:322
2609945994238,3cyclictest3427659-1kworker/0:0H23:57:390
2609945994237,3cyclictest3427659-1kworker/0:0H23:25:120
2609953994138,2cyclictest2696123-21kworker/3:100:34:103
2609953994138,2cyclictest2660014-21kworker/3:121:53:063
2609953994138,2cyclictest2654884-21kworker/3:021:38:413
2609953994138,2cyclictest2109825-1kworker/3:1H23:34:553
2609953994136,4cyclictest2109825-1kworker/3:1H23:00:463
2609953994136,3cyclictest2109825-1kworker/3:1H22:54:063
2609953994136,3cyclictest2109825-1kworker/3:1H21:44:543
2609953994135,3cyclictest2109825-1kworker/3:1H20:50:173
2609951994136,3cyclictest367561-1kworker/2:1H20:55:402
2609951994136,2cyclictest367561-1kworker/2:1H21:40:152
2609951994135,3cyclictest367561-1kworker/2:1H00:30:102
2609951994134,4cyclictest2658128-21kworker/2:022:49:592
2609945994137,3cyclictest3427659-1kworker/0:0H19:52:310
2609945994137,2cyclictest2677247-21kworker/0:123:07:510
2609945994136,3cyclictest3427659-1kworker/0:0H21:05:010
2609953994037,2cyclictest2617062-21kworker/3:020:14:333
2609953994036,3cyclictest2109825-1kworker/3:1H19:31:093
2609953994035,3cyclictest2109825-1kworker/3:1H23:36:053
2609951994035,3cyclictest367561-1kworker/2:1H21:45:262
2609951994035,2cyclictest367561-1kworker/2:1H23:15:112
2609945994037,2cyclictest2704991-21kworker/0:100:39:050
2609945994037,2cyclictest2656486-21kworker/0:221:35:390
2609945994037,2cyclictest2652738-21kworker/0:121:23:400
2609945994034,3cyclictest3427659-1kworker/0:0H20:50:170
2609953993936,2cyclictest2687744-21kworker/3:023:42:163
2609953993935,2cyclictest2109825-1kworker/3:1H23:26:083
2609951993934,3cyclictest367561-1kworker/2:1H22:51:412
2609951993934,3cyclictest367561-1kworker/2:1H20:00:002
2609951993931,4cyclictest367561-1kworker/2:1H20:31:282
2609945993936,2cyclictest3427659-1kworker/0:0H23:51:400
2609945993936,2cyclictest2672125-21kworker/0:222:39:400
2609951993833,3cyclictest367561-1kworker/2:1H21:27:412
2609951993833,2cyclictest367561-1kworker/2:1H21:20:012
2609951993832,3cyclictest367561-1kworker/2:1H22:44:562
2609945993835,2cyclictest3427659-1kworker/0:0H00:14:050
2609945993835,2cyclictest2699877-21kworker/0:000:29:410
2609945993835,2cyclictest2668617-21kworker/0:222:20:420
2609951993732,3cyclictest367561-1kworker/2:1H23:20:172
2609951993730,4cyclictest2668637-21latency_hist22:20:012
2609945993734,2cyclictest3427659-1kworker/0:0H20:43:320
2609945993734,2cyclictest2658129-21kworker/0:021:48:110
2609945993734,2cyclictest2647900-21kworker/0:221:05:450
2609953993631,2cyclictest2109825-1kworker/3:1H21:55:133
2609951993631,3cyclictest367561-1kworker/2:1H19:20:182
2609951993631,3cyclictest367561-1kworker/2:1H19:20:172
2609945993634,1cyclictest2670511-21kworker/0:022:40:260
2609945993631,3cyclictest2623288-21kworker/0:220:10:250
2609953993531,2cyclictest2109825-1kworker/3:1H20:35:143
2609951993531,3cyclictest367561-1kworker/2:1H23:58:402
2609951993531,3cyclictest367561-1kworker/2:1H22:12:052
2609951993531,3cyclictest367561-1kworker/2:1H19:10:172
2609951993531,2cyclictest367561-1kworker/2:1H23:40:142
2609951993528,4cyclictest367561-1kworker/2:1H20:50:172
2609945993531,2cyclictest3427659-1kworker/0:0H20:35:240
2609951993424,6cyclictest367561-1kworker/2:1H20:15:142
2609951993328,3cyclictest367561-1kworker/2:1H21:13:412
2609951993326,4cyclictest367561-1kworker/2:1H19:40:002
260995199323,12cyclictest2657488-21memory21:35:182
2609951993228,3cyclictest367561-1kworker/2:1H20:21:332
260995199316,23cyclictest2620961-21kernelversion19:40:172
2609951993127,2cyclictest367561-1kworker/2:1H22:21:122
2609951993125,3cyclictest2643047-21munin-run20:50:012
260995199306,16cyclictest2615451-21/usr/sbin/munin19:25:142
2609950993010,1cyclictest2610004-21latency_hist19:10:011
2609953992926,2cyclictest2109825-1kworker/3:1H23:48:403
2609953992919,8cyclictest2628432-21apt-get20:05:143
260994599291,4cyclictest2627102-21diskstats20:00:130
260995199284,15cyclictest0-21swapper/220:00:142
2609951992826,1cyclictest2624955-21kworker/2:120:11:322
2609951992823,3cyclictest367561-1kworker/2:1H22:35:132
2609951992819,6cyclictest2689379-21apt23:40:012
2609951992819,6cyclictest2647921-21sh21:05:002
260995199281,6cyclictest2634958-21apt-get20:25:122
2609951992816,9cyclictest384844-21systemd-journal00:15:002
260995199281,21cyclictest0-21swapper/223:00:002
2609951992724,2cyclictest2621724-21kworker/2:019:45:142
2609951992723,3cyclictest2658128-21kworker/2:021:55:122
2609951992722,3cyclictest367561-1kworker/2:1H20:05:162
2609951992717,6cyclictest2691279-21latency_hist23:50:012
2609945992724,2cyclictest2623288-21kworker/0:220:25:010
260995199267,6cyclictest384844-21systemd-journal22:05:012
260995199260,14cyclictest2652845-21latency_hist21:19:592
2609953992522,2cyclictest2617062-21kworker/3:019:39:553
2609951992518,4cyclictest0-21swapper/221:50:152
2609951992516,7cyclictest2665417-21idleruntime-cro22:10:002
2609945992521,2cyclictest3427659-1kworker/0:0H21:55:010
260995199244,8cyclictest2686357-21cat23:30:002
260995199232,10cyclictest2681055-21latency_hist23:10:012
2609951992216,3cyclictest2610011-21latency_hist19:10:012
2609951992211,5cyclictest384844-21systemd-journal22:30:012
260994599193,3cyclictest151rcu_preempt19:10:000
2609945991916,2cyclictest2670511-21kworker/0:022:48:200
260995399160,4cyclictest2610022-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional