You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 04:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 02, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
854722940,5sleep00-21swapper/019:05:130
84482240164,24sleep10-21swapper/119:05:071
103922236162,23sleep30-21swapper/319:09:493
101152228153,25sleep20-21swapper/219:06:132
2286421590,3sleep11050599cyclictest23:50:201
296682550,4sleep121-21rcuc/121:10:241
224952510,1sleep20-21swapper/222:20:132
1050199421,29cyclictest18724-21/usr/sbin/munin23:40:170
1050199402,34cyclictest2567-40dnf00:22:310
1050199401,27cyclictest4305-21/usr/sbin/munin23:00:160
1050199401,26cyclictest28390-21basename21:10:010
1050199401,24cyclictest836-21runrttasks22:30:170
1050599394,28cyclictest0-21swapper/122:45:421
1050199396,21cyclictest86750irq/128-enp1s0-21:15:120
1050199391,26cyclictest14064-21processes21:55:250
1050199386,22cyclictest17610-21/usr/sbin/munin20:45:270
1050199384,24cyclictest24569-21/usr/sbin/munin19:40:260
1050199381,29cyclictest0-21swapper/000:33:270
1050199381,24cyclictest86750irq/128-enp1s0-20:20:200
1050199379,13cyclictest7782-21munin-run00:40:010
10501993715,19cyclictest3383-21munin-run00:30:010
1050199370,27cyclictest0-21swapper/020:50:230
1050199370,27cyclictest0-21swapper/020:50:230
1051199361,24cyclictest0-21swapper/222:00:312
1050199361,32cyclictest10434-21perf21:50:000
1050199361,31cyclictest0-21swapper/022:10:130
1050199361,25cyclictest0-21swapper/021:45:010
1050199361,21cyclictest6010-21/usr/sbin/munin20:15:140
10501993611,12cyclictest6898-21munin-run23:10:020
10501993611,12cyclictest6898-21munin-run23:10:010
1050599351,14cyclictest0-21swapper/119:42:381
1050599350,16cyclictest0-21swapper/122:57:331
1050199358,12cyclictest15202-21idleruntime-cro20:40:000
1050199355,20cyclictest86550irq/126-enp1s0-00:15:170
1050199351,4cyclictest22372-21/usr/sbin/munin19:35:260
1050199351,22cyclictest21290-21sendmail_mailqu22:15:230
1050199351,16cyclictest0-21swapper/023:29:230
1050599341,25cyclictest0-21swapper/122:38:381
1050599341,24cyclictest0-21swapper/123:17:331
1050599341,15cyclictest0-21swapper/121:24:061
1050599341,13cyclictest0-21swapper/120:10:061
1050199343,20cyclictest86750irq/128-enp1s0-19:45:180
1050199341,5cyclictest10918-21cat20:30:020
1050199341,30cyclictest0-21swapper/022:43:220
10501993410,13cyclictest11013-21/usr/sbin/munin20:30:140
1050599331,23cyclictest0-21swapper/119:45:501
1050599331,14cyclictest0-21swapper/121:46:461
1050599331,13cyclictest14634-21snmpd20:33:021
1050199332,27cyclictest23940-40dnf22:21:310
1050199331,3cyclictest970-21/usr/sbin/munin20:00:230
1050199331,28cyclictest10591-21/usr/sbin/munin23:15:250
1050199331,24cyclictest0-21swapper/023:56:030
10501993311,20cyclictest1876-21idleruntime-cro22:55:000
1051199321,22cyclictest0-21swapper/222:46:092
1050599329,2cyclictest0-21swapper/121:00:141
1050599321,14cyclictest18927-21ntp_states22:10:181
1050599321,12cyclictest0-21swapper/119:25:321
1050599321,12cyclictest0-21swapper/119:13:091
1050199329,20cyclictest8705-21users23:10:230
1050199325,19cyclictest0-21swapper/020:55:220
1050199322,27cyclictest15786-21/usr/sbin/munin19:20:150
1050199321,28cyclictest10644-21/usr/sbin/munin21:50:150
1050199321,27cyclictest14577-21strings19:15:210
1050199321,21cyclictest0-21swapper/023:30:590
10501993210,18cyclictest19550-21unixbench_singl19:25:270
1050199320,19cyclictest0-21swapper/021:20:170
1051199311,24cyclictest0-21swapper/221:57:272
1051199311,23cyclictest0-21swapper/223:22:472
1050599311,25cyclictest0-21swapper/100:02:301
1050599311,23cyclictest0-21swapper/123:57:091
1050599311,23cyclictest0-21swapper/121:44:301
1050199313,20cyclictest0-21swapper/023:35:220
1050199311,3cyclictest26702-21/usr/sbin/munin00:00:180
1050199311,3cyclictest13261-21chrt23:22:140
1050199311,26cyclictest3451-21/usr/sbin/munin21:30:210
1050199311,26cyclictest16200-21/usr/sbin/munin20:40:180
1050199311,18cyclictest29110-21taskset00:10:100
1050199311,18cyclictest29110-21taskset00:10:100
1050199311,18cyclictest0-21swapper/022:55:190
10501993110,18cyclictest16157-21pmu-power22:00:210
1050599301,22cyclictest0-21swapper/122:27:501
1050599301,22cyclictest0-21swapper/100:34:451
1050599301,19cyclictest0-21swapper/123:31:171
1050599301,11cyclictest0-21swapper/123:11:411
1050599301,11cyclictest0-21swapper/120:09:341
1050599301,10cyclictest0-21swapper/122:51:341
1050599300,13cyclictest0-21swapper/120:58:321
1050199309,18cyclictest21301-21latency_hist23:50:030
1050199301,5cyclictest0-21swapper/022:08:040
1050199301,2cyclictest86850irq/129-enp1s0-21:35:130
1050199301,26cyclictest28969-21/usr/sbin/munin19:50:320
1050199301,26cyclictest28969-21/usr/sbin/munin19:50:320
1050199301,18cyclictest0-21swapper/022:38:320
1050199301,17cyclictest0-21swapper/000:05:250
1050199301,16cyclictest31176-21grep19:55:120
10511992915,5cyclictest10644-21/usr/sbin/munin21:50:082
1051199291,3cyclictest0-21swapper/223:25:112
1051199291,3cyclictest0-21swapper/222:30:552
1051199291,3cyclictest0-21swapper/219:57:352
10511992913,7cyclictest11771-21ntp_states19:10:232
1051199291,19cyclictest0-21swapper/221:04:412
1051199291,17cyclictest0-21swapper/220:36:232
1051199291,14cyclictest0-21swapper/200:10:432
1051199291,14cyclictest0-21swapper/200:10:432
1051199291,12cyclictest0-21swapper/219:24:432
1050599291,18cyclictest0-21swapper/119:54:541
1050599291,18cyclictest0-21swapper/119:54:541
1050599291,14cyclictest0-21swapper/122:34:221
1050599291,11cyclictest0-21swapper/123:40:531
1050599291,11cyclictest0-21swapper/120:18:061
1050599291,11cyclictest0-21swapper/100:38:131
1050599291,11cyclictest0-21swapper/100:27:491
1050199296,19cyclictest29248-21head21:10:190
1050199291,3cyclictest21289-21/usr/sbin/munin19:30:260
1050199291,25cyclictest11730-21/usr/sbin/munin19:10:220
1050199291,16cyclictest0-21swapper/022:27:130
1051599281,7cyclictest0-21swapper/323:49:213
1051199281,22cyclictest0-21swapper/221:28:472
1051199281,13cyclictest0-21swapper/221:38:332
1051199281,12cyclictest0-21swapper/223:11:192
1051199281,12cyclictest0-21swapper/220:57:592
1051199281,11cyclictest0-21swapper/222:11:192
1051199281,11cyclictest0-21swapper/200:33:152
1051199281,11cyclictest0-21swapper/200:15:532
1051199280,15cyclictest5893-21cpuspeed_turbos00:35:142
1050599286,3cyclictest0-21swapper/123:47:251
1050599281,23cyclictest0-21swapper/122:05:541
1050599281,13cyclictest0-21swapper/119:21:491
1050599281,12cyclictest0-21swapper/121:17:101
1050599281,11cyclictest0-21swapper/120:35:021
1050599281,11cyclictest0-21swapper/120:27:181
1050599281,10cyclictest0-21swapper/120:41:341
1050599281,10cyclictest0-21swapper/100:10:461
1050599281,10cyclictest0-21swapper/100:10:461
1050599280,14cyclictest0-21swapper/121:32:181
1050199285,20cyclictest4396-21gltestperf20:10:140
1050199281,24cyclictest21876-21diskstats23:50:100
10501992810,15cyclictest836-21runrttasks20:09:510
10501992810,15cyclictest26238-21wc21:05:010
1051599277,10cyclictest18805-21kernelversion19:25:193
1051199272,13cyclictest12716-21kworker/2:219:26:572
1051199271,9cyclictest0-21swapper/220:42:172
1051199271,8cyclictest0-21swapper/220:51:052
1051199271,8cyclictest0-21swapper/220:51:052
1051199271,3cyclictest0-21swapper/223:57:512
1051199271,13cyclictest0-21swapper/222:08:272
1051199271,13cyclictest0-21swapper/220:31:192
1051199271,13cyclictest0-21swapper/219:52:312
1051199271,13cyclictest0-21swapper/219:52:312
1051199271,11cyclictest0-21swapper/223:46:312
1051199271,11cyclictest0-21swapper/221:23:192
1051199271,10cyclictest0-21swapper/223:33:592
1051199270,12cyclictest0-21swapper/223:40:232
1050599271,17cyclictest0-21swapper/119:17:571
1050599271,11cyclictest0-21swapper/121:09:021
1050599271,10cyclictest0-21swapper/122:44:301
1050599271,10cyclictest0-21swapper/122:21:581
1050599270,12cyclictest0-21swapper/121:25:181
10501992717,2cyclictest23966-21kworker/0:222:48:150
1050199271,23cyclictest3227-21awk21:30:010
1051199267,8cyclictest0-21swapper/219:46:392
1051199262,14cyclictest11301-21awk23:20:012
1051199261,9cyclictest0-21swapper/222:55:512
1051199261,11cyclictest0-21swapper/219:32:232
1051199260,14cyclictest0-21swapper/222:16:172
1050599266,11cyclictest0-21swapper/120:46:381
1050599261,20cyclictest0-21swapper/122:01:501
1050599261,11cyclictest0-21swapper/100:06:211
1050599261,10cyclictest0-21swapper/123:28:211
1050599261,10cyclictest0-21swapper/123:20:371
100402261,21sleep310128-21irqstats23:15:193
1051199257,10cyclictest0-21swapper/223:35:302
1051199256,9cyclictest0-21swapper/200:07:342
1051199256,3cyclictest0-21swapper/221:09:032
1051199251,3cyclictest0-21swapper/222:29:422
1051199251,16cyclictest0-21swapper/200:29:432
1051199251,13cyclictest0-21swapper/223:02:392
1051199251,12cyclictest0-21swapper/222:43:192
1051199251,11cyclictest0-21swapper/219:40:552
1051199251,10cyclictest0-21swapper/221:34:172
1051199251,10cyclictest0-21swapper/221:17:112
1051199251,10cyclictest0-21swapper/220:10:552
1050599251,9cyclictest0-21swapper/122:16:061
1050599251,12cyclictest0-21swapper/120:20:061
1050599251,11cyclictest0-21swapper/120:04:461
1050599251,11cyclictest0-21swapper/119:55:341
1050599251,11cyclictest0-21swapper/100:15:101
1051599240,16cyclictest86550irq/126-enp1s0-00:26:203
1051199248,8cyclictest9752-21taskset21:44:282
1051199248,10cyclictest0-21swapper/220:00:252
1051199246,7cyclictest0-21swapper/200:24:232
1051199241,9cyclictest0-21swapper/221:12:312
1051199241,9cyclictest0-21swapper/220:28:332
1051199241,11cyclictest0-21swapper/221:48:092
1051199241,10cyclictest0-21swapper/222:54:312
1051199241,10cyclictest0-21swapper/219:37:032
1051199241,10cyclictest0-21swapper/200:04:552
1050599241,9cyclictest0-21swapper/121:37:181
1050599241,15cyclictest0-21swapper/123:09:331
1050599241,15cyclictest0-21swapper/123:09:331
1050599241,12cyclictest0-21swapper/119:35:341
1050599241,11cyclictest0-21swapper/123:02:531
1050599241,11cyclictest0-21swapper/121:57:261
1051599239,3cyclictest836-21runrttasks21:21:133
10515992310,3cyclictest16542-21awk19:20:183
1051199236,3cyclictest0-21swapper/220:21:472
1050599238,2cyclictest21742-21grep19:30:301
10505992314,3cyclictest19992-21cstates20:50:201
10505992314,3cyclictest19992-21cstates20:50:201
1050599231,10cyclictest0-21swapper/121:52:061
1050599231,10cyclictest0-21swapper/100:20:541
1051599225,3cyclictest86850irq/129-enp1s0-21:10:293
1051599221,7cyclictest0-21swapper/321:36:033
1051199228,3cyclictest0-21swapper/220:45:352
1051199225,3cyclictest0-21swapper/220:08:192
1051199221,9cyclictest0-21swapper/219:18:392
1051199221,8cyclictest0-21swapper/223:50:592
1051199221,7cyclictest0-21swapper/223:07:592
1051199221,7cyclictest0-21swapper/223:07:592
1051199221,10cyclictest0-21swapper/220:17:192
1050599221,3cyclictest0-21swapper/123:35:451
1051599212,8cyclictest13274-40dnf23:22:263
1051599211,3cyclictest0-21swapper/322:11:083
1051599211,16cyclictest17756-21tr20:45:133
1051599211,10cyclictest11457-21irqstats21:50:133
1051199211,11cyclictest0-21swapper/222:36:472
1051599205,8cyclictest86850irq/129-enp1s0-19:35:223
1051599201,7cyclictest0-21swapper/323:12:563
1051599201,6cyclictest0-21swapper/323:03:583
1051599201,5cyclictest20402-21diskstats22:15:133
1051599201,5cyclictest0-21swapper/321:31:153
1051599201,16cyclictest23238-21find20:55:243
1051599201,10cyclictest28277-21unixbench_multi19:45:263
1051599201,10cyclictest22700-21wc23:50:183
1051599191,9cyclictest0-21swapper/320:15:273
1051599191,9cyclictest0-21swapper/319:10:133
1051599191,6cyclictest0-21swapper/300:04:403
1051599191,5cyclictest0-21swapper/322:39:433
1051599191,11cyclictest0-21swapper/323:59:493
1051599191,10cyclictest8204-21/usr/sbin/munin20:20:133
1051599185,4cyclictest15439-21grep20:40:133
1051599183,6cyclictest0-21swapper/300:15:003
1051599183,6cyclictest0-21swapper/300:15:003
1051599182,13cyclictest25007-21irqstats21:00:183
1051599181,9cyclictest13218-21/usr/sbin/munin20:35:193
1051599181,9cyclictest0-21swapper/322:58:513
1051599181,7cyclictest0-21swapper/321:05:273
1051599181,7cyclictest0-21swapper/300:07:173
1051599181,6cyclictest670-21sh20:00:013
1051599181,3cyclictest0-21swapper/321:55:193
10515991812,3cyclictest29595-21cut22:40:133
10515991812,3cyclictest19814-21/usr/sbin/munin20:50:263
10515991812,3cyclictest19814-21/usr/sbin/munin20:50:263
1051599181,15cyclictest0-21swapper/323:06:273
1051599181,15cyclictest0-21swapper/323:06:273
1051599181,15cyclictest0-21swapper/321:15:123
1051599181,15cyclictest0-21swapper/319:50:563
1051599181,15cyclictest0-21swapper/319:50:563
1051599181,10cyclictest842-21lldpd00:21:203
1051599181,10cyclictest5807-21/usr/sbin/munin00:35:143
1051599181,10cyclictest5486-21sshd20:11:213
1051599181,10cyclictest32268-21/usr/sbin/munin22:50:203
1051599181,10cyclictest0-21swapper/300:15:143
138032170,2sleep30-21swapper/323:26:183
127542171,2sleep30-21swapper/320:32:093
1051599175,3cyclictest86850irq/129-enp1s0-22:20:233
1051599173,8cyclictest0-21swapper/323:44:223
1051599171,9cyclictest0-21swapper/323:36:433
1051599171,9cyclictest0-21swapper/322:02:423
1051599171,7cyclictest26652-21users22:30:223
1051599171,7cyclictest2167-21ps20:00:243
1051599171,7cyclictest0-21swapper/319:30:293
1051599171,6cyclictest8829-21needreboot21:40:203
1051599171,6cyclictest0-21swapper/321:25:513
10515991712,3cyclictest5076-21unixbench-2d00:30:263
10515991712,3cyclictest24735-21cstates19:40:143
1051599171,13cyclictest13588-21/usr/sbin/munin19:15:123
1051599171,10cyclictest842-21lldpd22:07:483
1051599170,5cyclictest0-21swapper/323:30:103
1051599161,9cyclictest0-21swapper/322:45:423
1051599161,7cyclictest0-21swapper/322:25:443
1051599161,6cyclictest3337-21chrt20:06:593
1051599161,4cyclictest0-21swapper/321:48:453
1051599141,10cyclictest10787-21latency_hist20:30:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional