You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 03, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
276462238161,24sleep10-21swapper/119:06:501
276462238161,24sleep10-21swapper/119:06:501
275662203166,24sleep30-21swapper/319:05:483
275662203166,24sleep30-21swapper/319:05:473
275652203167,24sleep20-21swapper/219:05:472
275652203167,24sleep20-21swapper/219:05:472
277432202165,24sleep00-21swapper/019:08:060
277432202165,24sleep00-21swapper/019:08:060
106102570,3sleep20-21swapper/222:20:162
134162440,2sleep00-21swapper/019:45:270
27976993810,12cyclictest20463-21awk20:05:010
2797699366,14cyclictest26066-21munin-run21:35:010
2797699351,11cyclictest0-21swapper/019:50:140
2797699331,6cyclictest7055-21kworker/0:119:35:140
2797699331,27cyclictest0-21swapper/000:26:380
2797699331,14cyclictest31653-21kworker/0:123:27:220
2797699331,14cyclictest31653-21kworker/0:123:27:220
2798399321,23cyclictest0-21swapper/122:28:431
2797699321,25cyclictest0-21swapper/000:36:300
2797699311,14cyclictest0-21swapper/023:15:020
2797699310,11cyclictest7637-21kworker/0:020:55:160
2797699301,19cyclictest0-21swapper/000:23:420
2797699301,14cyclictest0-21swapper/021:43:180
2797699301,12cyclictest0-21swapper/022:27:180
2797699301,12cyclictest0-21swapper/021:46:380
2797699300,16cyclictest0-21swapper/023:36:140
2797699300,14cyclictest0-21swapper/022:11:340
27983992919,2cyclictest836-21runrttasks00:26:001
2797699291,18cyclictest0-21swapper/023:22:140
2797699291,13cyclictest0-21swapper/023:11:420
2797699291,12cyclictest0-21swapper/023:45:100
2797699291,12cyclictest0-21swapper/022:57:020
2798399281,19cyclictest0-21swapper/100:14:381
2797699287,3cyclictest0-21swapper/022:33:580
2797699281,13cyclictest0-21swapper/000:04:460
2797699281,12cyclictest0-21swapper/000:13:100
2797699281,11cyclictest0-21swapper/021:57:020
2797699281,11cyclictest0-21swapper/000:16:300
291532270,3sleep20-21swapper/219:10:212
2798399271,6cyclictest19562-21grep21:15:141
2798399271,6cyclictest19562-21grep21:15:141
2797699272,13cyclictest18244-21munin-run20:00:010
2797699272,12cyclictest14195-21kworker/0:221:05:200
2797699271,17cyclictest0-21swapper/000:09:580
2797699271,15cyclictest0-21swapper/023:41:340
2797699271,13cyclictest589-21munin-run19:20:010
2797699271,13cyclictest14634-21snmpd21:01:160
2797699271,13cyclictest0-21swapper/023:08:220
2797699271,12cyclictest0-21swapper/022:40:060
2797699271,12cyclictest0-21swapper/021:27:180
2797699271,11cyclictest7444-21awk20:45:280
2797699271,10cyclictest11751-21kworker/0:023:55:240
2797699270,11cyclictest0-21swapper/023:30:460
2797699270,11cyclictest0-21swapper/022:24:540
2797699265,3cyclictest0-21swapper/023:53:020
2797699261,12cyclictest0-21swapper/022:18:460
2797699261,12cyclictest0-21swapper/020:29:420
2797699261,10cyclictest0-21swapper/022:03:020
2797699261,10cyclictest0-21swapper/021:24:060
2797699261,10cyclictest0-21swapper/019:41:180
2797699260,10cyclictest0-21swapper/022:52:220
2798899251,7cyclictest13692-21df_inode22:30:112
2798399256,10cyclictest14659-21ls22:30:201
2798399251,4cyclictest29753-21sleep23:15:011
2798399251,18cyclictest0-21swapper/122:02:561
2797699251,6cyclictest0-21swapper/019:20:300
2797699251,13cyclictest0-21swapper/022:06:460
2797699251,10cyclictest0-21swapper/022:36:140
27994992418,3cyclictest23007-21grep00:20:273
2798899242,6cyclictest971-40dnf20:31:342
27983992412,6cyclictest29478-21kworker/u8:223:29:421
27983992412,6cyclictest29478-21kworker/u8:223:29:421
2798399241,17cyclictest0-21swapper/123:43:471
2798399241,17cyclictest0-21swapper/121:45:311
2798399241,17cyclictest0-21swapper/120:53:341
2798399241,17cyclictest0-21swapper/120:36:511
2798399241,17cyclictest0-21swapper/120:12:241
2798399241,17cyclictest0-21swapper/120:12:241
2798399241,16cyclictest0-21swapper/121:35:041
2798399241,14cyclictest0-21swapper/123:01:451
2798399240,17cyclictest0-21swapper/119:31:291
2797699246,15cyclictest24854-21awk20:15:010
2797699246,15cyclictest24854-21awk20:15:010
2797699241,12cyclictest0-21swapper/020:16:380
2797699241,11cyclictest0-21swapper/019:34:300
2797699241,10cyclictest0-21swapper/021:51:580
2797699241,10cyclictest0-21swapper/021:35:100
2798899237,5cyclictest0-21swapper/222:52:182
2798399232,15cyclictest20876-21kworker/u8:100:35:481
2798399231,3cyclictest16459-21load22:35:181
2798399231,19cyclictest2979-21grep19:25:021
2798399231,17cyclictest0-21swapper/123:22:131
2798399231,17cyclictest0-21swapper/100:34:311
2798399231,17cyclictest0-21swapper/100:34:311
2798399231,17cyclictest0-21swapper/100:09:451
2798399231,16cyclictest0-21swapper/123:15:401
2798399231,16cyclictest0-21swapper/119:59:461
2798399231,16cyclictest0-21swapper/119:47:391
2798399231,10cyclictest0-21swapper/123:07:361
2798399230,17cyclictest0-21swapper/121:29:031
2797699231,15cyclictest0-21swapper/021:15:500
2797699231,15cyclictest0-21swapper/021:15:500
2797699231,13cyclictest18547-21seq21:11:100
2797699231,12cyclictest0-21swapper/019:14:130
2797699230,13cyclictest0-21swapper/022:45:340
2799499222,17cyclictest32702-21cut23:20:163
2799499221,9cyclictest69452sed23:40:023
2799499221,6cyclictest0-21swapper/321:54:563
2798399223,3cyclictest8739-21fschecks_count22:15:151
2798399222,11cyclictest18156-21smart_sda21:10:271
2798399221,9cyclictest21430-21grep21:20:131
2798399221,6cyclictest6331-21unixbench_multi23:35:261
27983992215,4cyclictest32180-21fschecks_time20:30:181
27983992214,3cyclictest0-21swapper/100:04:531
2798399221,3cyclictest30981-21cat19:15:041
2798399221,19cyclictest0-21swapper/120:40:491
2798399221,16cyclictest0-21swapper/121:30:021
2798399221,16cyclictest0-21swapper/119:44:431
2798399221,11cyclictest15936-21mailstats21:05:251
2798399221,11cyclictest0-21swapper/123:32:401
27983992210,3cyclictest10887-21ntp_states23:50:171
2798399220,5cyclictest0-21swapper/122:23:221
2798399220,2cyclictest13356-21latency21:00:171
2798399220,16cyclictest0-21swapper/122:12:301
2797699226,3cyclictest0-21swapper/023:00:540
2797699225,3cyclictest0-21swapper/020:51:500
2797699221,13cyclictest0-21swapper/000:32:220
2797699221,13cyclictest0-21swapper/000:32:220
2797699221,12cyclictest423-21kworker/0:020:34:200
2799499215,5cyclictest0-21swapper/321:55:583
2799499214,5cyclictest0-21swapper/321:29:133
2799499211,4cyclictest0-21swapper/300:15:533
27994992114,4cyclictest19502-21grep21:15:153
27994992114,4cyclictest19502-21grep21:15:153
2799499211,17cyclictest13575-21head23:55:283
2799499211,12cyclictest0-21swapper/300:04:153
2799499210,8cyclictest0-21swapper/300:34:263
2799499210,8cyclictest0-21swapper/300:34:263
2798899216,4cyclictest5750irq/9-acpi00:35:152
2798899215,4cyclictest224592wc21:20:252
2798899213,6cyclictest5301-21/usr/sbin/munin19:30:162
27988992116,3cyclictest14022-21grep21:00:252
2798899211,4cyclictest0-21swapper/200:25:592
2798399215,8cyclictest22930-21sed00:20:271
2798399211,8cyclictest30979-21grep20:25:261
2798399211,7cyclictest0-21swapper/122:44:001
27983992114,2cyclictest0-21swapper/121:57:121
2798399211,15cyclictest0-21swapper/122:06:011
2798399211,15cyclictest0-21swapper/120:56:131
2798399211,15cyclictest0-21swapper/120:22:591
2798399211,15cyclictest0-21swapper/119:54:381
2798399211,15cyclictest0-21swapper/119:37:181
2798399211,10cyclictest0-21swapper/121:40:401
2798399210,6cyclictest0-21swapper/123:57:051
2798399210,16cyclictest0-21swapper/120:04:291
2797699216,3cyclictest0-21swapper/020:08:460
2799499205,7cyclictest0-21swapper/322:06:283
2799499205,4cyclictest0-21swapper/323:14:143
2799499201,9cyclictest0-21swapper/300:35:183
27994992013,4cyclictest19004-21sensors_temp22:40:253
2799499201,12cyclictest0-21swapper/322:56:083
2799499201,11cyclictest3705-21/usr/sbin/munin20:40:173
2799499201,11cyclictest21286-21/usr/sbin/munin21:20:163
2799499201,11cyclictest0-21swapper/322:17:573
2799499201,10cyclictest24952-21/usr/sbin/munin23:00:273
2799499201,10cyclictest10287-21/usr/sbin/munin20:55:233
2799499200,5cyclictest0-21swapper/323:32:513
2798899206,3cyclictest19697-21gunzip20:00:232
2798899205,3cyclictest24762-21hddtemp_smartct21:30:162
2798899202,5cyclictest876-21systemd-logind23:21:352
2798899202,5cyclictest24251-21/usr/sbin/munin00:30:132
2798899202,5cyclictest24251-21/usr/sbin/munin00:30:132
2798899201,7cyclictest0-21swapper/223:45:162
2798899201,6cyclictest3341-21/usr/sbin/munin22:00:272
27988992014,3cyclictest31729-21sed19:15:192
27988992013,4cyclictest12476-21head19:45:172
2798399204,10cyclictest0-21swapper/122:47:091
2798399203,10cyclictest0-21swapper/121:53:011
2798399201,6cyclictest6503-21gltestperf20:45:171
2798399201,6cyclictest0-21swapper/100:18:261
27983992013,4cyclictest21558-21cat20:05:191
2798399201,12cyclictest0-21swapper/122:50:171
2798399200,6cyclictest0-21swapper/123:46:031
2798399200,4cyclictest22436-21kworker/u8:122:58:571
2798399200,14cyclictest0-21swapper/120:19:331
2797699207,2cyclictest0-21swapper/020:39:180
2797699200,13cyclictest2625-21kworker/0:019:25:320
2799499198,3cyclictest16599-21ntpq22:35:213
2799499196,4cyclictest26903-21gltestperf21:35:173
2799499195,7cyclictest0-21swapper/323:46:033
2799499195,3cyclictest0-21swapper/300:28:023
2799499195,2cyclictest86550irq/126-enp1s0-00:05:423
2799499192,7cyclictest16272-21/usr/sbin/munin19:55:123
2799499192,5cyclictest0-21swapper/323:52:553
2799499191,5cyclictest0-21swapper/321:40:563
27994991915,2cyclictest12050irq/123-ahci[0019:47:343
2799499191,3cyclictest0-21swapper/322:45:463
27994991913,4cyclictest41-21ksoftirqd/319:41:003
2799499191,10cyclictest5301-21/usr/sbin/munin19:30:273
2799499191,10cyclictest25072-21/usr/sbin/munin20:15:173
2799499191,10cyclictest0-21swapper/323:08:423
2799499191,10cyclictest0-21swapper/322:53:383
2799499190,5cyclictest0-21swapper/321:49:473
2798899195,3cyclictest86850irq/129-enp1s0-21:40:192
2798899195,3cyclictest86550irq/126-enp1s0-00:20:152
2798899194,3cyclictest86550irq/126-enp1s0-19:50:182
2798899193,5cyclictest12624-21head23:55:162
2798899192,4cyclictest2224-21ls19:20:252
2798899192,4cyclictest11355-21ntp_states20:55:232
2798899192,14cyclictest30086-21cpuspeed_turbos23:15:102
2798899192,10cyclictest17610-21kworker/2:221:46:422
2798899191,8cyclictest0-21swapper/221:50:222
2798899191,7cyclictest768-21in:imjournal23:30:002
2798899191,7cyclictest768-21in:imjournal23:30:002
2798899191,6cyclictest16272-21/usr/sbin/munin19:55:132
2798899191,6cyclictest0-21swapper/200:00:482
2798899191,5cyclictest768-21in:imjournal23:52:352
27988991911,4cyclictest27912-21cpuspeed_turbos23:10:082
2798399191,10cyclictest5065-21latency_hist19:30:021
2798399190,18cyclictest0-21swapper/119:11:361
2797699191,10cyclictest27268-21/usr/sbin/munin20:20:240
2797699190,11cyclictest0-21swapper/020:42:060
186382191,2sleep30-21swapper/321:12:223
69462181,2sleep30-21swapper/323:40:033
61462181,3sleep30-21swapper/322:10:073
2799499189,3cyclictest0-21swapper/321:00:213
2799499187,3cyclictest0-21swapper/323:26:273
2799499187,3cyclictest0-21swapper/323:26:273
2799499186,5cyclictest0-21swapper/323:16:113
2799499184,3cyclictest0-21swapper/322:30:343
2799499181,5cyclictest0-21swapper/322:29:253
2799499181,5cyclictest0-21swapper/321:31:523
2799499181,3cyclictest4927-21users22:00:313
27994991813,3cyclictest30307-21df_inode19:15:023
27994991812,3cyclictest3491-21cut19:25:163
27994991812,3cyclictest25029-21cut20:15:023
27994991812,3cyclictest25029-21cut20:15:023
2799499181,15cyclictest0-21swapper/319:21:583
2799499181,10cyclictest16657-21diskmemload22:24:193
2799499181,10cyclictest0-21swapper/300:14:353
2798899184,3cyclictest12050irq/123-ahci[0022:10:012
2798899183,7cyclictest0-21swapper/220:44:282
2798899182,5cyclictest0-21swapper/200:07:302
2798899181,6cyclictest0-21swapper/223:34:122
2798899181,6cyclictest0-21swapper/222:27:032
2798899181,5cyclictest27536-21latency_hist23:10:002
2798899181,5cyclictest17706-21irqstats21:10:222
2798899181,4cyclictest16669-21munin-run21:10:012
27988991812,3cyclictest28257-21idleruntime-cro21:40:012
27988991812,3cyclictest25891-21cut20:15:192
2798899181,15cyclictest0-21swapper/222:40:152
27988991811,4cyclictest8119-21dump-pmu-power22:15:012
2799499171,6cyclictest0-21swapper/320:00:203
27994991712,3cyclictest8290-21cat19:35:193
27994991712,3cyclictest30936-21sed20:25:273
2799499171,15cyclictest0-21swapper/320:51:083
2799499171,15cyclictest0-21swapper/320:38:343
2799499171,14cyclictest0-21swapper/320:20:143
2799499171,10cyclictest0-21swapper/321:06:563
2799499171,10cyclictest0-21swapper/319:15:163
2798899175,3cyclictest86550irq/126-enp1s0-21:15:232
2798899175,3cyclictest86550irq/126-enp1s0-21:15:232
2798899172,14cyclictest0-21swapper/220:12:302
2798899172,14cyclictest0-21swapper/220:12:302
2798899172,12cyclictest27916-21gltestperf20:20:162
2798899171,7cyclictest0-21swapper/221:29:472
2798899171,6cyclictest4690-21users19:25:272
2798899171,6cyclictest24923-21cat23:00:012
2798899171,5cyclictest4885-21cpuspeed_turbos23:35:112
2798899171,4cyclictest31603-21cat20:30:022
2798899171,4cyclictest1475-21/usr/sbin/munin21:55:142
27988991712,3cyclictest8350-21/usr/sbin/munin22:15:282
27988991712,3cyclictest17323-21cat00:10:092
27988991712,2cyclictest20679-21/usr/sbin/munin20:05:202
2798899171,15cyclictest0-21swapper/220:49:552
2798899171,14cyclictest0-21swapper/222:39:482
27988991711,3cyclictest9457-21sh19:40:022
27988991711,3cyclictest19704-21cut00:15:152
27988991710,4cyclictest2156-21head20:35:172
159222171,2sleep30-21swapper/319:53:343
2799499161,12cyclictest31673-21/usr/sbin/munin20:30:233
2799499161,12cyclictest20679-21/usr/sbin/munin20:05:183
2798899161,4cyclictest0-21swapper/223:01:362
2798899161,2cyclictest0-21swapper/222:49:442
27988991611,3cyclictest9777-21ls19:40:122
27988991611,3cyclictest836-21runrttasks23:44:152
27988991611,3cyclictest10193-21date20:55:012
2799499155,3cyclictest86550irq/126-enp1s0-20:45:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional