You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-23 - 20:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Nov 23, 2024 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
295702239162,24sleep20-21swapper/207:09:022
294732239162,24sleep00-21swapper/007:07:470
294142238163,23sleep30-21swapper/307:07:023
296432203165,24sleep10-21swapper/107:09:581
29746999490,3cyclictest0-21swapper/211:33:342
296062690,5sleep232-21ksoftirqd/208:20:152
153482570,7sleep3401ktimersoftd/309:00:223
166662530,3sleep10-21swapper/111:35:151
2975199406,24cyclictest32374-21latency12:10:203
2975199406,24cyclictest32374-21latency12:10:203
29746994036,3cyclictest0-21swapper/209:06:172
2975199386,11cyclictest10458-21unixbench_singl11:15:263
2975199377,22cyclictest0-21swapper/310:10:163
29751993716,12cyclictest30551-21/usr/sbin/munin12:05:253
2975199368,8cyclictest11825-21latency_hist08:55:013
29746993513,3cyclictest28072-21/usr/sbin/munin10:45:182
29746993511,5cyclictest0-21swapper/210:20:192
2975199349,9cyclictest21396-21cat11:45:193
29751993411,11cyclictest836-21runrttasks08:45:413
29751993410,7cyclictest26450-21kworker/u8:209:25:093
29746993413,3cyclictest18038-21/usr/sbin/munin07:55:102
29746993412,3cyclictest3273-21/usr/sbin/munin08:35:242
29746993411,4cyclictest8876-21ls12:30:222
2973499348,12cyclictest22835-21munin-run09:20:010
2975199339,17cyclictest25008-21sed10:35:263
2975199338,9cyclictest27807-21df_inode09:30:203
2975199337,7cyclictest86550irq/126-enp1s0-09:50:173
29751993310,8cyclictest23450-21crond10:35:013
2974699331,8cyclictest0-21swapper/210:25:132
2974699331,6cyclictest0-21swapper/210:35:182
29746993312,3cyclictest3555-21/usr/sbin/munin12:20:212
29746993311,3cyclictest6824-21dump-pmu-power07:30:012
29746993311,3cyclictest13647-21/usr/sbin/munin07:45:252
29746993310,4cyclictest22450-21acpi08:05:112
2975199329,15cyclictest31099-21cat08:25:013
29751993214,9cyclictest7444-21chrt12:26:363
2974699329,4cyclictest29820-21fschecks_time12:05:172
2974699323,11cyclictest0-21swapper/210:08:102
29746993229,2cyclictest0-21swapper/210:31:162
2974699321,12cyclictest0-21swapper/208:04:132
2974699320,13cyclictest0-21swapper/208:18:302
2974199324,22cyclictest23397-40dnf10:34:331
2975199319,13cyclictest31898-21latency_hist07:15:023
2975199318,10cyclictest30691-21ntp_kernel_err09:35:193
2975199317,15cyclictest6090-21/usr/sbin/munin09:55:163
2975199317,13cyclictest4519-21/usr/sbin/munin11:05:263
29751993114,8cyclictest19934-21smart_sda09:10:293
29751993111,12cyclictest22594-21runrttasks09:17:053
29751993110,12cyclictest27110-21/usr/sbin/munin12:00:253
2974699312,5cyclictest30953-21hddtemp_smartct10:50:192
2974699311,7cyclictest1505-21ntpq07:15:222
2974699311,5cyclictest18086-21latency_hist11:40:032
2974699311,4cyclictest0-21swapper/211:56:012
29746993111,3cyclictest20712-21expr11:45:132
2974699311,13cyclictest0-21swapper/209:50:302
2974699311,12cyclictest0-21swapper/208:27:172
2974699310,14cyclictest0-21swapper/211:01:182
2974699310,14cyclictest0-21swapper/208:33:492
2973499317,11cyclictest0-21swapper/012:00:000
2975199308,13cyclictest0-21swapper/312:00:013
29751993014,7cyclictest20243-21/usr/sbin/munin08:00:113
2974699301,5cyclictest27844-21latency_hist10:45:032
2974699301,4cyclictest27236-21perf09:30:022
29746993011,3cyclictest9967-21cat07:35:162
2974699301,13cyclictest0-21swapper/211:10:282
2974699301,13cyclictest0-21swapper/210:58:542
2973499308,10cyclictest0-21swapper/010:30:010
2973499302,15cyclictest17817-21munin-run07:55:010
2973499302,15cyclictest17817-21munin-run07:55:010
2975199299,15cyclictest0-21swapper/312:32:233
2975199299,12cyclictest784-21NetworkManager07:50:333
2975199299,12cyclictest784-21NetworkManager07:50:333
2975199299,12cyclictest14979-21ntp_states11:30:183
2975199298,13cyclictest780-21rngd09:07:313
2975199297,12cyclictest1336-21/usr/sbin/munin12:15:193
2975199297,11cyclictest12927-21chrt11:23:133
2975199293,20cyclictest0-21swapper/311:10:293
2975199292,21cyclictest0-21swapper/310:00:593
2975199292,21cyclictest0-21swapper/310:00:593
29751992919,6cyclictest14911-21smart_sda07:45:273
29751992910,6cyclictest28072-21/usr/sbin/munin10:45:233
29751992910,6cyclictest1694-21/usr/sbin/munin09:45:173
2974699291,7cyclictest0-21swapper/210:13:422
2974699291,6cyclictest14634-21snmpd08:44:022
2974699291,4cyclictest10889-21awk11:20:022
2974699291,4cyclictest0-21swapper/212:04:262
29746992912,3cyclictest10931-21ntpq08:50:232
2974699291,16cyclictest0-21swapper/209:42:302
29746992911,4cyclictest12980-21grep07:40:262
29746992911,3cyclictest1401-21ls12:15:132
2974699291,13cyclictest0-21swapper/209:58:492
2973499291,13cyclictest26616-21munin-run08:15:010
2975199289,16cyclictest13686-21sh11:30:013
2975199289,15cyclictest2665-21/usr/sbin/munin07:20:253
2975199288,9cyclictest3273-21/usr/sbin/munin08:35:223
2975199287,11cyclictest4857-21/usr/sbin/munin07:25:173
2975199285,7cyclictest11782-21sed10:05:253
29751992819,5cyclictest8120-21ntp_states07:30:243
29751992814,11cyclictest32219-21/usr/sbin/munin09:40:143
29751992811,8cyclictest23119-21hddtemp_smartct08:05:153
29751992811,6cyclictest86550irq/126-enp1s0-11:35:313
2974699281,7cyclictest32585-21ntp_states12:10:232
2974699281,7cyclictest32585-21ntp_states12:10:232
2974699281,5cyclictest0-21swapper/209:35:132
2974699281,13cyclictest0-21swapper/209:21:422
283252280,4sleep12974199cyclictest12:00:241
2975199279,15cyclictest11450-21/usr/sbin/munin07:40:153
2975199278,6cyclictest27025-21cstates08:15:143
2975199278,5cyclictest86550irq/126-enp1s0-10:50:153
29751992719,5cyclictest2840-40dnf08:32:313
29751992718,6cyclictest10583-21grep12:35:173
29751992713,11cyclictest19650-21sshd07:55:313
29751992711,5cyclictest18254-21proc_pri10:20:243
2974699273,3cyclictest0-21swapper/211:53:252
2974699273,2cyclictest86750irq/128-enp1s0-09:45:162
2974699271,13cyclictest0-21swapper/209:31:422
2974699271,12cyclictest0-21swapper/212:37:012
2974699271,12cyclictest0-21swapper/209:11:022
2974699271,11cyclictest0-21swapper/211:08:242
2974699271,11cyclictest0-21swapper/210:16:292
2974699271,11cyclictest0-21swapper/207:53:092
2974699271,11cyclictest0-21swapper/207:53:092
29746992710,2cyclictest226542sleep209:17:522
2973499271,11cyclictest0-21swapper/011:16:010
2975199269,14cyclictest32228-21grep08:25:193
2975199268,11cyclictest86550irq/126-enp1s0-07:15:033
2975199263,13cyclictest0-21swapper/310:15:263
29751992619,4cyclictest26226-21diskstats10:40:143
29751992617,6cyclictest3904-21diskstats12:20:153
2975199261,18cyclictest0-21swapper/309:20:143
29751992610,7cyclictest32480-21/usr/sbin/munin10:55:113
29751992610,4cyclictest12936-21wc08:55:173
29751992610,4cyclictest12936-21wc08:55:173
2974699261,5cyclictest9628-21latency_hist08:50:022
2974699261,13cyclictest0-21swapper/211:41:572
2974699261,11cyclictest0-21swapper/208:13:182
2974699261,10cyclictest0-21swapper/207:13:172
29741992613,9cyclictest0-21swapper/111:28:441
2975199254,9cyclictest19606-21expr10:25:133
29751992518,4cyclictest23059-21diskstats11:50:143
29751992518,4cyclictest18666-21/usr/sbin/munin11:40:133
2974699254,3cyclictest12050irq/123-ahci[0009:05:022
2974699251,4cyclictest0-21swapper/210:02:502
2974699251,4cyclictest0-21swapper/210:02:502
2974699251,14cyclictest0-21swapper/208:58:142
2974699251,14cyclictest0-21swapper/208:58:142
2974699250,12cyclictest0-21swapper/211:24:302
2974199255,7cyclictest8740-40dnf07:31:341
2974199251,21cyclictest26622-21perf08:15:011
29734992510,13cyclictest0-21swapper/010:35:010
29751992417,4cyclictest24993-21diskstats08:10:153
29751992415,6cyclictest9601-21diskstats07:35:143
29751992415,6cyclictest5470-21/usr/sbin/munin08:40:233
29751992415,6cyclictest2314-21/usr/sbin/munin11:00:203
2974699241,11cyclictest0-21swapper/207:34:232
2974699241,10cyclictest0-21swapper/212:28:252
2974199241,4cyclictest2963-21swap09:45:241
2974199241,2cyclictest0-21swapper/110:55:121
2974199240,9cyclictest0-21swapper/108:55:311
2974199240,9cyclictest0-21swapper/108:55:311
2973499245,16cyclictest29431-21munin-run09:35:010
2973499241,12cyclictest20293-21munin-run11:45:010
2973499241,11cyclictest31905-21munin-run07:15:010
2973499241,11cyclictest12411-21sshd10:09:310
2974199239,6cyclictest3617-1kworker/u9:111:16:421
29741992318,2cyclictest4579-21netstat12:20:211
2974199231,3cyclictest19911-21grep10:25:161
2974199231,19cyclictest27520-21chrt10:40:561
2974199231,19cyclictest25125-21sed09:25:011
2973499235,8cyclictest21139-21kworker/u8:011:02:240
2973499235,4cyclictest24364-21sshd08:09:310
2973499231,8cyclictest11450-21/usr/sbin/munin07:40:150
2973499231,13cyclictest0-21swapper/011:24:460
2973499231,11cyclictest29845-21sshd10:47:310
2973499231,11cyclictest23680-21/usr/sbin/munin10:35:180
92712221,2sleep10-21swapper/108:45:321
29741992218,3cyclictest0-21swapper/107:55:201
29741992216,3cyclictest12022-21memory11:20:181
2974199221,14cyclictest0-21swapper/111:33:181
29741992211,3cyclictest29655-21/usr/sbin/munin09:35:241
29741992211,3cyclictest24411-21latency_hist08:10:011
2974199220,20cyclictest0-21swapper/107:15:251
2973499228,3cyclictest31084-21sshd12:07:310
2973499223,8cyclictest86750irq/128-enp1s0-12:15:180
2973499221,6cyclictest0-21swapper/009:25:280
2973499221,10cyclictest0-21swapper/012:10:190
2973499221,10cyclictest0-21swapper/012:10:190
2974199219,3cyclictest0-21swapper/112:09:041
2974199218,8cyclictest0-21swapper/111:55:151
2974199218,2cyclictest5983-21grep11:05:251
2974199212,8cyclictest11213-21ntp_states12:35:251
29741992117,2cyclictest307522sleep108:21:461
29741992116,3cyclictest11768-21cstates07:40:141
2974199211,19cyclictest0-21swapper/110:17:331
2974199211,19cyclictest0-21swapper/107:20:181
2974199211,18cyclictest0-21swapper/109:11:001
2974199211,13cyclictest0-21swapper/110:01:311
2974199211,13cyclictest0-21swapper/110:01:311
2973499216,3cyclictest86750irq/128-enp1s0-12:20:140
2973499215,3cyclictest86550irq/126-enp1s0-10:20:220
2973499212,4cyclictest32594-21tr10:55:120
2973499211,7cyclictest16498-21NetworkManager10:15:320
2973499211,7cyclictest0-21swapper/007:23:220
2973499211,5cyclictest0-21swapper/010:50:590
2974699201,6cyclictest0-21swapper/211:27:492
2974699200,11cyclictest0-21swapper/207:20:052
2974199206,3cyclictest0-21swapper/110:46:211
29741992017,2cyclictest0-21swapper/108:42:291
2974199201,6cyclictest5252-21sort09:50:301
29741992013,2cyclictest0-21swapper/108:25:221
2974199201,2cyclictest0-21swapper/107:35:201
2974199201,18cyclictest0-21swapper/112:25:541
2974199201,18cyclictest0-21swapper/110:36:271
2974199201,18cyclictest0-21swapper/108:54:541
2974199201,18cyclictest0-21swapper/107:25:031
2974199201,17cyclictest0-21swapper/110:10:281
2974199201,17cyclictest0-21swapper/107:45:071
2974199201,13cyclictest0-21swapper/112:31:521
2973499205,4cyclictest7189-21cat12:25:250
2973499205,3cyclictest86750irq/128-enp1s0-11:35:120
2973499203,2cyclictest86750irq/128-enp1s0-12:30:310
2973499201,7cyclictest0-21swapper/011:11:360
2973499201,5cyclictest11549-21sshd08:51:310
2973499201,4cyclictest0-21swapper/011:07:480
2973499201,10cyclictest0-21swapper/008:32:310
293752201,2sleep10-21swapper/109:34:291
317732191,2sleep10-21swapper/107:13:311
2974199199,2cyclictest0-21swapper/110:22:111
2974199196,4cyclictest28825-21awk08:20:011
2974199192,4cyclictest31345-21ntp_states10:50:231
2974199192,3cyclictest86750irq/128-enp1s0-12:10:201
2974199192,3cyclictest86750irq/128-enp1s0-12:10:201
2974199191,6cyclictest0-21swapper/111:01:311
2974199191,4cyclictest6696-21gltestperf09:55:151
2974199191,17cyclictest0-21swapper/111:51:161
2974199191,17cyclictest0-21swapper/111:40:151
2974199191,17cyclictest0-21swapper/111:13:221
2974199191,17cyclictest0-21swapper/109:42:391
2974199191,17cyclictest0-21swapper/109:04:391
2974199191,17cyclictest0-21swapper/108:03:471
29741991910,3cyclictest25581-21expr09:25:121
2974199190,6cyclictest0-21swapper/112:15:211
2974199190,18cyclictest0-21swapper/109:07:181
2974199190,18cyclictest0-21swapper/107:52:261
2974199190,18cyclictest0-21swapper/107:52:261
2973499197,9cyclictest20912-21hddtemp_smartct08:00:160
2973499197,3cyclictest20113-21sort09:10:310
2973499193,4cyclictest86750irq/128-enp1s0-09:40:220
2973499191,8cyclictest0-21swapper/012:35:180
2973499191,6cyclictest9253-21/usr/sbin/munin07:35:180
2973499191,6cyclictest6090-21/usr/sbin/munin09:55:130
2973499191,6cyclictest31670-21cat09:40:020
2973499191,6cyclictest24095-21timerandwakeup11:50:240
2973499191,6cyclictest16461-21/usr/sbin/munin09:05:220
2973499191,5cyclictest8285-21/usr/sbin/munin10:00:150
2973499191,5cyclictest8285-21/usr/sbin/munin10:00:150
2973499191,3cyclictest13647-21/usr/sbin/munin07:45:130
29734991912,4cyclictest1062-21head07:15:160
2973499191,16cyclictest0-21swapper/008:45:310
17582190,2sleep10-21swapper/108:30:181
144792191,2sleep00-21swapper/010:12:500
66582181,3sleep00-21swapper/007:27:590
29741991814,3cyclictest0-21swapper/111:45:191
2974199180,17cyclictest0-21swapper/108:35:121
2974199180,16cyclictest0-21swapper/109:15:061
2973499182,3cyclictest3888-21seq09:50:090
2973499181,6cyclictest0-21swapper/012:03:200
2973499181,6cyclictest0-21swapper/010:41:580
2973499181,6cyclictest0-21swapper/009:45:030
2973499181,5cyclictest14254-21/usr/sbin/munin09:00:140
2973499181,4cyclictest23894-21cut09:20:180
29734991813,3cyclictest14517-21grep11:30:130
29734991812,3cyclictest5535-21cpuspeed_turbos08:40:110
122312181,2sleep10-21swapper/110:07:131
2973499171,6cyclictest0-21swapper/011:25:520
2973499171,2cyclictest12280-21cstates08:55:120
2973499171,2cyclictest12280-21cstates08:55:120
29734991712,3cyclictest5016-21seq08:37:150
2973499171,13cyclictest20516-21/usr/sbin/munin11:45:200
29734991710,3cyclictest18152-21cpuspeed_turbos07:55:110
2973499161,14cyclictest0-21swapper/008:15:340
29734991610,3cyclictest836-21runrttasks08:20:030
29734991610,3cyclictest8198-21pmu-power07:30:250
29734991510,3cyclictest31549-21cat08:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional