You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 15:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 27, 2024 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
187262242165,25sleep10-21swapper/107:06:271
189382237164,22sleep30-21swapper/307:09:113
188262204166,25sleep20-21swapper/207:07:452
186382201166,23sleep00-21swapper/007:05:290
1121021780,4sleep31910799cyclictest10:30:133
19101999794,2cyclictest0-21swapper/211:15:012
318252750,2sleep00-21swapper/008:50:110
42872650,4sleep230-21rcuc/210:15:122
184612620,4sleep218465-21fschecks_time12:00:172
225372580,4sleep20-21swapper/212:10:132
19088993610,12cyclictest23166-21idleruntime-cro09:45:010
1908899313,13cyclictest26963-21awk08:40:010
1909699301,21cyclictest0-21swapper/109:58:361
1909699301,18cyclictest31862-21hddtemp_smartct12:30:181
1908899301,14cyclictest6793-21awk11:35:010
1908899291,12cyclictest12398-21/usr/sbin/munin09:20:280
1910199283,8cyclictest7034-40dnf07:53:332
1909699286,3cyclictest0-21swapper/109:35:191
1909699286,3cyclictest0-21swapper/109:35:191
1909699281,12cyclictest0-21swapper/109:44:301
1908899280,13cyclictest1806-21awk10:10:020
1909699272,11cyclictest17169-21awk10:45:011
1909699271,18cyclictest2880-21taskset12:38:471
1909699271,18cyclictest0-21swapper/109:53:511
1909699271,17cyclictest20855-21smart_sda10:50:271
1909699270,21cyclictest0-21swapper/110:00:191
1908899272,13cyclictest26568-21munin-run12:20:010
1908899271,9cyclictest0-21swapper/009:50:160
19096992620,3cyclictest28762-21idleruntime-cro12:25:021
1909699261,6cyclictest18771-21/usr/sbin/munin12:00:201
1909699261,17cyclictest0-21swapper/112:06:061
1909699261,17cyclictest0-21swapper/110:45:201
1909699261,17cyclictest0-21swapper/110:45:201
1908899261,13cyclictest12170-21munin-run09:20:010
7402250,2sleep00-21swapper/007:40:100
1910199250,23cyclictest0-21swapper/207:15:302
1909699251,18cyclictest0-21swapper/111:54:451
1908899251,12cyclictest11574-21munin-run08:05:020
315402240,4sleep121-21rcuc/107:35:141
1910199246,10cyclictest0-21swapper/210:00:192
1909699249,9cyclictest0-21swapper/111:18:331
1909699241,17cyclictest0-21swapper/111:48:261
1909699241,17cyclictest0-21swapper/111:48:261
1909699241,17cyclictest0-21swapper/111:31:031
1909699241,16cyclictest0-21swapper/111:06:361
1909699241,16cyclictest0-21swapper/110:26:551
1909699241,16cyclictest0-21swapper/109:49:151
1909699241,16cyclictest0-21swapper/109:24:231
1909699241,16cyclictest0-21swapper/109:05:321
1909699241,10cyclictest24096-21cpuspeed_turbos11:00:121
1909699240,15cyclictest0-21swapper/112:15:281
1908899241,9cyclictest9356-21sh08:00:010
1909699237,10cyclictest0-21swapper/110:23:591
19096992319,3cyclictest0-21swapper/110:38:391
19096992319,3cyclictest0-21swapper/107:19:471
19096992318,4cyclictest0-21swapper/108:25:161
19096992318,3cyclictest0-21swapper/111:40:281
1909699231,5cyclictest0-21swapper/112:27:201
1909699231,5cyclictest0-21swapper/111:56:131
1909699231,19cyclictest15266-21hddtemp_smartct09:25:171
1909699231,19cyclictest14333-21df08:10:131
1909699231,17cyclictest0-21swapper/110:30:511
1909699231,16cyclictest0-21swapper/111:14:121
1909699231,16cyclictest0-21swapper/110:08:421
1909699231,16cyclictest0-21swapper/109:15:261
1909699231,16cyclictest0-21swapper/108:45:151
1909699230,16cyclictest0-21swapper/111:22:121
19088992316,4cyclictest30594-21tail10:00:150
1908899231,3cyclictest0-21swapper/011:02:000
1908899231,10cyclictest291972interrupts11:10:200
1910799222,12cyclictest12995-21/usr/sbin/munin10:35:133
19107992216,3cyclictest6766-21head07:50:263
1910799221,5cyclictest0-21swapper/312:34:193
1910199229,4cyclictest0-21swapper/209:54:342
1910199221,11cyclictest0-21swapper/212:38:462
1909699221,15cyclictest0-21swapper/111:35:331
1909699221,15cyclictest0-21swapper/110:11:091
1909699221,15cyclictest0-21swapper/109:12:201
1909699221,15cyclictest0-21swapper/108:55:251
1909699221,15cyclictest0-21swapper/108:50:081
1909699221,12cyclictest9359-21latency_hist08:00:021
1909699221,12cyclictest24974-21munin-node08:35:121
1909699221,11cyclictest0-21swapper/110:55:031
1909699220,6cyclictest0-21swapper/110:18:071
1909699220,20cyclictest0-21swapper/108:06:451
227392211,2sleep00-21swapper/007:15:190
21542211,3sleep00-21swapper/011:22:100
1910799216,6cyclictest0-21swapper/312:10:193
19107992110,5cyclictest0-21swapper/310:16:093
19107992110,3cyclictest13469-21ntp_states09:20:223
1910199211,8cyclictest23379-21/usr/sbin/munin09:45:292
1910199211,8cyclictest0-21swapper/212:20:142
1910199211,7cyclictest0-21swapper/208:50:552
19096992117,3cyclictest0-21swapper/108:22:461
19096992117,3cyclictest0-21swapper/107:50:191
19096992117,3cyclictest0-21swapper/107:23:191
19096992113,3cyclictest0-21swapper/109:31:201
1909699211,2cyclictest0-21swapper/107:25:171
1909699211,19cyclictest0-21swapper/107:12:251
1909699211,15cyclictest0-21swapper/111:25:281
1909699211,15cyclictest0-21swapper/109:02:011
1909699211,14cyclictest0-21swapper/108:40:051
1909699211,11cyclictest3001-21/usr/sbin/munin07:45:251
1909699210,6cyclictest0-21swapper/112:10:081
1908899212,14cyclictest86750irq/128-enp1s0-12:05:230
1908899211,13cyclictest0-21swapper/009:57:160
1908899211,11cyclictest25890-21unixbench_multi07:20:280
19088992110,3cyclictest6723-21memory09:05:210
1910799209,3cyclictest9419-21irqstats10:25:193
1910799209,3cyclictest7776-21diskmemload12:07:253
1910799206,6cyclictest0-21swapper/312:20:043
1910799206,4cyclictest0-21swapper/312:00:313
1910799206,3cyclictest7776-21diskmemload09:50:363
1910799205,5cyclictest0-21swapper/309:25:573
1910799205,5cyclictest0-21swapper/309:18:303
1910799202,15cyclictest19864-21grep08:20:243
1910799202,10cyclictest32180-21/usr/sbin/munin10:05:253
1910799202,10cyclictest2614-21/usr/sbin/munin11:25:243
1910799201,6cyclictest0-21swapper/309:41:553
1910799201,12cyclictest0-21swapper/309:45:353
1910799201,11cyclictest30311-21df10:00:123
1910799201,11cyclictest19591-21/usr/sbin/munin10:50:193
1910799201,11cyclictest16192-21/usr/sbin/munin08:15:163
1910799200,8cyclictest26448-21kworker/u8:012:20:023
1910199208,2cyclictest30-21rcuc/210:55:232
1910199206,4cyclictest780-21rngd11:25:252
1910199206,3cyclictest86550irq/126-enp1s0-11:00:242
1910199205,3cyclictest12050irq/123-ahci[0007:40:012
1910199202,6cyclictest21184-21/usr/sbin/munin09:40:132
1910199202,16cyclictest10570-21dump-pmu-power10:30:012
1910199202,15cyclictest16839-21cut08:15:192
1910199202,15cyclictest15806-21grep10:40:152
1910199201,6cyclictest28166-21users07:25:302
1910199201,6cyclictest27004-21irqstats11:05:202
19101992013,4cyclictest32143-21grep12:30:222
1910199201,16cyclictest15437-21grep08:10:302
1910199201,10cyclictest0-21swapper/211:43:442
1910199201,10cyclictest0-21swapper/210:06:162
19096992016,3cyclictest0-21swapper/108:31:031
19096992016,3cyclictest0-21swapper/108:31:031
1909699201,11cyclictest9593-21/usr/sbin/munin08:00:211
1909699201,10cyclictest0-21swapper/108:16:581
19088992014,3cyclictest8983-21munin-run11:40:010
1908899201,11cyclictest27897-21basename08:40:200
1908899201,11cyclictest22634-21kernelversion10:55:220
1908899201,11cyclictest12995-21/usr/sbin/munin10:35:150
1908899201,10cyclictest8726-21idleruntime09:10:180
1908899201,10cyclictest19295-21/usr/sbin/munin07:10:130
1908899201,10cyclictest0-21swapper/008:16:070
19088992010,3cyclictest86750irq/128-enp1s0-11:27:490
201772191,2sleep00-21swapper/008:22:510
1910799198,3cyclictest7090-21/usr/sbin/munin10:20:163
1910799198,3cyclictest0-21swapper/311:09:333
1910799196,4cyclictest0-21swapper/309:11:393
1910799195,5cyclictest0-21swapper/311:21:243
1910799191,7cyclictest4203-21kworker/u8:011:00:423
1910799191,6cyclictest28944-21perl11:10:163
19107991915,2cyclictest30970-21awk12:30:023
1910799191,4cyclictest0-21swapper/310:40:053
1910799191,3cyclictest0-21swapper/311:46:473
1910799191,3cyclictest0-21swapper/311:46:473
1910799191,11cyclictest0-21swapper/311:58:033
1910799191,11cyclictest0-21swapper/310:59:303
1910799191,10cyclictest7346-21df_inode11:35:113
1910799191,10cyclictest4813-21/usr/sbin/munin11:30:153
1910799191,10cyclictest3603-21/usr/sbin/munin09:00:303
1910799191,10cyclictest30966-21/usr/sbin/munin07:35:133
1910799191,10cyclictest1383-21/usr/sbin/munin08:55:143
1910199195,3cyclictest86750irq/128-enp1s0-11:55:132
1910199195,3cyclictest86550irq/126-enp1s0-08:00:322
1910199194,5cyclictest28991-21/usr/sbin/munin12:25:182
1910199191,7cyclictest0-21swapper/209:57:102
1910199191,6cyclictest0-21swapper/208:20:232
1910199191,5cyclictest24597-21/usr/sbin/munin12:15:212
1910199191,5cyclictest17388-21/usr/sbin/munin10:45:222
1910199191,5cyclictest17388-21/usr/sbin/munin10:45:222
1910199191,5cyclictest12662-21smart_sda11:45:262
1910199191,5cyclictest12662-21smart_sda11:45:262
1910199191,5cyclictest0-21swapper/210:14:212
19101991914,3cyclictest32-21ksoftirqd/208:40:012
19101991913,3cyclictest8006-21/usr/sbin/munin09:10:192
19101991913,3cyclictest14669-21ntp_states11:50:242
1910199191,11cyclictest0-21swapper/211:16:242
1910199191,10cyclictest0-21swapper/210:37:492
1910199191,10cyclictest0-21swapper/209:39:002
1910199191,10cyclictest0-21swapper/209:39:002
1909699197,2cyclictest29633-21sed07:30:181
1909699191,17cyclictest0-21swapper/107:40:081
1908899198,3cyclictest13598-21/usr/sbin/munin11:50:300
1908899192,5cyclictest17163-21munin-run10:45:010
19088991912,4cyclictest3699-21cpuspeed_turbos09:00:140
1908899191,15cyclictest32554-21munin-run11:20:020
1908899191,11cyclictest0-21swapper/008:05:110
1908899191,10cyclictest23379-21/usr/sbin/munin09:45:150
1908899191,10cyclictest0-21swapper/011:57:020
1908899191,10cyclictest0-21swapper/010:34:150
1908899191,10cyclictest0-21swapper/010:18:060
119502190,4sleep21910199cyclictest10:30:252
71232181,2sleep00-21swapper/007:54:370
28382181,2sleep30-21swapper/312:38:173
1910799186,5cyclictest2-21kthreadd11:16:373
1910799186,3cyclictest0-21swapper/309:36:103
1910799186,3cyclictest0-21swapper/309:36:103
1910799183,8cyclictest0-21swapper/309:57:323
1910799182,13cyclictest16788-21/usr/sbin/munin09:30:213
1910799181,7cyclictest29378-21gltestperf07:30:143
19107991813,3cyclictest7391-21/usr/sbin/munin07:55:123
19107991813,3cyclictest18850-21unixbench-2d10:45:273
19107991813,3cyclictest18850-21unixbench-2d10:45:273
19107991811,4cyclictest26466-21grep08:35:283
19107991811,4cyclictest21254-21idleruntime-cro07:15:013
1910799181,11cyclictest0-21swapper/311:50:233
1910799181,11cyclictest0-21swapper/310:11:033
1910199185,3cyclictest86850irq/129-enp1s0-08:05:182
1910199182,7cyclictest457-21kworker/2:009:06:102
1910199182,6cyclictest30-21rcuc/207:20:222
1910199182,3cyclictest0-21swapper/209:22:492
1910199182,15cyclictest0-21swapper/208:33:372
1910199182,15cyclictest0-21swapper/208:33:372
1910199181,6cyclictest0-21swapper/208:48:152
1910199181,5cyclictest6407-21/usr/sbin/munin10:20:132
1910199181,5cyclictest3246-21sshd08:58:332
1910199181,4cyclictest313-21/usr/sbin/munin11:20:142
1910199181,4cyclictest21255-21latency_hist07:15:012
19101991813,3cyclictest3001-21/usr/sbin/munin07:45:272
19101991813,3cyclictest28770-21/usr/sbin/munin07:30:142
19101991813,3cyclictest12164-21dump-pmu-power09:20:012
19101991812,4cyclictest20660-21runrttasks08:25:142
19101991812,3cyclictest3965-21forks09:00:162
1910199181,15cyclictest5557-21if_enp2s011:30:182
1910199181,11cyclictest0-21swapper/209:25:372
1910199181,10cyclictest0-21swapper/210:52:412
1910199181,10cyclictest0-21swapper/209:30:092
1908899181,9cyclictest15915-21sshd08:14:330
1908899181,8cyclictest7776-21diskmemload11:47:290
1908899181,8cyclictest7776-21diskmemload11:47:290
1908899181,7cyclictest0-21swapper/012:31:110
1908899181,7cyclictest0-21swapper/011:05:270
1908899181,7cyclictest0-21swapper/010:28:440
1908899181,6cyclictest0-21swapper/012:24:270
19088991813,3cyclictest18579-21grep12:00:170
19088991812,3cyclictest9547-21df_inode11:40:130
1908899181,14cyclictest7776-21diskmemload10:21:210
1908899181,10cyclictest0-21swapper/007:25:190
76672171,2sleep30-21swapper/309:08:453
261642171,2sleep30-21swapper/307:22:473
1910799177,8cyclictest3992-21needreboot07:45:203
1910799171,4cyclictest9593-21/usr/sbin/munin08:00:133
19107991712,3cyclictest717-21/usr/sbin/munin07:40:153
1910799171,15cyclictest0-21swapper/308:42:483
1910799171,14cyclictest24144-21sh07:20:013
1910799171,13cyclictest30362-21needreboot08:45:213
19107991711,3cyclictest20702-21grep08:25:143
1910199173,3cyclictest86750irq/128-enp1s0-12:05:282
1910199171,4cyclictest8619-21sshd11:35:332
1910199171,15cyclictest0-21swapper/207:56:182
1910199171,15cyclictest0-21swapper/207:41:372
1908899175,4cyclictest7776-21diskmemload12:10:060
1908899173,7cyclictest29371-21/usr/sbin/munin08:45:190
1908899171,9cyclictest0-21swapper/008:30:200
1908899171,9cyclictest0-21swapper/008:30:200
1908899171,8cyclictest0-21swapper/009:35:270
1908899171,8cyclictest0-21swapper/009:35:270
1908899171,8cyclictest0-21swapper/007:45:180
1908899171,7cyclictest0-21swapper/010:50:120
1908899171,7cyclictest0-21swapper/010:12:370
1908899171,7cyclictest0-21swapper/009:30:330
1908899171,4cyclictest28770-21/usr/sbin/munin07:30:200
1908899171,2cyclictest30473-21sed12:25:280
1908899171,15cyclictest0-21swapper/007:35:510
1908899171,14cyclictest0-21swapper/008:25:060
108592171,2sleep30-21swapper/311:41:073
30212160,2sleep00-21swapper/008:55:380
1910799162,12cyclictest12350-21cut08:05:143
1910799161,6cyclictest27656-21awk07:25:233
1910799161,4cyclictest0-21swapper/308:50:233
19107991611,2cyclictest22780-21/usr/sbin/munin08:30:103
19107991611,2cyclictest22780-21/usr/sbin/munin08:30:103
1910799161,10cyclictest0-21swapper/308:11:243
19101991612,2cyclictest28683-21unixbench_multi08:40:312
1908899164,3cyclictest0-21swapper/009:25:060
1908899161,9cyclictest0-21swapper/010:45:050
1908899161,9cyclictest0-21swapper/010:45:050
1908899161,14cyclictest0-21swapper/012:36:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional