You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-14 - 07:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Mar 14, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
760421290,2sleep07605-21/usr/sbin/munin00:05:260
2854121020,2sleep20-21swapper/223:04:172
2854121020,2sleep20-21swapper/223:04:172
377999986,89cyclictest0-21swapper/019:15:250
3779998812,62cyclictest16316-21cron19:50:090
28442860,3sleep10-21swapper/121:42:111
177192840,7sleep2301ktimersoftd/222:03:252
377999832,6cyclictest3698-21fschecks_time23:15:240
377999801,7cyclictest2491-21fschecks_count20:40:300
377999792,6cyclictest8518-21grep23:20:380
377999791,8cyclictest22845-21/usr/sbin/munin21:25:270
377999782,3cyclictest0-21swapper/022:25:130
377999771,60cyclictest25637-21basename20:15:280
377999771,5cyclictest0-21swapper/021:00:270
3779997611,45cyclictest19758-21sh23:36:520
377999751,6cyclictest29234-21latency_hist22:20:110
377999751,5cyclictest0-21swapper/000:40:280
3779997512,55cyclictest14753-21diskmemload23:41:470
3779997512,55cyclictest14753-21diskmemload23:41:470
377999745,66cyclictest6850-21unixbench_singl19:20:390
377999731,7cyclictest0-21swapper/021:22:390
377999731,7cyclictest0-21swapper/021:22:390
377999731,56cyclictest32436-21fschecks_time23:55:240
377999731,4cyclictest18848-21latency_hist21:20:110
235302730,1sleep10-21swapper/100:28:051
204902730,2sleep30-21swapper/300:24:413
377999721,7cyclictest26921-21sh00:35:100
377999721,6cyclictest8543-21taskset19:25:370
377999721,6cyclictest19027-21ntpq19:55:350
377999719,49cyclictest4332-21awk21:45:100
377999711,58cyclictest14753-21diskmemload00:13:210
377999711,57cyclictest19036-21aten_rbpower_vo22:05:230
377999711,43cyclictest23388-21chrt22:56:340
377999702,5cyclictest4363-21ps00:00:360
3779997015,52cyclictest12877-21sh22:41:100
377999699,49cyclictest24872-21sh22:14:350
377999699,47cyclictest14753-21diskmemload23:12:440
377999694,56cyclictest21289-21ssh22:55:070
377999691,7cyclictest0-21swapper/019:50:270
377999691,7cyclictest0-21swapper/000:27:120
377999681,5cyclictest0-21swapper/020:35:100
377999673,52cyclictest14753-21diskmemload00:18:050
377999672,5cyclictest30136-21grep20:25:370
377999672,5cyclictest30136-21grep20:25:370
377999671,7cyclictest0-21swapper/021:53:030
377999671,5cyclictest0-21swapper/023:52:420
377999671,55cyclictest9-21ksoftirqd/000:24:550
377999662,5cyclictest0-21swapper/023:26:170
377999661,7cyclictest395-21latency_hist23:10:100
377999661,6cyclictest5042-21ssh22:30:580
377999661,5cyclictest0-21swapper/023:05:110
377999661,5cyclictest0-21swapper/023:05:110
377999661,43cyclictest15777-21wc22:45:320
3779996611,40cyclictest0-21swapper/020:46:150
377999657,48cyclictest14078-21sh21:58:510
3779996517,36cyclictest0-21swapper/021:47:340
377999642,4cyclictest0-21swapper/022:01:250
377999632,5cyclictest8050irq/56-dwc_otg_20:15:110
377999631,7cyclictest0-21swapper/000:37:290
378099624,6cyclictest7950irq/56-dwc_otg20:20:101
378099623,6cyclictest0-21swapper/122:35:331
378099623,6cyclictest0-21swapper/122:35:331
377999628,46cyclictest16925-21ssh23:34:090
377999621,5cyclictest288432sleep023:49:400
3779996210,41cyclictest0-21swapper/021:36:320
378099618,5cyclictest8150irq/56-dwc_otg_23:55:101
377999618,39cyclictest7950irq/56-dwc_otg22:35:580
377999618,39cyclictest7950irq/56-dwc_otg22:35:580
377999618,39cyclictest0-21swapper/021:31:190
377999611,6cyclictest7424-21modprobe20:55:100
3779996112,32cyclictest0-21swapper/021:06:060
3779996015,23cyclictest8050irq/56-dwc_otg_20:20:270
377999601,46cyclictest320-21ssh22:24:580
3779996011,40cyclictest10588-21taskset19:31:230
125302600,4sleep2378199cyclictest23:25:462
3781995910,29cyclictest20637-21ssh22:06:292
378099591,13cyclictest4202-21ps23:15:311
377999592,7cyclictest0-21swapper/020:03:500
97482580,1sleep30-21swapper/321:00:263
377999581,5cyclictest0-21swapper/020:57:590
377999581,5cyclictest0-21swapper/020:37:590
3781995718,24cyclictest0-21swapper/220:45:362
378199566,15cyclictest0-21swapper/219:15:252
3781995612,21cyclictest31573-21ls00:40:322
378099562,15cyclictest0-21swapper/120:40:411
378099561,5cyclictest0-21swapper/120:50:211
378099561,11cyclictest7950irq/56-dwc_otg23:55:291
311462560,3sleep2301ktimersoftd/222:21:412
3781995517,7cyclictest17810-21apt-get23:35:182
378099551,14cyclictest9550irq/48-DMA14:55:081
377999552,7cyclictest0-21swapper/020:06:360
378199548,14cyclictest3485-21aten_rbpower_vo23:15:212
378199547,22cyclictest1967-21ls22:25:392
3781995416,20cyclictest8724-21/usr/sbin/munin19:25:392
3781995412,20cyclictest4706-21sed00:00:392
378099543,8cyclictest0-21swapper/123:15:101
378099542,18cyclictest0-21swapper/100:40:181
377999545,40cyclictest0-21swapper/019:42:200
377999545,40cyclictest0-21swapper/019:42:200
377999541,40cyclictest0-21swapper/019:37:510
3781995316,18cyclictest11207-21/usr/sbin/munin21:55:442
378099521,4cyclictest4822-21users00:00:411
377999517,3cyclictest8050irq/56-dwc_otg_21:11:420
378199506,22cyclictest8050irq/56-dwc_otg_22:45:222
378199504,34cyclictest0-21swapper/219:45:052
378199504,34cyclictest0-21swapper/219:45:052
33872500,1sleep30-21swapper/322:28:063
378199497,26cyclictest0-21swapper/219:45:222
3781994911,24cyclictest1982-21ssh23:57:422
378199486,25cyclictest27730-21ssh22:17:342
3781994813,18cyclictest23595-21ssh23:41:502
3781994813,18cyclictest23595-21ssh23:41:502
378199477,25cyclictest0-21swapper/221:00:232
378199477,24cyclictest0-21swapper/220:05:242
378199476,22cyclictest23374-21ssh22:11:332
3781994713,23cyclictest14753-21diskmemload22:41:302
3781994711,19cyclictest9021-21seq20:57:382
378199468,15cyclictest2318-21aten_rbpower_en20:40:272
378199467,25cyclictest0-21swapper/221:40:262
378199467,25cyclictest0-21swapper/220:30:232
3781994614,17cyclictest6417-21ntp_states19:20:342
378199458,22cyclictest16825-21ssh21:16:322
378199457,24cyclictest0-21swapper/221:10:242
378199454,33cyclictest8050irq/56-dwc_otg_22:55:202
3781994528,13cyclictest29349-21ssh21:35:172
3781994512,20cyclictest8356-21sed21:50:112
3781994510,20cyclictest14753-21diskmemload22:39:482
3781994510,20cyclictest14753-21diskmemload22:39:482
31762452,13sleep00-21swapper/019:10:500
31762452,13sleep00-21swapper/019:10:500
378299444,4cyclictest0-21swapper/321:15:303
378199449,15cyclictest31675-21rm23:54:572
378199449,13cyclictest380-21idleruntime-cro23:10:102
378199447,27cyclictest24548-21ssh21:26:282
378199447,22cyclictest15282-21ssh00:16:162
3781994421,4cyclictest11615-21fschecks_count21:05:252
3781994411,10cyclictest25654-21aten_rbpower_po20:15:282
378099442,8cyclictest8150irq/56-dwc_otg_23:40:231
378099442,8cyclictest8150irq/56-dwc_otg_23:40:231
3780994413,15cyclictest0-21swapper/121:15:301
161502440,2sleep316149-21ssh22:45:373
378199439,9cyclictest19412-21ls19:55:402
3781994312,12cyclictest8648-21grep00:05:402
3781994312,12cyclictest19058-21ssh00:21:332
3781994310,18cyclictest21439-21ssh21:23:002
3781994310,18cyclictest21439-21ssh21:23:002
378099431,6cyclictest7950irq/56-dwc_otg21:45:261
378099431,27cyclictest25480-21diskmemload23:00:241
378099431,27cyclictest25480-21diskmemload23:00:241
378099431,24cyclictest0-21swapper/119:15:251
378199428,7cyclictest26736-21chrt00:34:472
378199427,20cyclictest0-21swapper/219:35:282
378199427,12cyclictest20907-21ssh22:54:182
378199423,14cyclictest10503-21rm23:24:312
378099425,28cyclictest12169-21ls19:35:411
304392420,1sleep20-21swapper/200:39:542
177162420,4sleep117691-21fschecks_time00:20:291
378199419,12cyclictest22949-21ssh00:26:482
378199418,18cyclictest28499-21ssh21:34:042
378199418,12cyclictest17158-21missed_timers19:50:322
378199416,22cyclictest27740-21ssh23:47:272
378199416,21cyclictest1560-21ssh23:12:092
378199416,20cyclictest12822-21ssh00:13:572
3781994116,11cyclictest15454-21ssh23:31:122
378199410,14cyclictest7443-21awk20:55:102
3780994121,4cyclictest28947-21idleruntime-cro20:25:101
378099401,4cyclictest8150irq/56-dwc_otg_00:15:191
3782993918,5cyclictest31847-21switchtime00:40:373
378299391,4cyclictest8150irq/56-dwc_otg_22:40:233
378199396,21cyclictest0-21swapper/220:20:242
3781993920,8cyclictest0-21swapper/220:01:092
3782993826,9cyclictest0-21swapper/300:25:163
378199388,12cyclictest8451-21ssh21:50:172
378199386,13cyclictest10796-21taskset19:34:032
3780993816,6cyclictest1106450http21:05:111
378099381,26cyclictest25850-21grep23:45:281
378099381,26cyclictest25820-21cut22:15:241
378299378,14cyclictest0-21swapper/322:59:463
3782993711,8cyclictest22536-21munin-run21:25:103
3782993711,8cyclictest22536-21munin-run21:25:103
378199372,19cyclictest303-21chrt20:35:182
378199372,18cyclictest30437-21seq20:26:262
378199372,18cyclictest30437-21seq20:26:262
378099379,5cyclictest449-21basename22:25:101
378099371,22cyclictest10188-21cat19:30:351
75372360,2sleep10-21swapper/119:25:211
3782993617,5cyclictest1832-21sh23:12:423
3782993615,5cyclictest7374-21/usr/sbin/munin00:05:343
3781993621,7cyclictest8150irq/56-dwc_otg_22:31:012
378099367,12cyclictest0-21swapper/100:14:281
378099366,14cyclictest8150irq/56-dwc_otg_21:10:391
378099364,19cyclictest7950irq/56-dwc_otg22:15:111
378099362,9cyclictest0-21swapper/122:25:291
378099361,24cyclictest0-21swapper/120:25:241
378099361,24cyclictest0-21swapper/120:25:241
320772361,3sleep332086-21ssh22:23:323
260602361,30sleep30-21swapper/322:15:273
378299352,3cyclictest0-21swapper/321:52:493
378299352,19cyclictest7950irq/56-dwc_otg00:40:103
378099356,10cyclictest8050irq/56-dwc_otg_21:54:201
378099352,5cyclictest0-21swapper/121:31:521
3780993512,8cyclictest14737-21kworker/u8:123:00:101
378099351,13cyclictest7950irq/56-dwc_otg21:20:251
378099351,13cyclictest7950irq/56-dwc_otg21:20:251
378299349,6cyclictest8150irq/56-dwc_otg_22:05:373
378299345,6cyclictest7950irq/56-dwc_otg19:45:103
378299345,6cyclictest7950irq/56-dwc_otg19:45:103
378299344,15cyclictest8150irq/56-dwc_otg_21:40:333
378299342,4cyclictest8050irq/56-dwc_otg_23:55:323
378099347,5cyclictest7950irq/56-dwc_otg21:35:291
378099346,4cyclictest8050irq/56-dwc_otg_23:35:131
378099345,4cyclictest0-21swapper/120:00:291
378099342,5cyclictest14735-21munin-run22:00:101
378099341,24cyclictest8150irq/56-dwc_otg_21:26:101
378099341,23cyclictest0-21swapper/121:05:171
378099341,22cyclictest0-21swapper/123:05:301
378099341,17cyclictest7162-21snmpd00:35:311
3780993411,5cyclictest18897-21acpi22:05:191
378299339,11cyclictest0-21swapper/321:57:373
378299338,5cyclictest8050irq/56-dwc_otg_22:35:323
378299338,5cyclictest8050irq/56-dwc_otg_22:35:323
378299335,14cyclictest8150irq/56-dwc_otg_20:40:313
378299334,5cyclictest26837-21sed23:45:403
378299333,15cyclictest8150irq/56-dwc_otg_23:20:243
378299331,21cyclictest26550-21timerwakeupswit20:15:413
378299331,18cyclictest0-21swapper/321:33:273
378199331,3cyclictest8150irq/56-dwc_otg_20:10:222
378099336,5cyclictest15420-21grep22:00:301
378099335,13cyclictest8150irq/56-dwc_otg_20:30:311
3780993316,5cyclictest1292-21runrttasks23:30:361
378099331,23cyclictest0-21swapper/122:33:431
378099331,20cyclictest0-21swapper/123:25:361
378099331,14cyclictest24070-21grep20:10:271
378099331,13cyclictest15434-21fschecks_count22:45:281
117242331,29sleep311729-21ssh00:11:343
378299325,5cyclictest7950irq/56-dwc_otg21:48:553
378299325,14cyclictest8150irq/56-dwc_otg_20:05:323
378299325,13cyclictest8150irq/56-dwc_otg_23:53:253
3782993219,10cyclictest30105-21ssh23:05:363
378299321,6cyclictest8050irq/56-dwc_otg_21:25:253
3782993215,7cyclictest0-21swapper/323:16:323
3782993215,5cyclictest11763-21if_err_eth021:05:273
3782993213,5cyclictest37-21rcuc/323:40:313
3782993213,5cyclictest37-21rcuc/323:40:313
3782993211,5cyclictest37-21rcuc/323:35:183
378099328,3cyclictest0-21swapper/123:20:371
378099326,5cyclictest7950irq/56-dwc_otg19:20:251
378099321,5cyclictest7950irq/56-dwc_otg20:55:301
3780993211,4cyclictest7972-21cat00:05:311
32662323,22sleep10-21swapper/119:12:001
32662323,22sleep10-21swapper/119:12:001
20062326,11sleep30-21swapper/319:10:243
20062326,11sleep30-21swapper/319:10:243
378299312,5cyclictest0-21swapper/320:23:523
3782993123,6cyclictest29330-21/usr/sbin/munin21:35:173
378299311,21cyclictest0-21swapper/322:01:063
378299311,16cyclictest7950irq/56-dwc_otg21:12:213
378099317,4cyclictest0-21swapper/100:34:081
378099316,5cyclictest0-21swapper/122:41:541
378099316,4cyclictest8150irq/56-dwc_otg_19:40:251
378099316,4cyclictest8150irq/56-dwc_otg_19:40:251
378099311,20cyclictest0-21swapper/120:00:101
3780993110,4cyclictest851-21/usr/sbin/munin20:35:331
378299309,13cyclictest37-21rcuc/319:50:373
378299309,10cyclictest0-21swapper/322:53:333
378299306,14cyclictest8150irq/56-dwc_otg_23:30:113
378299305,7cyclictest24999-21ssh00:30:513
378299305,13cyclictest0-21swapper/300:17:063
378299302,21cyclictest12132-21timerwakeupswit19:35:413
378299301,5cyclictest17544-21ssh23:35:113
3782993010,13cyclictest37-21rcuc/323:00:223
3782993010,13cyclictest37-21rcuc/323:00:223
378099306,5cyclictest8050irq/56-dwc_otg_19:45:221
378099305,4cyclictest0-21swapper/122:50:471
378099305,10cyclictest0-21swapper/120:47:481
378099302,17cyclictest0-21swapper/119:55:101
378299299,13cyclictest9543-21/usr/sbin/munin19:30:253
378299295,13cyclictest7140-21sh22:35:103
378299292,5cyclictest0-21swapper/320:55:413
3782992923,4cyclictest7078-21idleruntime-cro00:05:103
3782992923,4cyclictest24265-21ls20:10:293
3782992923,4cyclictest2140-21latency_hist20:40:113
378299291,9cyclictest21933-21latency_hist20:05:113
3782992912,5cyclictest37-21rcuc/320:50:323
378299286,6cyclictest8050irq/56-dwc_otg_20:25:283
378299286,6cyclictest8050irq/56-dwc_otg_20:25:283
378299284,6cyclictest8150irq/56-dwc_otg_20:45:363
378299282,19cyclictest0-21swapper/320:34:173
378299282,17cyclictest0-21swapper/322:11:493
378299275,12cyclictest8150irq/56-dwc_otg_19:55:373
3782992721,4cyclictest5830-21aten_rbpower_vo19:20:253
3782992720,5cyclictest1292-21runrttasks19:15:383
378299271,11cyclictest8050irq/56-dwc_otg_19:25:243
31712273,14sleep20-21swapper/219:10:482
31712273,14sleep20-21swapper/219:10:482
378299261,4cyclictest0-21swapper/319:45:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional