You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-11 - 22:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Apr 11, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
290127737,754sleep22937-21ntpq07:29:502
26982275125,713sleep227018-21ntpq10:44:502
26982275125,713sleep227018-21ntpq10:44:502
280126946,679sleep01357-1kworker/0:0H11:14:480
280126946,679sleep01357-1kworker/0:0H11:14:480
12196253417,504sleep012239-21ntpq07:59:530
31544251213,484sleep231566-21awk07:19:242
18399242915,400sleep018443-21ntpq08:19:530
65792408371,23sleep36581-21sshd07:40:583
65792408371,23sleep36581-21sshd07:40:583
5596233827,300sleep25644-21awk07:39:442
5596233827,300sleep25644-21awk07:39:442
10503231517,284sleep010527-21fschecks_count09:44:420
1376230011,275sleep01384-21ntp_states07:24:520
1689122947,274sleep0370-21snmpd12:06:420
25551228012,204sleep225680-21latency_hist08:44:262
25309224911,146sleep225327-21fschecks_count06:59:452
9924224324,208sleep2370-21snmpd09:41:452
4416224017,210sleep04453-21sed07:34:490
21629223719,202sleep321691-21sendmail_mailst12:24:553
11723222323,177sleep111933-21cp11:49:271
21232221433,123sleep221321-21grep12:24:452
15397221214,116sleep214913-21/usr/sbin/munin08:09:532
3051522029,82sleep230572-21ls10:59:482
1384522008,181sleep1370-21snmpd11:57:221
1219722008,153sleep212249-21if_eth009:49:452
2009821958,27sleep220186-21grep12:19:492
1394219511,170sleep21437-21uname07:24:542
16514219256,84sleep20-21swapper/212:04:542
5654218833,116sleep25776-21cat11:24:562
8636218581,23sleep20-21swapper/209:39:272
5354218212,127sleep25483-21latency_hist07:39:282
3202121699,115sleep20-21swapper/209:04:262
21623216418,13sleep20-21swapper/210:24:592
21623216418,13sleep20-21swapper/210:24:592
20710216244,16sleep20-21swapper/210:23:582
17734216218,126sleep00-21swapper/008:19:240
17734216218,126sleep00-21swapper/008:19:240
15596216018,25sleep20-21swapper/212:04:262
2836921576,73sleep20-21swapper/210:49:462
18715215518,118sleep018734-21latency_hist12:14:260
32619215324,104sleep132758-21unixbench_singl11:04:561
18549215320,63sleep218639-21mailstats08:20:072
1135721487,130sleep2370-21snmpd09:45:172
4113214553,67sleep20-21swapper/209:20:002
12485214215,90sleep324222-1kworker/3:2H09:49:543
10542214162,63sleep210611-21sort07:54:552
10542214162,63sleep210611-21sort07:54:552
32336214057,41sleep20-21swapper/209:04:422
8991213730,26sleep28639-21/usr/sbin/munin07:49:582
8991213730,26sleep28639-21/usr/sbin/munin07:49:582
24833213715,69sleep30-21swapper/310:39:293
1847121365,17sleep20-21swapper/212:11:492
1767221328,57sleep217281-21/usr/sbin/munin10:09:482
8357213010,69sleep20-21swapper/211:38:092
25042212828,23sleep20-21swapper/210:39:412
12084212827,25sleep20-21swapper/207:59:512
120621288,81sleep21269-21if_err_eth011:09:462
18622212522,81sleep118707-21latency_hist10:14:281
3227421238,104sleep232318-21ntpq11:04:492
14045212217,24sleep20-21swapper/211:59:252
26372212123,81sleep20-21swapper/208:44:552
1552021198,16sleep20-21swapper/210:04:272
10971211820,75sleep311038-21switchtime09:44:533
31329211614,80sleep10-21swapper/109:00:001
1957021158,58sleep219567-21apt-get08:24:392
328621148,93sleep23384-21latency_hist11:19:272
328621148,93sleep23384-21latency_hist11:19:272
1891221135,18sleep20-21swapper/210:15:422
13365211326,76sleep2286-21dbus-daemon11:54:242
25065211016,78sleep325025-21apt-get10:39:583
6107210918,11sleep20-21swapper/209:25:582
6107210918,11sleep20-21swapper/209:25:582
2271521090,12sleep30-21swapper/310:29:433
22693210918,76sleep022733-21cpu08:34:430
22693210918,76sleep022733-21cpu08:34:430
7995210812,82sleep28087-21sort09:34:262
4067210810,26sleep20-21swapper/211:19:592
24500210611,55sleep224524-21munin-run10:39:242
22691210511,79sleep222713-21fschecks_count10:29:432
21829210415,68sleep021982-21sshd08:29:580
31840210318,72sleep131877-21grep07:19:441
15795210322,16sleep20-21swapper/210:04:432
15795210322,16sleep20-21swapper/210:04:432
10111210323,37sleep20-21swapper/211:44:272
23095210223,14sleep20-21swapper/208:34:532
23095210223,14sleep20-21swapper/208:34:532
18641210117,67sleep318719-21threads08:20:083
16755210112,72sleep116458-21/usr/sbin/munin08:14:491
16755210112,72sleep116458-21/usr/sbin/munin08:14:491
9668210049,18sleep20-21swapper/211:40:072
3221321008,17sleep20-21swapper/207:19:532
1489729913,67sleep314979-21/usr/sbin/munin12:00:033
525429716,64sleep10-21swapper/111:24:471
2842829716,62sleep0227-21systemd-journal07:09:250
2506829717,38sleep00-21swapper/010:39:590
2496629716,60sleep324916-21latency_hist06:59:263
102429731,51sleep11078-21ls11:09:401
2882829616,58sleep028897-21fschecks_time07:09:410
2824629614,69sleep028282-21fschecks_count10:49:440
77802949,73sleep230594-1kworker/2:2H11:33:572
689929430,43sleep16954-21latency_hist07:44:271
689929430,43sleep16954-21latency_hist07:44:271
22992945,76sleep20-21swapper/209:09:592
164462945,77sleep20-21swapper/208:14:292
164462945,77sleep20-21swapper/208:14:292
138672925,57sleep20-21swapper/209:59:062
1096729213,56sleep311119-21grep07:55:043
1096729213,56sleep311119-21grep07:55:043
2489429135,36sleep124923-21munin-run06:59:241
958229018,51sleep39728-21sed09:39:523
3005829010,22sleep230066-21modprobe10:59:242
2629529014,60sleep1286-21dbus-daemon08:44:541
1650829014,58sleep116628-21/usr/sbin/munin10:04:591
1650829014,58sleep116628-21/usr/sbin/munin10:04:591
3081228615,57sleep130820-21awk10:59:551
291272868,57sleep20-21swapper/208:54:362
291272868,57sleep20-21swapper/208:54:362
71432859,59sleep20-21swapper/207:44:452
71432859,59sleep20-21swapper/207:44:452
413328517,44sleep34227-21fschecks_time07:34:443
2782128518,23sleep10-21swapper/108:49:511
2782128518,23sleep10-21swapper/108:49:511
997328423,33sleep00-21swapper/007:54:240
997328423,33sleep00-21swapper/007:54:240
419128314,41sleep34329-21unixbench_singl11:20:003
1919828328,31sleep10-21swapper/110:19:241
1709828314,55sleep317108-21modprobe12:09:243
133552838,58sleep20-21swapper/208:04:402
999728228,35sleep310006-21modprobe11:44:243
3234628211,56sleep032352-21ntp_states11:04:510
288142827,14sleep20-21swapper/207:09:392
285002826,42sleep228449-21latency_hist07:09:272
211442827,57sleep20-21swapper/208:29:402
2758428134,14sleep20-21swapper/208:49:432
2758428134,14sleep20-21swapper/208:49:432
132928117,46sleep31371-21/usr/sbin/munin07:24:513
10212817,54sleep30-21swapper/311:09:383
2112628021,37sleep321206-21fschecks_time10:24:473
2112628021,37sleep321206-21fschecks_time10:24:473
1041928016,44sleep110392-21/usr/sbin/munin09:44:391
1580327912,51sleep315810-21/usr/sbin/munin10:04:443
1580327912,51sleep315810-21/usr/sbin/munin10:04:443
839127813,47sleep18406-21sh07:49:241
839127813,47sleep18406-21sh07:49:241
3102527819,35sleep331157-21head07:14:573
2790527821,25sleep10-21swapper/107:04:581
2654227819,30sleep30-21swapper/307:04:273
2394227827,32sleep10-21swapper/108:39:241
2394227827,32sleep10-21swapper/108:39:241
192342785,37sleep20-21swapper/212:18:272
192342785,37sleep20-21swapper/212:18:272
2762827715,49sleep327706-21sed08:49:443
2762827715,49sleep327706-21sed08:49:443
2372327727,28sleep023748-21fschecks_count06:54:430
264827514,39sleep32675-21awk09:14:243
1255527519,25sleep30-21swapper/308:00:013
850627432,23sleep00-21swapper/007:49:270
850627432,23sleep00-21swapper/007:49:270
3159627423,24sleep00-21swapper/007:19:260
696227317,36sleep37004-21awk11:29:503
3088627311,37sleep30-21swapper/310:59:573
583827218,34sleep30-21swapper/309:24:583
583827218,34sleep30-21swapper/309:24:583
320827222,24sleep10-21swapper/111:19:261
320827222,24sleep10-21swapper/111:19:261
1810427214,42sleep118113-21/usr/sbin/munin08:19:461
1668927215,34sleep316724-21/usr/sbin/munin08:14:473
1668927215,34sleep316724-21/usr/sbin/munin08:14:473
488127111,45sleep00-21swapper/009:24:270
2944327112,39sleep129126-21/usr/sbin/munin08:55:001
2944327112,39sleep129126-21/usr/sbin/munin08:55:001
1470127113,29sleep10-21swapper/111:59:571
2418427022,33sleep224266-21switchtime06:54:532
152792705,55sleep00-21swapper/012:00:570
418926913,24sleep30-21swapper/309:20:013
1260726915,40sleep112688-21mailstats11:50:071
112072696,13sleep00-21swapper/007:55:210
112072696,13sleep00-21swapper/007:55:210
635026820,20sleep00-21swapper/011:29:260
29502685,28sleep20-21swapper/209:15:042
289432687,14sleep10-21swapper/110:49:581
243372688,11sleep00-21swapper/010:37:160
2176126813,39sleep121887-21sshd12:24:571
2161626816,38sleep121678-21awk08:29:551
1394226816,38sleep314031-21ls09:59:273
69226718,20sleep00-21swapper/007:24:270
102522670,7sleep10-21swapper/109:44:261
373126625,26sleep10-21swapper/109:19:511
315526610,15sleep30-21swapper/311:19:243
315526610,15sleep30-21swapper/311:19:243
1602226618,32sleep016094-21/usr/sbin/munin10:04:490
1602226618,32sleep016094-21/usr/sbin/munin10:04:490
269926519,26sleep10-21swapper/109:14:261
235372656,10sleep00-21swapper/010:30:100
1956726521,27sleep119586-21apt-get12:19:311
195492657,16sleep10-21swapper/108:24:381
1924926523,20sleep30-21swapper/310:19:263
183362658,38sleep10-21swapper/112:10:021
153872655,47sleep00-21swapper/010:03:480
152742658,11sleep10-21swapper/110:02:201
77382648,11sleep00-21swapper/011:33:220
645626413,30sleep00-21swapper/009:29:270
645626413,30sleep00-21swapper/009:29:270
287026415,35sleep12900-21awk07:29:491
2797026423,24sleep328110-21latency_hist10:49:273
2797026423,24sleep328110-21latency_hist10:49:273
2662226420,23sleep00-21swapper/007:04:280
1276326413,31sleep312862-21timerwakeupswit11:50:093
2964026310,40sleep329678-21processes08:55:063
2964026310,40sleep329678-21processes08:55:063
2853826317,23sleep30-21swapper/307:09:283
2744626319,22sleep00-21swapper/008:49:410
2744626319,22sleep00-21swapper/008:49:410
2156726313,35sleep021599-21ntp_states12:24:510
1717826315,23sleep10-21swapper/112:09:261
1157026317,24sleep10-21swapper/107:59:261
1157026317,24sleep10-21swapper/107:59:261
107426312,23sleep00-21swapper/009:09:240
67662625,11sleep00-21swapper/007:43:210
67662625,11sleep00-21swapper/007:43:210
665026212,37sleep16693-21fschecks_count11:29:441
3011026215,25sleep00-21swapper/010:59:260
2721926215,25sleep00-21swapper/010:44:560
2721926215,25sleep00-21swapper/010:44:560
2296026213,23sleep30-21swapper/308:34:503
2296026213,23sleep30-21swapper/308:34:503
208102627,10sleep00-21swapper/012:23:500
2016626212,27sleep00-21swapper/010:19:520
1930626218,28sleep119345-21modprobe12:19:231
1930626218,28sleep119345-21modprobe12:19:231
754826111,25sleep30-21swapper/307:44:583
754826111,25sleep30-21swapper/307:44:583
247826116,19sleep00-21swapper/007:29:360
2471126114,28sleep124516-21apt-get10:39:271
1861526111,36sleep018621-21latency_hist10:14:270
1790426122,19sleep30-21swapper/312:09:563
878926011,13sleep10-21swapper/111:39:371
71612606,13sleep30-21swapper/309:29:593
2492826019,24sleep025080-21grep08:40:000
228126017,26sleep00-21swapper/011:14:240
2237026019,20sleep10-21swapper/110:29:261
2237026019,20sleep10-21swapper/110:29:261
2082526012,22sleep30-21swapper/308:29:253
1619026025,20sleep10-21swapper/108:14:241
484525916,26sleep14948-21users07:34:581
1985225920,21sleep10-21swapper/110:19:451
132472595,10sleep00-21swapper/011:52:530
1224125915,26sleep112265-21ntp_states07:59:541
105652598,12sleep00-21swapper/011:44:490
871525815,22sleep00-21swapper/011:39:320
3273025812,21sleep10-21swapper/109:04:531
3062225814,17sleep00-21swapper/010:59:500
2091625817,20sleep30-21swapper/312:24:263
2090125814,19sleep00-21swapper/008:29:260
1381525816,24sleep30-21swapper/308:04:533
577825711,28sleep35893-21users11:24:573
2593725715,17sleep00-21swapper/008:44:450
139522578,15sleep00-21swapper/011:58:460
1393025716,19sleep10-21swapper/108:04:551
93025612,23sleep10-21swapper/107:24:401
694125617,20sleep10-21swapper/109:29:531
3232025613,23sleep332344-21ntp_states11:04:503
2704325620,21sleep327081-21modprobe08:49:243
2549025617,17sleep10-21swapper/106:59:491
1993025618,23sleep319937-21ntp_states10:19:473
926125514,26sleep09298-21/usr/sbin/munin09:39:450
650825519,15sleep10-21swapper/109:29:291
2901025516,18sleep30-21swapper/307:09:443
2496425513,24sleep124516-21apt-get10:39:321
2423925513,25sleep324232-21apt-get08:39:433
1511925514,25sleep30-21swapper/308:09:463
84592545,37sleep00-21swapper/009:37:470
475325416,18sleep00-21swapper/011:24:260
293925412,27sleep32955-21ntp_states07:29:513
138882549,23sleep30-21swapper/311:57:573
1385625418,17sleep00-21swapper/008:04:540
1036725415,23sleep310376-21diskstats11:44:443
902325312,23sleep39091-21memory07:49:593
902325312,23sleep39091-21memory07:49:593
627625312,28sleep1370-21snmpd09:28:071
3226125312,24sleep332256-21apt-get09:04:393
3029025312,24sleep130285-21apt-get07:14:391
300222536,15sleep30-21swapper/310:58:563
2939125314,21sleep029442-21irqstats08:54:590
2939125314,21sleep029442-21irqstats08:54:590
2823825313,24sleep328137-21/usr/sbin/munin10:49:433
2235825313,23sleep022367-21munin-run10:29:250
2235825313,23sleep022367-21munin-run10:29:250
1493625312,24sleep014931-21apt-get08:09:390
876425212,23sleep18791-21df07:49:491
876425212,23sleep18791-21df07:49:491
3072325214,22sleep030718-21apt-get08:59:430
148125215,23sleep01537-21tune2fs09:09:450
1243425215,23sleep012445-21ntp_states09:49:510
333125115,20sleep00-21swapper/009:19:260
2911025117,19sleep10-21swapper/107:09:461
2656325111,23sleep126582-21apt-get10:44:361
2656325111,23sleep126582-21apt-get10:44:361
2391525113,23sleep124057-21cut10:34:271
1198225113,22sleep111985-21apt-get09:49:361
794525012,22sleep07973-21munin-run09:34:240
1778925016,18sleep30-21swapper/310:09:533
336424911,16sleep10-21swapper/109:19:271
3225824912,22sleep332290-21ntp_states07:19:543
1935724918,15sleep30-21swapper/312:19:253
1935724918,15sleep30-21swapper/312:19:253
1370024914,17sleep00-21swapper/009:56:580
345124711,20sleep13446-21apt-get11:19:421
136232477,12sleep10-21swapper/109:55:581
2470624514,14sleep10-21swapper/108:39:571
3119124313,15sleep30-21swapper/308:59:583
1414724210,16sleep314166-21apt-get09:59:313
32842419,17sleep30-21swapper/309:19:253
2220424114,17sleep30-21swapper/308:31:553
189982399,19sleep0370-21snmpd10:16:490
84542388,14sleep00-21swapper/011:39:240
80782376,12sleep10-21swapper/111:34:321
86352357,14sleep10-21swapper/109:39:261
83412357,11sleep30-21swapper/311:37:583
299072318,12sleep10-21swapper/110:57:251
190652317,12sleep00-21swapper/012:16:150
190652317,12sleep00-21swapper/012:16:150
84062307,14sleep30-21swapper/309:37:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional