You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Thu Apr 03, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4716234287,3406sleep04678-21apt-get22:45:280
154022331410,3255sleep315404-21apt-get21:25:063
154022331410,3255sleep315404-21apt-get21:25:063
326727668,746sleep33314-21ntpq22:40:153
10342265214,620sleep110348-21/usr/sbin/munin21:05:221
12919260113,571sleep112994-21/usr/sbin/munin19:30:271
3068225858,560sleep230686-21ntpq00:20:172
2240424158,393sleep0194-21jbd2/mmcblk2p2-23:50:000
26957233740,242sleep327085-21timerwakeupswit20:15:263
30540228286,180sleep330618-21sendmail_mailtr22:20:213
30540228286,180sleep330618-21sendmail_mailtr22:20:213
7134227013,203sleep37154-21sh19:14:553
2206224374,86sleep30-21swapper/320:40:213
2206224374,86sleep30-21swapper/320:40:213
1102322237,206sleep3370-21snmpd21:08:193
12771221433,118sleep312898-21latency_hist21:14:573
3146222129,189sleep016934-1kworker/0:1H20:30:090
18009220842,141sleep318141-21unixbench_singl21:30:353
27635220522,159sleep127789-21cat22:09:571
9638218871,43sleep30-21swapper/321:04:563
9638218871,43sleep30-21swapper/321:04:563
2962621839,119sleep329637-21apt-get20:25:073
29939217944,45sleep3227-21systemd-journal00:19:543
2104217528,78sleep32281-21latency_hist18:59:573
2636721678,77sleep30-21swapper/322:05:153
25413216212,130sleep325546-21sort20:10:263
3167521598,138sleep331711-21expr22:25:133
28341215936,111sleep328401-21find20:20:273
13646215710,79sleep30-21swapper/321:20:093
7399215216,76sleep30-21swapper/319:15:073
2576215015,120sleep22581-21ntpq20:40:282
2576215015,120sleep22581-21ntpq20:40:282
21703214162,58sleep30-21swapper/321:45:253
21703214162,58sleep30-21swapper/321:45:253
2018121408,77sleep30-21swapper/319:55:163
6757213737,68sleep30-21swapper/320:55:073
31531213210,109sleep031564-21cat00:24:550
4998212918,26sleep30-21swapper/319:05:303
27196212744,29sleep30-21swapper/300:06:543
2471421268,103sleep324785-21cut00:00:123
17470212569,38sleep317504-21proc_pri19:45:553
29194212439,50sleep229406-21apt-key00:14:562
2747321206,104sleep00-21swapper/020:18:420
982211739,29sleep30-21swapper/322:30:183
982211739,29sleep30-21swapper/322:30:183
22893211737,65sleep122906-21idleruntime-cro20:04:541
1882521168,96sleep029299-21sshd23:37:240
16890211522,69sleep016975-21latency_hist23:29:570
16434211545,51sleep30-21swapper/319:44:543
29617211316,79sleep229127-21apt-get20:25:062
24207211241,14sleep30-21swapper/320:06:543
24207211241,14sleep30-21swapper/320:06:543
1117211212,81sleep21165-21ntp_states22:30:212
1117211212,81sleep21165-21ntp_states22:30:212
8806211114,83sleep18814-21awk21:00:261
8806211114,83sleep18814-21awk21:00:261
10275211111,86sleep310355-21latency_hist19:24:563
10275211111,86sleep310355-21latency_hist19:24:563
647221109,90sleep3370-21snmpd20:54:383
17215210919,72sleep216711-21apt-get21:30:062
5912210818,72sleep26063-21unixbench_multi20:50:252
25624210710,47sleep325639-21vmstat22:00:223
13079210635,18sleep313104-21munin-run23:14:543
5320210513,78sleep25325-21sh19:06:552
14969210516,68sleep30-21swapper/319:39:583
14969210516,68sleep30-21swapper/319:39:583
29556210411,77sleep229081-21apt-get00:15:002
2440921035,87sleep00-21swapper/021:59:260
1872121038,24sleep30-21swapper/319:50:153
30215210031,13sleep30-21swapper/300:20:023
13715210024,52sleep20-21swapper/223:19:542
930229921,61sleep29433-21cat23:00:202
1222829921,62sleep012273-21/usr/sbin/munin23:10:190
810129814,65sleep08202-21sed20:59:560
146972988,77sleep314838-21sed23:20:323
626629731,14sleep30-21swapper/322:50:123
794529615,66sleep17541-21apt-get22:55:021
64429629,13sleep30-21swapper/300:29:553
2847729616,65sleep128590-21timerandwakeup20:20:291
196802968,15sleep30-21swapper/321:40:123
1908429615,67sleep019152-21/usr/sbin/munin19:50:240
1413229612,68sleep014214-21mailstats21:20:230
1209029628,16sleep30-21swapper/319:30:093
1065429531,14sleep30-21swapper/323:05:183
236902938,38sleep3370-21snmpd21:53:243
1960329312,64sleep219604-21sshd19:52:202
145529319,53sleep30-21swapper/320:35:293
1022529316,60sleep110245-21idleruntime-cro19:24:541
1022529316,60sleep110245-21idleruntime-cro19:24:541
97482917,71sleep30-21swapper/323:02:193
415729129,41sleep24168-21modprobe22:44:542
119329012,61sleep01235-21ntp_states18:55:260
2889628958,16sleep30-21swapper/300:13:243
1334528915,55sleep013360-21apt19:34:540
491128815,20sleep34924-21modprobe20:49:543
491128815,20sleep34924-21modprobe20:49:543
1377028814,57sleep013856-21latency_hist23:19:560
46828723,37sleep10-21swapper/122:29:551
2161728722,38sleep021636-21apt-get20:00:090
292828637,34sleep33012-21cat19:00:263
202828629,35sleep12080-21systemd-run18:59:551
1855328616,50sleep018638-21cat21:34:560
1581828610,54sleep30-21swapper/323:25:163
55828522,39sleep1553-21apt-get20:35:101
2675028515,57sleep026793-21processes22:05:240
220992857,68sleep3370-21snmpd23:49:463
803928410,61sleep28151-21cat19:15:262
315932849,58sleep031660-21df22:25:100
1372828411,16sleep30-21swapper/323:19:553
194062837,16sleep30-21swapper/323:40:103
158442820,6sleep20-21swapper/219:40:282
281872800,7sleep20-21swapper/220:20:212
1484128011,51sleep014919-21/usr/sbin/munin23:20:330
2499827921,30sleep10-21swapper/122:00:121
226562798,17sleep30-21swapper/323:50:193
217992796,63sleep00-21swapper/023:45:520
1070527917,31sleep20-21swapper/223:05:212
2999127813,15sleep10-21swapper/100:19:561
278642788,15sleep30-21swapper/322:10:113
365827716,47sleep03695-21grep20:45:110
365827716,47sleep03695-21grep20:45:110
35427710,17sleep00-21swapper/020:34:570
2665127722,17sleep20-21swapper/200:05:242
26112778,21sleep32637-21munin-run22:39:543
2385927719,34sleep223984-21grep21:54:562
61552768,15sleep00-21swapper/019:10:200
241152765,60sleep30-21swapper/321:55:383
2209227611,53sleep1370-21snmpd21:48:091
2209227611,53sleep1370-21snmpd21:48:091
1845727611,46sleep30-21swapper/323:34:573
674927518,39sleep16812-21/usr/sbin/munin22:50:261
2626227523,32sleep126330-21fschecks_time00:05:141
2481527526,29sleep20-21swapper/222:00:092
227082755,59sleep30-21swapper/320:02:303
923327416,41sleep19301-21switchtime23:00:191
2943327451,13sleep30-21swapper/322:15:443
2943327451,13sleep30-21swapper/322:15:443
2853127327,23sleep10-21swapper/122:10:301
1939727318,21sleep10-21swapper/121:40:021
1271927323,33sleep012725-21modprobe21:14:540
3218527216,37sleep132328-21/usr/sbin/munin20:30:381
1384427211,16sleep20-21swapper/219:35:162
1384427211,16sleep20-21swapper/219:35:162
1052027220,33sleep210514-21apt-get19:25:112
2823327122,24sleep20-21swapper/222:10:222
2449627123,27sleep10-21swapper/121:59:561
1862127115,37sleep118625-21/usr/sbin/munin19:50:131
1773827118,30sleep017850-21sort23:30:260
172152717,43sleep10-21swapper/123:30:141
1540127117,37sleep214918-21apt-get21:25:052
1540127117,37sleep214918-21apt-get21:25:052
1191127113,45sleep211947-21fschecks_count23:10:132
840427014,37sleep08411-21modprobe22:59:540
314362709,46sleep30-21swapper/320:30:083
2752127019,35sleep20-21swapper/220:19:212
1680627026,25sleep10-21swapper/123:29:541
46912698,13sleep30-21swapper/322:45:113
366026912,40sleep23773-21sed19:04:572
2525826917,30sleep20-21swapper/200:00:232
243926917,23sleep00-21swapper/019:00:140
1107826925,22sleep10-21swapper/119:25:261
937826811,25sleep20-21swapper/219:20:232
937826811,25sleep20-21swapper/219:20:232
653426814,35sleep06515-21latency_hist20:54:550
2844326819,22sleep00-21swapper/022:10:290
206672689,14sleep20-21swapper/223:44:572
1100626816,34sleep011075-21mailstats19:25:250
2323426717,32sleep123367-21unixbench_multi21:50:311
1988526713,39sleep119980-21/usr/sbin/munin23:40:201
1727526719,26sleep20-21swapper/223:30:152
1653526725,26sleep116455-21latency_hist19:44:561
103492679,14sleep20-21swapper/221:05:232
80842665,11sleep30-21swapper/322:55:453
65926612,38sleep0779-21latency_hist00:29:560
2058226626,19sleep00-21swapper/019:55:250
1340826621,21sleep20-21swapper/221:19:562
790326513,22sleep00-21swapper/019:15:240
2680026513,35sleep126884-21grep20:15:241
233572657,20sleep10-21swapper/123:50:331
157026513,35sleep11617-21users22:30:281
157026513,35sleep11617-21users22:30:281
1452026518,29sleep114604-21mailstats23:20:301
2821126412,35sleep028217-21ntp_states00:10:220
240892645,11sleep30-21swapper/323:56:413
2282126411,39sleep022852-21latency21:50:220
2258826417,31sleep222654-21idleruntime23:50:182
204642647,44sleep20-21swapper/223:43:362
353826317,23sleep10-21swapper/122:40:211
2644526317,21sleep00-21swapper/000:05:170
2219826322,22sleep10-21swapper/123:49:561
2033126322,19sleep10-21swapper/119:55:181
189282635,10sleep30-21swapper/321:37:133
1456026314,30sleep113621-21/usr/sbin/munin21:20:281
140042637,15sleep10-21swapper/119:35:191
140042637,15sleep10-21swapper/119:35:191
130052635,10sleep30-21swapper/321:15:213
117442636,11sleep30-21swapper/319:29:323
541226221,22sleep20-21swapper/222:45:592
312426214,22sleep00-21swapper/022:40:120
3099826215,21sleep00-21swapper/020:29:560
25562629,17sleep00-21swapper/022:39:110
2513126216,24sleep024743-21/usr/sbin/munin20:10:180
1920826220,19sleep00-21swapper/021:39:570
187782625,11sleep30-21swapper/323:36:463
873326113,23sleep10-21swapper/119:19:571
499226116,19sleep10-21swapper/120:49:561
499226116,19sleep10-21swapper/120:49:561
451326117,24sleep10-21swapper/119:05:201
2821826110,37sleep128225-21awk00:10:231
2533926116,28sleep20-21swapper/220:10:252
2134026118,24sleep00-21swapper/021:45:180
2134026118,24sleep00-21swapper/021:45:180
1913726110,35sleep219179-21/usr/sbin/munin21:39:552
1207626113,17sleep20-21swapper/221:10:282
959326013,26sleep20-21swapper/221:04:552
959326013,26sleep20-21swapper/221:04:552
556226013,18sleep00-21swapper/019:09:550
3012626017,27sleep130131-21/usr/sbin/munin22:20:101
3012626017,27sleep130131-21/usr/sbin/munin22:20:101
294622606,42sleep20-21swapper/222:16:082
294622606,42sleep20-21swapper/222:16:082
1740526017,25sleep00-21swapper/019:45:520
1693326018,20sleep10-21swapper/119:45:041
1593226023,22sleep00-21swapper/021:25:360
1593226023,22sleep00-21swapper/021:25:360
80025922,20sleep00-21swapper/022:30:150
80025922,20sleep00-21swapper/022:30:150
3135125917,21sleep20-21swapper/222:24:562
3135125917,21sleep20-21swapper/222:24:562
3049225912,21sleep10-21swapper/100:20:121
2386725918,20sleep10-21swapper/120:05:271
2386725918,20sleep10-21swapper/120:05:271
235625920,20sleep10-21swapper/119:00:111
208325920,24sleep22065-21/usr/sbin/munin18:59:562
1962825917,27sleep219679-21if_eth021:40:112
1478625912,33sleep0370-21snmpd19:38:330
1478625912,33sleep0370-21snmpd19:38:330
132552597,40sleep10-21swapper/121:18:341
758225815,22sleep00-21swapper/022:54:570
57825814,20sleep20-21swapper/200:29:542
2654325820,18sleep20-21swapper/222:05:182
2525325819,22sleep10-21swapper/120:10:241
2355225813,30sleep223581-21nfs4_client20:05:202
2355225813,30sleep223581-21nfs4_client20:05:202
1768325810,12sleep20-21swapper/219:45:582
3062125710,22sleep00-21swapper/022:20:220
3062125710,22sleep00-21swapper/022:20:220
2915325720,20sleep00-21swapper/020:24:560
2554125713,19sleep10-21swapper/100:00:261
1907125720,16sleep20-21swapper/223:39:562
190325716,19sleep20-21swapper/220:40:002
184122576,39sleep10-21swapper/121:33:441
1600025720,18sleep10-21swapper/121:25:371
1600025720,18sleep10-21swapper/121:25:371
1503425713,29sleep215176-21latency_hist21:24:572
1499825710,36sleep20-21swapper/223:21:272
1209225718,18sleep10-21swapper/123:10:161
1150925719,19sleep00-21swapper/023:09:550
3215425615,26sleep132198-21ntpq00:25:211
2378125613,28sleep023866-21awk20:05:260
2378125613,28sleep023866-21awk20:05:260
879925514,17sleep00-21swapper/021:00:250
879925514,17sleep00-21swapper/021:00:250
654725517,16sleep10-21swapper/120:54:561
3219425514,18sleep20-21swapper/222:25:242
3112625514,17sleep10-21swapper/120:29:581
2439125517,22sleep024472-21latency_hist23:59:570
1937425516,23sleep00-21swapper/023:40:060
1647325515,26sleep016504-21munin-run19:44:550
1555025514,24sleep215547-21apt-get23:25:072
1200425516,20sleep10-21swapper/121:10:271
988225312,24sleep09876-21apt-get21:05:090
685625312,23sleep10-21swapper/120:55:131
673125313,24sleep16784-21users19:10:291
365525313,24sleep23651-21apt-get20:45:082
365525313,24sleep23651-21apt-get20:45:082
1208925312,25sleep212086-21apt-get19:30:082
738925213,17sleep20-21swapper/220:55:262
626525213,23sleep26261-21apt-get22:50:112
2472825215,21sleep024724-21apt-get22:00:030
2124025215,17sleep20-21swapper/221:45:162
2124025215,17sleep20-21swapper/221:45:162
113612527,34sleep10-21swapper/123:08:021
467525111,25sleep14173-21apt-get22:45:051
3133925113,23sleep231421-21store20:29:592
266425012,15sleep10-21swapper/122:39:561
2463125013,24sleep024699-21df00:00:090
186372506,34sleep10-21swapper/123:34:591
218324911,22sleep02196-21df_inode20:40:180
218324911,22sleep02196-21df_inode20:40:180
205512497,33sleep00-21swapper/021:43:120
2017624910,24sleep220180-21diskstats19:55:152
869024710,19sleep00-21swapper/023:00:060
718624712,19sleep20-21swapper/219:14:582
2171724711,21sleep221724-21diskstats20:00:152
2432624612,21sleep224327-21sshd21:58:232
23452469,21sleep12391-21irqstats20:40:231
23452469,21sleep12391-21irqstats20:40:231
136412466,31sleep10-21swapper/123:18:561
2433924411,17sleep20-21swapper/223:59:542
905924215,13sleep00-21swapper/019:20:170
905924215,13sleep00-21swapper/019:20:170
845624111,15sleep20-21swapper/222:59:562
295592409,15sleep00-21swapper/000:15:020
188582399,16sleep10-21swapper/121:36:181
2426023711,14sleep10-21swapper/123:58:531
297142355,13sleep00-21swapper/022:19:250
297142355,13sleep00-21swapper/022:19:250
2962723510,12sleep10-21swapper/122:18:171
2962723510,12sleep10-21swapper/122:18:171
24952358,11sleep20-21swapper/222:38:242
1333723510,14sleep00-21swapper/021:19:380
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional