You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 03:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Apr 02, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28495210099,978sleep028011-1kworker/0:2H00:25:190
694129267,904sleep36989-21ntpq23:05:563
1542526986,680sleep315436-21ntpq19:45:253
1542526986,680sleep315436-21ntpq19:45:253
23780246511,436sleep123814-21unin-run20:19:591
531924456,428sleep3370-21snmpd19:14:123
4575240013,372sleep04586-21ntpq19:10:260
2596523965,381sleep37745-21kworker/3:000:17:183
2449233418,290sleep32481-21modprobe20:54:583
2449233418,290sleep32481-21modprobe20:54:583
2116523240,65sleep10-21swapper/100:00:151
9237225118,212sleep09297-21/usr/sbin/munin19:25:280
24839224850,179sleep30-21swapper/300:10:523
31198224659,105sleep331328-21unixbench_multi20:40:373
8847224461,164sleep38909-21fschecks_time19:25:183
20618222720,189sleep3312-21rs:main3
13671222316,189sleep313713-21latency23:30:183
106392221128,62sleep30-21swapper/319:30:233
15849221539,91sleep30-21swapper/323:40:153
9451220828,118sleep39602-21sed23:15:003
106622078,85sleep30-21swapper/319:00:273
106622078,85sleep30-21swapper/319:00:273
10819220534,108sleep310965-21grep21:20:343
5397220016,170sleep15420-21/usr/sbin/munin19:14:591
30732219811,123sleep330763-21latency_hist22:34:593
22305219410,171sleep322391-21grep22:04:593
799121930,23sleep3383-21cyclictest19:20:333
3030219211,107sleep33102-21mailstats19:05:283
523821910,6sleep00-21swapper/021:00:360
23178217976,20sleep30-21swapper/320:15:013
28714216956,100sleep328756-21latency_hist20:34:593
7877216883,45sleep30-21swapper/321:10:083
7877216883,45sleep30-21swapper/321:10:083
18479216717,134sleep218597-21latency_hist23:49:592
12053216017,121sleep20-21swapper/219:35:192
6745215719,118sleep00-21swapper/021:05:290
6745215719,118sleep00-21swapper/021:05:290
24330215411,24sleep20-21swapper/222:10:192
23827214910,123sleep323947-21latency_hist00:09:593
1648214916,119sleep01680-21proc_pri20:50:270
1648214916,119sleep01680-21proc_pri20:50:270
4291214846,89sleep34325-21date20:59:593
4291214846,89sleep34325-21date20:59:593
15562214619,108sleep215589-21fschecks_time21:40:152
15352214518,19sleep00-21swapper/023:40:000
6517214265,14sleep30-21swapper/321:05:243
6517214265,14sleep30-21swapper/321:05:243
7189213816,106sleep17304-21timerwakeupswit23:06:021
2823621387,119sleep328277-21ntpq22:25:223
5193213714,109sleep25231-21processes23:00:222
16277213558,52sleep30-21swapper/321:40:293
25598213439,15sleep30-21swapper/320:25:143
12202213457,66sleep312285-21tr21:25:353
12202213457,66sleep312285-21tr21:25:353
1414321337,114sleep314289-21unixbench_singl19:40:373
1414321337,114sleep314289-21unixbench_singl19:40:373
27564213158,37sleep30-21swapper/300:20:583
2162921297,53sleep30-21swapper/320:10:003
1741521280,53sleep20-21swapper/219:50:322
9800212342,66sleep39803-21cron21:19:573
9800212342,66sleep39803-21cron21:19:573
2674212319,18sleep30-21swapper/322:50:303
2674212319,18sleep30-21swapper/322:50:303
32060212234,19sleep30-21swapper/322:40:203
2140212212,94sleep22191-21apt-key22:50:002
168121210,9sleep10-21swapper/120:50:271
168121210,9sleep10-21swapper/120:50:271
28586212017,87sleep328647-21fschecks_time00:25:263
22164211943,15sleep30-21swapper/300:04:583
27618211812,89sleep127749-21cat22:25:001
723211629,11sleep30-21swapper/320:48:083
21022211423,71sleep220984-21apt-get22:00:302
26883211229,22sleep326968-21cat22:20:263
19471211210,86sleep319617-21unixbench_singl23:50:373
4541210910,34sleep34537-21apt-get21:00:153
1520621099,41sleep315266-21cat21:40:003
6976210813,80sleep27003-21latency_hist19:20:002
3887210822,65sleep04032-21latency_hist19:10:010
31380210514,17sleep30-21swapper/322:40:003
2470921059,81sleep324836-21timerwakeupswit20:20:303
2470921059,81sleep324836-21timerwakeupswit20:20:303
18389210515,70sleep018456-21sendmail-msp23:49:570
1942210425,25sleep3370-21snmpd22:49:293
29112210321,65sleep20-21swapper/222:29:572
20989210319,66sleep020481-21apt-get22:00:080
19346210116,24sleep30-21swapper/320:00:273
1752721016,18sleep30-21swapper/319:51:193
4986210014,73sleep05006-21cut23:00:160
13489210016,68sleep013545-21fschecks_count23:30:150
302672999,32sleep330405-21date20:40:003
2426729817,62sleep224269-21apt-get00:10:102
3068829713,68sleep230731-21systemd-run22:34:582
645529637,33sleep10-21swapper/121:05:221
645529637,33sleep10-21swapper/121:05:221
261529618,62sleep00-21swapper/022:50:190
261529618,62sleep00-21swapper/022:50:190
1338829614,65sleep213469-21fschecks_time21:30:222
1206329624,53sleep212086-21/usr/sbin/munin21:25:262
1206329624,53sleep212086-21/usr/sbin/munin21:25:262
832629510,17sleep10-21swapper/123:10:221
1910829418,60sleep119147-21ntpq20:00:211
1780529423,13sleep30-21swapper/323:45:323
93529318,53sleep1951-21latency_hist20:49:591
198929318,41sleep12200-21fschecks_time19:05:021
1579229316,62sleep215847-21interrupts23:40:142
3272829232,26sleep20-21swapper/218:59:582
2381528920,22sleep20-21swapper/220:20:002
208792898,66sleep320933-21sed22:00:033
208792898,66sleep320933-21sed22:00:033
60242888,20sleep36086-21cmp23:04:593
1180828827,33sleep10-21swapper/123:25:121
295028719,47sleep22463-21apt-get20:55:092
295028719,47sleep22463-21apt-get20:55:092
1506228720,39sleep015103-21fschecks_count19:45:180
1506228720,39sleep015103-21fschecks_count19:45:180
964928621,12sleep30-21swapper/323:15:293
1883828610,58sleep30-21swapper/321:50:223
791128520,48sleep27965-21ls21:10:162
791128520,48sleep27965-21ls21:10:162
2422728518,19sleep10-21swapper/122:10:181
1822928524,30sleep30-21swapper/321:49:453
1543728511,36sleep10-21swapper/121:40:061
852128427,35sleep18645-21latency_hist19:25:001
577828440,23sleep10-21swapper/119:15:191
130232846,67sleep30-21swapper/323:29:093
235662839,15sleep30-21swapper/322:06:573
192628340,20sleep00-21swapper/019:05:010
192628340,20sleep00-21swapper/019:05:010
1176428316,48sleep111665-21/usr/sbin/munin21:25:161
1176428316,48sleep111665-21/usr/sbin/munin21:25:161
664828220,10sleep30-21swapper/319:16:223
448928216,54sleep04556-21latency_hist22:59:590
408528220,10sleep30-21swapper/322:55:223
1927328219,40sleep218713-21/usr/sbin/munin20:00:262
1515428118,46sleep115181-21modprobe21:39:581
2599828015,33sleep00-21swapper/020:25:230
1808828024,44sleep30-21swapper/319:55:183
1094228012,52sleep211065-21cat23:20:242
2067127921,29sleep10-21swapper/100:00:001
1991627915,49sleep119988-21latency_hist21:55:001
153482796,61sleep315350-21munin-run23:39:593
1504627918,31sleep20-21swapper/219:45:162
1504627918,31sleep20-21swapper/219:45:162
136582796,12sleep30-21swapper/321:30:273
95527824,20sleep20-21swapper/220:50:002
2371927820,37sleep123739-21apt-get00:09:571
1380827820,23sleep00-21swapper/019:40:300
1380827820,23sleep00-21swapper/019:40:300
801327725,37sleep18028-21sort21:10:311
801327725,37sleep18028-21sort21:10:311
1316427719,26sleep00-21swapper/023:29:590
131372779,14sleep10-21swapper/119:39:581
121132777,33sleep30-21swapper/319:35:203
2741827618,38sleep227561-21grep00:20:562
256662765,59sleep30-21swapper/322:15:213
2538327623,34sleep025396-21modprobe22:14:580
2415127618,39sleep024260-21grep20:20:190
2415127618,39sleep024260-21grep20:20:190
2195627620,31sleep221960-21diskstats20:10:192
742727522,30sleep27444-21munin-run21:09:592
742727522,30sleep27444-21munin-run21:09:592
2294627512,44sleep222988-21/usr/sbin/munin22:05:232
2163927518,41sleep121727-21cat22:01:181
102512757,52sleep30-21swapper/323:20:053
2943727419,31sleep029507-21/usr/sbin/munin22:30:160
243742747,48sleep30-21swapper/322:10:203
236722745,12sleep30-21swapper/320:18:483
30527320,33sleep3336-21munin-run18:59:593
2982127313,44sleep129855-21cat00:30:001
2728827318,27sleep20-21swapper/220:30:022
228322730,7sleep00-21swapper/022:05:200
2537527218,22sleep10-21swapper/120:25:001
2537527218,22sleep10-21swapper/120:25:001
134332727,51sleep10-21swapper/119:40:181
134332727,51sleep10-21swapper/119:40:181
211122707,48sleep020879-21/usr/sbin/munin00:00:150
163932705,54sleep00-21swapper/021:40:580
807626919,29sleep00-21swapper/021:10:340
807626919,29sleep00-21swapper/021:10:340
648826912,42sleep25997-21apt-get23:05:102
58592695,18sleep10-21swapper/123:03:201
564626913,24sleep00-21swapper/019:15:100
3176226922,24sleep00-21swapper/020:44:580
2299626912,26sleep20-21swapper/200:05:302
1354026917,33sleep113634-21/usr/sbin/munin21:30:251
3202026825,19sleep00-21swapper/018:55:280
234012685,52sleep00-21swapper/000:05:560
178642689,43sleep118002-21latency_hist19:55:001
1001626824,29sleep210042-21ls19:29:582
2376226716,33sleep023779-21ls20:19:580
1882426617,22sleep00-21swapper/020:00:160
1753626614,26sleep10-21swapper/123:45:261
143526615,27sleep11582-21unixbench_singl22:45:311
133726618,29sleep01434-21/usr/sbin/munin22:45:300
3209026525,20sleep10-21swapper/118:55:291
290622655,12sleep00-21swapper/022:29:190
267252655,11sleep00-21swapper/000:20:300
1746026513,26sleep00-21swapper/021:45:300
1495026519,26sleep00-21swapper/021:37:190
144212656,15sleep00-21swapper/021:33:430
1113126513,35sleep011240-21users19:30:320
2567626413,34sleep125683-21/usr/sbin/munin20:25:161
25012646,43sleep20-21swapper/222:50:042
25012646,43sleep20-21swapper/222:50:042
203182646,15sleep00-21swapper/021:58:020
203182646,15sleep00-21swapper/021:58:020
2746526311,32sleep00-21swapper/020:30:160
2617026317,30sleep026186-21apt-get00:19:570
2458226317,27sleep00-21swapper/000:10:450
1393526317,23sleep20-21swapper/223:30:262
97462625,46sleep20-21swapper/221:19:162
97462625,46sleep20-21swapper/221:19:162
542126212,36sleep25446-21latency_hist19:15:002
41722626,48sleep20-21swapper/222:56:302
385326222,25sleep13930-21sed22:55:001
385326222,25sleep13930-21sed22:55:001
3181926216,23sleep20-21swapper/222:40:152
2985726210,35sleep230005-21latency_hist00:30:012
2813126215,32sleep128221-21load22:25:201
2106326213,31sleep021191-21/usr/sbin/munin20:05:330
2002526215,33sleep120043-21ls23:54:591
1934426216,26sleep20-21swapper/223:50:362
3205626122,18sleep00-21swapper/020:45:140
2424926115,24sleep10-21swapper/100:10:091
1085526117,23sleep10-21swapper/123:20:231
818626016,29sleep28192-21diskstats23:10:192
3181326024,17sleep20-21swapper/220:45:002
2607126016,18sleep10-21swapper/122:19:591
2155126017,28sleep121595-21awk20:09:581
2056626010,35sleep020567-21sshd23:59:180
2098925917,20sleep20-21swapper/200:00:132
1604625911,35sleep016050-21ntp_states23:40:210
120425913,31sleep21286-21find19:00:302
120425913,31sleep21286-21find19:00:302
1084225918,18sleep10-21swapper/119:30:291
1008025916,26sleep110077-21apt-get21:20:121
1008025916,26sleep110077-21apt-get21:20:121
360025816,20sleep00-21swapper/020:55:400
360025816,20sleep00-21swapper/020:55:400
2749725817,26sleep227507-21/usr/sbin/munin20:30:182
264125818,24sleep22750-21fschecks_time19:05:192
1903025815,23sleep00-21swapper/021:50:270
1377325814,29sleep113807-21/usr/sbin/munin23:30:211
81925711,33sleep1876-21awk19:00:211
81925711,33sleep1876-21awk19:00:211
3229025712,30sleep032414-21sed22:40:250
260992575,10sleep20-21swapper/200:19:032
103722575,12sleep20-21swapper/221:20:252
2968225616,22sleep20-21swapper/220:35:312
2760925622,20sleep10-21swapper/120:30:291
2612525616,20sleep20-21swapper/222:20:002
1052725613,23sleep110553-21ntp_states21:20:281
981125511,29sleep19812-21sshd19:27:181
656225513,25sleep06507-21apt-get23:05:450
2204925518,17sleep00-21swapper/020:10:210
1723925515,24sleep217268-21irqstats21:45:242
1370725513,21sleep20-21swapper/219:40:282
1370725513,21sleep20-21swapper/219:40:282
118325513,24sleep21179-21apt-get20:50:132
118325513,24sleep21179-21apt-get20:50:132
1057925513,26sleep010635-21sendmail_mailqu21:20:310
721825414,23sleep27212-21apt-get19:20:172
3241525413,22sleep132515-21users22:40:261
2462825418,14sleep20-21swapper/220:20:292
2462825418,14sleep20-21swapper/220:20:292
1962025417,24sleep019634-21/usr/sbin/munin23:50:380
1544525412,29sleep115490-21processes19:45:271
1544525412,29sleep115490-21processes19:45:271
1201825413,25sleep212076-21if_err_eth023:25:212
2895925311,25sleep128953-21apt-get20:35:141
2760725314,22sleep027617-21munin-run22:24:590
1831825317,18sleep10-21swapper/121:49:591
847625216,18sleep00-21swapper/023:10:250
3136725212,24sleep231379-21munin-run22:39:592
3059025211,16sleep10-21swapper/120:40:221
2669125214,22sleep126186-21apt-get00:20:111
2037425213,23sleep220375-21diskstats20:05:192
1171825214,22sleep011763-21df21:25:150
1171825214,22sleep011763-21df21:25:150
1033525213,22sleep210329-21apt-get19:30:142
1004525217,15sleep20-21swapper/223:20:002
2965525112,22sleep129704-21irqstats22:30:211
1215425114,17sleep00-21swapper/019:35:210
45352507,19sleep10-21swapper/121:00:131
329825014,21sleep13365-21memory20:55:331
329825014,21sleep13365-21memory20:55:331
2253125011,22sleep122539-21apt-get22:05:091
583424914,20sleep20-21swapper/221:04:592
1006024913,21sleep010077-21apt-get21:20:110
1006024913,21sleep010077-21apt-get21:20:110
2920324812,21sleep029232-21irqstats20:35:210
2620824814,18sleep10-21swapper/100:19:581
1700624810,15sleep00-21swapper/019:50:270
1684424816,18sleep10-21swapper/123:44:581
1067724813,19sleep00-21swapper/023:20:180
851024710,21sleep08520-21munin-run19:24:590
185662477,15sleep20-21swapper/221:50:132
1852024714,15sleep20-21swapper/220:00:022
1529924513,19sleep215346-21systemctl23:39:582
151022455,27sleep20-21swapper/221:39:192
999424410,20sleep010019-21munin-run23:19:580
2602024212,12sleep00-21swapper/022:19:580
2194924210,15sleep10-21swapper/120:10:181
312112415,26sleep10-21swapper/122:38:281
204672418,18sleep220489-21munin-run21:59:582
204672418,18sleep220489-21munin-run21:59:582
185032409,14sleep10-21swapper/120:00:011
206862387,13sleep10-21swapper/122:00:011
206862387,13sleep10-21swapper/122:00:011
100352389,15sleep10-21swapper/123:19:591
180682377,12sleep00-21swapper/019:55:030
2027223510,15sleep20-21swapper/223:55:282
311192346,12sleep00-21swapper/022:37:180
150272316,13sleep10-21swapper/123:36:261
43102295,13sleep10-21swapper/122:58:191
258812287,12sleep20-21swapper/222:18:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional