You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-15 - 07:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Jan 15, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28250298223,924sleep128291-21ntpq21:49:511
2278726068,579sleep1194-21jbd2/mmcblk2p2-19:44:461
2278726068,579sleep1194-21jbd2/mmcblk2p2-19:44:461
7187243017,398sleep07188-21sshd22:32:420
28103236282,214sleep128260-21tail20:00:011
28103236282,214sleep128260-21tail20:00:011
718623150,27sleep112-21rcu_sched20:39:281
35862311166,110sleep13609-21modprobe22:19:271
704822680,82sleep112-21rcu_sched00:29:591
21095226776,22sleep10-21swapper/121:24:441
10373224547,183sleep110408-21proc_pri19:04:571
10373224547,183sleep110408-21proc_pri19:04:571
29199223815,208sleep229210-21unin-run21:54:292
3229222411,196sleep23291-21sendmail_mailst20:25:052
3229222411,196sleep23291-21sendmail_mailst20:25:052
19325221036,154sleep119351-21modprobe19:34:281
19325221036,154sleep119351-21modprobe19:34:281
17028221012,17sleep10-21swapper/123:09:461
2516822059,171sleep10-21swapper/123:39:291
12310220212,129sleep112352-21wc19:10:021
2544421858,166sleep125495-21wc21:39:301
2640621818,67sleep10-21swapper/119:55:001
2640621818,67sleep10-21swapper/119:55:001
32188218023,92sleep10-21swapper/122:04:531
3033521729,148sleep130455-21cat23:55:001
1007216574,22sleep10-21swapper/120:19:311
1007216574,22sleep10-21swapper/120:19:311
7679216310,115sleep17636-21apt-get20:40:061
26127216019,124sleep226271-21cat21:44:312
30268215728,68sleep130317-21latency_hist20:09:291
29548215511,128sleep129649-21fschecks_time21:54:481
16322215411,60sleep10-21swapper/119:24:311
17018215215,124sleep217086-21mailstats19:25:052
562621508,73sleep10-21swapper/100:25:281
2006121507,133sleep1370-21snmpd21:19:111
2006121507,133sleep1370-21snmpd21:19:111
12014215013,72sleep112027-21modprobe20:54:271
20310214716,116sleep220439-21sed19:35:022
20310214716,116sleep220439-21sed19:35:022
32377214529,92sleep332513-21unixbench_singl22:04:553
512521428,60sleep10-21swapper/120:30:011
2606821427,65sleep126209-21apt-config23:44:301
2140521379,89sleep121570-21sed23:24:561
2383121369,66sleep1286-21dbus-daemon21:34:291
1399821349,62sleep114086-21cat19:14:581
1517121335,93sleep10-21swapper/121:01:421
2839213211,86sleep12931-21grep20:24:561
2839213211,86sleep12931-21grep20:24:561
29327213113,53sleep11-21systemd23:54:271
29327213113,53sleep11-21systemd23:54:271
21327213153,60sleep021404-21/usr/sbin/munin23:24:550
376721305,89sleep10-21swapper/122:19:311
2709521306,43sleep10-21swapper/123:45:191
2709521306,43sleep10-21swapper/123:45:191
10905212966,33sleep10-21swapper/122:47:001
2632021287,50sleep10-21swapper/121:44:381
2632021287,50sleep10-21swapper/121:44:381
6155212714,95sleep16248-21fschecks_time22:29:491
3135021225,85sleep10-21swapper/100:01:461
26089212010,93sleep126125-21latency_hist21:44:301
2527521207,104sleep06455-21kworker/u8:119:51:030
2527521207,104sleep06455-21kworker/u8:119:51:030
1817221198,101sleep1370-21snmpd21:10:141
6973211815,87sleep36986-21/usr/sbin/munin00:29:583
31856211710,88sleep131984-21latency_hist20:14:301
2059121165,11sleep10-21swapper/121:22:461
2059121165,11sleep10-21swapper/121:22:461
179021165,11sleep10-21swapper/122:11:401
179021165,11sleep10-21swapper/122:11:401
5946211521,70sleep05963-21uniq20:34:480
11593211519,75sleep011110-21apt-get22:49:380
24284211418,54sleep10-21swapper/119:49:431
24284211418,54sleep10-21swapper/119:49:431
10201211443,56sleep210224-21/usr/sbin/munin22:44:472
2777621135,11sleep00-21swapper/023:49:250
2777621135,11sleep00-21swapper/023:49:250
2618211222,73sleep22651-21grep00:14:472
2618211222,73sleep22651-21grep00:14:472
10520211215,80sleep10-21swapper/120:49:311
4583211118,76sleep24592-21/usr/sbin/munin22:24:412
31644211017,23sleep00-21swapper/022:04:400
30594210921,75sleep230636-21fschecks_count20:09:472
32322210724,61sleep032388-21wc00:04:520
24653210510,14sleep30-21swapper/323:34:593
28532210418,64sleep228043-21/usr/sbin/munin23:49:562
28532210418,64sleep228043-21/usr/sbin/munin23:49:562
1591021036,60sleep10-21swapper/121:04:531
1591021036,60sleep10-21swapper/121:04:531
18913210217,65sleep218992-21fschecks_time21:14:482
18913210217,65sleep218992-21fschecks_time21:14:482
5774210149,41sleep15804-21munin-run22:29:281
11627210113,73sleep311645-21uniq19:09:353
26634210019,62sleep226645-21/usr/sbin/munin23:45:082
26634210019,62sleep226645-21/usr/sbin/munin23:45:082
21730210016,67sleep321872-21unixbench_singl21:24:583
21399210051,32sleep10-21swapper/119:39:491
21399210051,32sleep10-21swapper/119:39:491
1265329921,15sleep10-21swapper/120:54:481
1652829832,33sleep10-21swapper/119:24:471
812529763,19sleep10-21swapper/122:34:571
783929616,65sleep37900-21fschecks_time20:40:203
2623229612,67sleep026277-21awk19:54:550
2623229612,67sleep026277-21awk19:54:550
2119329610,72sleep021244-21cpuspeed19:39:440
2119329610,72sleep021244-21cpuspeed19:39:440
1127229620,58sleep011448-21munin-node19:09:320
1127229620,58sleep011448-21munin-node19:09:320
91929514,11sleep10-21swapper/100:09:371
218752959,16sleep00-21swapper/021:24:590
962429418,61sleep29782-21latency_hist22:44:312
279129322,33sleep10-21swapper/100:14:501
279129322,33sleep10-21swapper/100:14:501
207252937,54sleep10-21swapper/119:37:111
207252937,54sleep10-21swapper/119:37:111
1320829328,11sleep10-21swapper/122:54:441
33702929,17sleep30-21swapper/322:16:433
1161029210,60sleep10-21swapper/122:49:391
3209229020,33sleep10-21swapper/120:14:421
837528926,41sleep08520-21grep22:35:000
308702896,12sleep00-21swapper/023:58:460
958328833,38sleep19612-21munin-run19:04:281
2653028819,46sleep326295-21/usr/sbin/munin21:44:453
2653028819,46sleep326295-21/usr/sbin/munin21:44:453
136652886,19sleep00-21swapper/022:54:550
872128713,61sleep015780-21ntpd18:59:520
70828739,32sleep10-21swapper/100:09:291
3161128722,43sleep231731-21latency_hist00:04:292
933628611,63sleep10-21swapper/122:41:521
2226928610,36sleep10-21swapper/123:29:401
1858728615,57sleep00-21swapper/021:14:290
711528422,42sleep37136-21modprobe20:39:273
41532835,43sleep10-21swapper/100:20:561
2598828318,48sleep326027-21http23:44:283
1979428324,38sleep219826-21idleruntime-cro23:19:282
1380428322,45sleep313883-21find22:54:583
621728215,53sleep06235-21latency_hist00:29:270
2203628116,49sleep322177-21cat23:29:303
1850828127,33sleep20-21swapper/221:14:272
1815628127,34sleep30-21swapper/319:29:503
1815628127,34sleep30-21swapper/319:29:503
12472819,51sleep30-21swapper/300:09:513
2526828026,32sleep20-21swapper/223:39:302
2353328015,49sleep023571-21awk23:34:270
554027916,48sleep05603-21lockfile-create20:34:280
283127912,31sleep00-21swapper/020:24:550
283127912,31sleep00-21swapper/020:24:550
257372795,62sleep00-21swapper/023:41:140
320932789,49sleep20-21swapper/220:14:432
2965127820,28sleep20-21swapper/221:54:492
299427716,42sleep32506-21/usr/sbin/munin00:14:563
299427716,42sleep32506-21/usr/sbin/munin00:14:563
1984627622,25sleep00-21swapper/023:19:300
189682767,14sleep10-21swapper/123:14:541
1538927619,43sleep015429-21awk21:04:280
3181727518,41sleep331818-21/usr/sbin/munin00:04:353
1546327516,42sleep215542-21/usr/sbin/munin19:19:562
1257127513,47sleep01-21systemd20:54:430
123427515,16sleep30-21swapper/322:10:003
123427515,16sleep30-21swapper/322:10:003
2161727421,28sleep20-21swapper/221:24:572
112527414,45sleep21164-21ntp_kernel_pll_22:09:552
112527414,45sleep21164-21ntp_kernel_pll_22:09:552
2594827325,21sleep30-21swapper/319:54:503
2594827325,21sleep30-21swapper/319:54:503
2274227312,44sleep022786-21/usr/sbin/munin19:44:450
2274227312,44sleep022786-21/usr/sbin/munin19:44:450
1700427324,27sleep20-21swapper/221:09:312
1700427324,27sleep20-21swapper/221:09:312
150882736,10sleep10-21swapper/123:04:081
2530927211,47sleep325312-21sshd19:51:303
2530927211,47sleep325312-21sshd19:51:303
204402720,8sleep30-21swapper/319:35:023
204402720,8sleep30-21swapper/319:35:023
1666327211,41sleep0227-21systemd-journal23:09:280
159002725,11sleep10-21swapper/123:04:531
1205527212,23sleep20-21swapper/219:09:552
761527121,28sleep37620-21acpi22:34:393
562027110,36sleep212-21rcu_sched20:34:302
363827118,29sleep00-21swapper/022:19:300
1930927115,37sleep318510-21/usr/sbin/munin23:15:013
310942705,11sleep10-21swapper/122:01:061
2381827025,27sleep00-21swapper/021:34:280
94122697,13sleep00-21swapper/022:42:470
2970826913,41sleep329722-21irqstats21:54:503
265052696,51sleep30-21swapper/323:44:363
265052696,51sleep30-21swapper/323:44:363
2435126915,22sleep00-21swapper/023:34:560
2048726916,28sleep00-21swapper/023:24:300
1381926916,25sleep00-21swapper/019:14:530
963526813,40sleep39658-21ntp_states20:44:573
761626817,32sleep27128-21apt-get20:39:392
582226822,21sleep30-21swapper/322:29:303
2823526820,23sleep00-21swapper/021:49:500
203442685,11sleep10-21swapper/123:23:181
1783026813,26sleep00-21swapper/019:29:310
49732676,47sleep00-21swapper/022:24:510
2941526711,29sleep30-21swapper/323:54:293
2941526711,29sleep30-21swapper/323:54:293
188732670,7sleep00-21swapper/019:30:020
188732670,7sleep00-21swapper/019:30:020
1574826713,20sleep00-21swapper/021:04:480
1574826713,20sleep00-21swapper/021:04:480
131826713,30sleep00-21swapper/000:09:520
712026621,21sleep20-21swapper/200:30:022
2913226618,28sleep00-21swapper/020:05:080
2422426622,22sleep30-21swapper/321:34:473
1173826616,35sleep20-21swapper/222:50:042
1122026623,24sleep30-21swapper/319:09:313
1122026623,24sleep30-21swapper/319:09:313
939826516,33sleep29450-21if_eth020:44:522
914926526,19sleep20-21swapper/219:00:062
3097926416,28sleep331044-21ps20:10:003
274126420,23sleep00-21swapper/000:14:490
274126420,23sleep00-21swapper/000:14:490
217222645,10sleep20-21swapper/223:26:062
1630726426,17sleep30-21swapper/321:05:043
1630726426,17sleep30-21swapper/321:05:043
1468126425,22sleep30-21swapper/319:19:283
454826319,27sleep34635-21gpgv00:24:293
2484426320,29sleep224918-21switchtime19:50:002
2484426320,29sleep224918-21switchtime19:50:002
2282426314,32sleep322831-21/usr/sbin/munin19:44:483
2282426314,32sleep322831-21/usr/sbin/munin19:44:483
1703826315,31sleep217045-21/usr/sbin/munin23:09:472
1539626320,26sleep315389-21apt-get23:04:393
1041026316,21sleep20-21swapper/219:04:582
1041026316,21sleep20-21swapper/219:04:582
953826217,20sleep30-21swapper/322:44:283
698926216,23sleep00-21swapper/000:29:590
3255426219,24sleep30-21swapper/320:14:563
2639826212,34sleep026404-21diskstats21:44:420
2639826212,34sleep026404-21diskstats21:44:420
2127626214,22sleep30-21swapper/319:39:473
2127626214,22sleep30-21swapper/319:39:473
1678826218,23sleep30-21swapper/319:24:573
1505426215,22sleep00-21swapper/019:19:460
644626121,22sleep30-21swapper/322:29:553
499226118,27sleep24941-21apt-get00:25:092
3172226115,28sleep231727-21sshd22:04:422
204342615,40sleep20-21swapper/223:24:282
1965826114,19sleep30-21swapper/321:15:013
1965826114,19sleep30-21swapper/321:15:013
1449526120,19sleep30-21swapper/320:59:513
45502609,39sleep34571-21kernelversion20:29:503
427826019,17sleep00-21swapper/022:24:310
2613226017,22sleep20-21swapper/219:54:532
2613226017,22sleep20-21swapper/219:54:532
191526016,23sleep00-21swapper/020:20:020
191526016,23sleep00-21swapper/020:20:020
1318826015,19sleep20-21swapper/220:55:052
764025918,21sleep00-21swapper/020:39:410
740225915,21sleep20-21swapper/222:34:302
146125913,18sleep30-21swapper/320:19:533
146125913,18sleep30-21swapper/320:19:533
1408725912,22sleep212-21rcu_sched19:14:592
1389225915,21sleep30-21swapper/320:59:303
1360825913,22sleep20-21swapper/222:54:532
1121925917,27sleep011312-21dpkg22:49:300
314825816,23sleep23229-21users22:15:002
2896525818,25sleep228962-21apt-get20:04:462
2768425814,19sleep00-21swapper/019:59:510
2768425814,19sleep00-21swapper/019:59:510
1704625812,24sleep30-21swapper/323:09:483
1511225815,25sleep015140-21munin-run23:04:280
1210625816,24sleep30-21swapper/320:54:293
1144425818,18sleep00-21swapper/020:50:020
585725715,26sleep25854-21apt-get20:34:422
3256425716,26sleep032626-21ps20:14:580
3101425716,22sleep30-21swapper/323:59:303
2549625717,21sleep20-21swapper/221:39:312
2234825715,26sleep222357-21diskstats23:29:452
207292576,39sleep00-21swapper/019:37:140
207292576,39sleep00-21swapper/019:37:140
206342575,16sleep30-21swapper/321:23:213
206342575,16sleep30-21swapper/321:23:213
2990525614,26sleep029911-21ntp_states21:54:550
1554325615,20sleep30-21swapper/319:19:573
982825515,23sleep39825-21apt-get22:44:363
72725517,22sleep00-21swapper/022:09:410
72725517,22sleep00-21swapper/022:09:410
463725517,23sleep04786-21date00:24:300
1412825514,23sleep214122-21apt-get20:59:412
1201925517,18sleep30-21swapper/322:50:093
316225410,29sleep33198-21ntp_states20:25:023
316225410,29sleep33198-21ntp_states20:25:023
3057925411,31sleep030587-21df20:09:450
2921125414,20sleep30-21swapper/321:54:303
1516425413,25sleep315291-21wc23:04:313
139725416,24sleep21403-21ntp_states00:09:552
548125316,19sleep30-21swapper/300:25:263
2981825316,20sleep20-21swapper/223:54:482
2288225316,18sleep30-21swapper/323:29:583
1935525317,23sleep219403-21munin-run19:34:292
1935525317,23sleep219403-21munin-run19:34:292
1083425312,25sleep210841-21df_inode20:49:482
947525216,19sleep00-21swapper/020:44:540
2826225210,22sleep228315-21users20:00:022
2826225210,22sleep228315-21users20:00:022
2176725218,18sleep20-21swapper/219:39:592
2176725218,18sleep20-21swapper/219:39:592
788525114,21sleep27914-21irqstats22:34:502
313962516,36sleep30-21swapper/300:02:203
2904425114,22sleep329050-21diskstats20:04:513
2323225114,18sleep20-21swapper/221:30:022
182762517,33sleep30-21swapper/321:11:303
1700325113,22sleep017027-21df23:09:450
118925110,17sleep20-21swapper/220:19:442
118925110,17sleep20-21swapper/220:19:442
3245725011,15sleep20-21swapper/200:04:542
2861525016,16sleep00-21swapper/023:49:580
2861525016,16sleep00-21swapper/023:49:580
2741125012,20sleep327408-21apt-get19:59:423
2741125012,20sleep327408-21apt-get19:59:423
2365225014,16sleep20-21swapper/219:45:032
2365225014,16sleep20-21swapper/219:45:032
158472509,17sleep20-21swapper/223:04:502
2413324912,21sleep024129-21apt-get21:34:400
3135624813,19sleep00-21swapper/022:04:280
2303124817,17sleep30-21swapper/321:29:593
841624711,14sleep30-21swapper/318:59:473
204382475,12sleep20-21swapper/221:20:492
204382475,12sleep20-21swapper/221:20:492
2019324715,15sleep00-21swapper/021:19:300
2019324715,15sleep00-21swapper/021:19:300
1230324711,21sleep012309-21ntp_states19:10:010
1882324610,18sleep00-21swapper/021:14:420
1882324610,18sleep00-21swapper/021:14:420
3140924511,19sleep231504-21cut22:04:302
2758424410,13sleep20-21swapper/221:49:282
2758424410,13sleep20-21swapper/221:49:282
2603624410,19sleep326067-21munin-run21:44:283
44632399,14sleep20-21swapper/200:24:282
42672396,16sleep30-21swapper/322:24:303
42262399,16sleep20-21swapper/222:24:292
315602388,14sleep00-21swapper/000:04:270
1516323810,15sleep20-21swapper/223:04:302
258182379,13sleep00-21swapper/021:41:430
2047523710,13sleep30-21swapper/323:24:293
312972327,11sleep30-21swapper/322:03:423
256002326,15sleep20-21swapper/223:39:342
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional