You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 08:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Fri Nov 22, 2024 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
100972940,4sleep22527399cyclictest23:31:082
100972940,4sleep22527399cyclictest23:31:082
155552930,2sleep3381ktimersoftd/322:10:583
155552930,2sleep3381ktimersoftd/322:10:583
230542740,3sleep223042-21ssh21:39:212
230542740,3sleep223042-21ssh21:39:212
2527499716,19cyclictest0-21swapper/300:34:273
25271997131,5cyclictest0-21swapper/022:19:220
25271997125,6cyclictest0-21swapper/023:14:200
25271996943,5cyclictest8160-21grep00:14:340
25271996939,13cyclictest0-21swapper/022:59:100
50122680,2sleep15010-21irqstats23:24:311
2527199677,29cyclictest9-21ksoftirqd/021:24:300
25271996725,19cyclictest0-21swapper/022:14:240
2527199667,29cyclictest9-21ksoftirqd/019:29:380
2527199656,55cyclictest0-21swapper/022:39:230
2527199656,55cyclictest0-21swapper/022:39:230
2527199642,28cyclictest0-21swapper/019:39:210
25271996417,29cyclictest3723-21cron21:14:110
25271996417,29cyclictest3723-21cron21:14:110
2527199641,30cyclictest0-21swapper/021:14:220
2527199622,28cyclictest0-21swapper/023:19:240
2527199621,16cyclictest16196-21latency_hist20:19:110
25271996114,14cyclictest0-21swapper/023:09:220
25274996010,10cyclictest8050irq/56-dwc_otg_21:19:213
2527199601,28cyclictest6293-21uniq21:59:260
25271996010,32cyclictest9-21ksoftirqd/019:24:270
25271995922,9cyclictest0-21swapper/020:49:210
25271995921,24cyclictest0-21swapper/000:39:210
25271995825,5cyclictest0-21swapper/023:04:210
25271995815,5cyclictest1292-21runrttasks21:19:350
2527199565,29cyclictest0-21swapper/000:29:400
2527199565,29cyclictest0-21swapper/000:29:400
2527199564,25cyclictest0-21swapper/019:59:210
25271995611,20cyclictest9-21ksoftirqd/021:54:520
25271995524,12cyclictest0-21swapper/020:29:210
2527199552,30cyclictest11423-21runrttasks00:19:240
2527399542,31cyclictest0-21swapper/222:14:252
25271995410,24cyclictest464-21in:imuxsock00:35:540
2527199535,9cyclictest8150irq/56-dwc_otg_22:24:230
2527199532,29cyclictest13940-21grep20:09:380
25271995321,18cyclictest0-21swapper/022:44:220
25271995228,5cyclictest8050irq/56-dwc_otg_22:59:220
25271995222,13cyclictest8150irq/56-dwc_otg_23:24:340
2527299519,32cyclictest0-21swapper/119:59:221
2527199515,25cyclictest8253-21aten_rbpower_vo23:29:240
2527199515,25cyclictest8253-21aten_rbpower_vo23:29:240
25271995117,4cyclictest29170-21sh23:59:200
25271995117,4cyclictest29170-21sh23:59:200
2527199511,35cyclictest1340-21ls00:04:350
2527199511,30cyclictest10067-21fschecks_count22:04:270
2527199511,29cyclictest5840-21fschecks_count19:49:290
2527299505,4cyclictest0-21swapper/122:39:231
2527299505,4cyclictest0-21swapper/122:39:231
2527199506,30cyclictest0-21swapper/021:53:540
25271995031,8cyclictest101ktimersoftd/020:09:110
25271995028,5cyclictest8050irq/56-dwc_otg_23:54:310
25271995020,14cyclictest19064-21latency_hist23:44:100
2527199501,46cyclictest24503-21cat20:39:350
25271995013,24cyclictest0-21swapper/021:04:160
2527299499,6cyclictest8150irq/56-dwc_otg_22:09:231
2527299499,6cyclictest8150irq/56-dwc_otg_22:09:231
25271994926,10cyclictest8150irq/56-dwc_otg_20:44:270
25271994924,11cyclictest8150irq/56-dwc_otg_00:29:110
249552493,10sleep00-21swapper/019:13:370
2527399483,3cyclictest0-21swapper/223:19:242
2527299482,4cyclictest0-21swapper/123:04:111
2527199487,27cyclictest9-21ksoftirqd/022:49:280
25271994827,6cyclictest8050irq/56-dwc_otg_21:29:280
25271994825,12cyclictest0-21swapper/022:34:410
2527199482,31cyclictest0-21swapper/020:54:320
25271994822,6cyclictest8050irq/56-dwc_otg_21:34:440
2527499477,25cyclictest0-21swapper/319:39:163
25274994726,5cyclictest18468-21aten_rbpower_vo00:29:263
25274994726,5cyclictest18468-21aten_rbpower_vo00:29:263
2527199476,17cyclictest0-21swapper/021:44:110
2527199476,17cyclictest0-21swapper/021:44:110
25271994726,4cyclictest15362-21ssh23:39:110
2527199472,5cyclictest28680-21apt19:24:100
25271994719,5cyclictest5314-21unixbench_multi00:09:430
2527199471,24cyclictest131-21systemd-journal21:45:180
2527499467,24cyclictest0-21swapper/322:49:243
25274994613,18cyclictest7950irq/56-dwc_otg23:14:213
2527299466,34cyclictest0-21swapper/122:29:231
25272994614,29cyclictest0-21swapper/120:49:231
25271994622,13cyclictest18242-21/usr/sbin/munin20:24:330
25271994618,7cyclictest17278-21latency_hist22:14:110
25271994618,7cyclictest17278-21latency_hist22:14:110
25271994613,6cyclictest32616-21wget19:34:230
2527199461,23cyclictest19642-21fschecks_time23:44:290
98902450,18sleep22527399cyclictest00:16:252
2527499457,25cyclictest0-21swapper/320:24:243
2527299455,25cyclictest0-21swapper/119:44:271
2527299455,25cyclictest0-21swapper/119:44:271
2527299454,7cyclictest0-21swapper/121:59:221
25271994519,14cyclictest0-21swapper/022:33:040
25271994516,21cyclictest0-21swapper/020:37:470
25271994516,21cyclictest0-21swapper/020:37:470
25271994512,8cyclictest25611-21fschecks_time19:14:280
2527499448,26cyclictest0-21swapper/320:04:253
2527199442,6cyclictest32004-21unixbench_multi20:59:360
25271994422,13cyclictest0-21swapper/020:20:180
2527499437,26cyclictest0-21swapper/323:09:173
2527499437,22cyclictest0-21swapper/320:34:233
2527499437,22cyclictest0-21swapper/320:34:233
2527399433,7cyclictest0-21swapper/221:34:282
2527299438,10cyclictest0-21swapper/122:44:221
2527299435,30cyclictest11-21rcu_preempt00:29:221
2527299435,30cyclictest11-21rcu_preempt00:29:221
2527199431,23cyclictest8084-21ntpq19:54:330
203912430,8sleep32527499cyclictest20:29:263
84392421,37sleep38507-21sed00:14:383
2527399422,18cyclictest12279-21fschecks_time23:34:332
25273994220,4cyclictest18272-21sh23:42:432
2527399421,5cyclictest8150irq/56-dwc_otg_23:09:222
2527399421,5cyclictest8150irq/56-dwc_otg_21:09:222
2527399421,5cyclictest8150irq/56-dwc_otg_21:09:222
2527299428,23cyclictest7950irq/56-dwc_otg00:39:391
2527299427,25cyclictest0-21swapper/122:54:271
2527299422,11cyclictest0-21swapper/100:24:231
25272994219,4cyclictest10098-21fschecks_time22:04:281
2527299421,7cyclictest11709-21/usr/sbin/munin21:24:291
271032411,35sleep30-21swapper/321:44:273
25274994118,5cyclictest9022-21unixbench_multi22:44:383
2527399413,15cyclictest0-21swapper/222:24:232
2527299417,24cyclictest0-21swapper/123:09:161
2527299415,4cyclictest7950irq/56-dwc_otg00:19:231
2527299411,9cyclictest107850ssh22:34:381
25272994118,9cyclictest7950irq/56-dwc_otg21:29:341
2527199412,21cyclictest0-21swapper/019:45:380
2527199412,21cyclictest0-21swapper/019:45:380
25271994114,16cyclictest0-21swapper/023:50:090
2527399404,17cyclictest8050irq/56-dwc_otg_23:09:122
2527299406,22cyclictest0-21swapper/120:59:211
2527299406,17cyclictest8050irq/56-dwc_otg_21:44:111
2527299406,17cyclictest8050irq/56-dwc_otg_21:44:111
2527299405,28cyclictest8150irq/56-dwc_otg_00:04:201
25272994017,15cyclictest0-21swapper/100:35:541
216542400,3sleep10-21swapper/121:36:591
25274993911,13cyclictest8150irq/56-dwc_otg_23:52:323
2527399399,10cyclictest0-21swapper/219:39:212
2527399396,21cyclictest0-21swapper/219:44:272
2527399396,21cyclictest0-21swapper/219:44:272
25273993916,6cyclictest20219-21/usr/sbin/munin23:44:372
2527299395,24cyclictest7852-21ssh00:14:301
25272993919,4cyclictest13654-21sh22:51:531
25272993916,6cyclictest0-21swapper/119:29:341
49652380,1sleep20-21swapper/221:57:492
2527499387,19cyclictest0-21swapper/320:44:233
2527399382,7cyclictest0-21swapper/200:04:202
2527399381,5cyclictest963-21missed_timers22:34:362
2527299385,21cyclictest7950irq/56-dwc_otg21:50:581
2527299384,7cyclictest0-21swapper/123:19:241
25272993816,4cyclictest4001-21sh21:55:521
131752380,4sleep231-21ksoftirqd/222:50:512
2527499377,19cyclictest8150irq/56-dwc_otg_00:44:103
2527499376,21cyclictest21300-21ssh22:19:233
2527499373,29cyclictest8150irq/56-dwc_otg_23:19:223
2527499371,23cyclictest29988-21ls22:29:363
2527399377,4cyclictest0-21swapper/223:29:112
2527399377,16cyclictest17280-21latency_hist22:14:112
2527399377,16cyclictest17280-21latency_hist22:14:112
2527399371,25cyclictest0-21swapper/223:50:182
2527399371,22cyclictest29252-21ls20:54:262
2527299377,22cyclictest0-21swapper/123:59:231
2527299377,22cyclictest0-21swapper/123:59:231
2527299377,21cyclictest7950irq/56-dwc_otg20:44:241
2527299377,21cyclictest7950irq/56-dwc_otg19:54:101
2527299373,21cyclictest7950irq/56-dwc_otg23:55:211
2527299372,5cyclictest0-21swapper/123:14:201
25272993710,5cyclictest0-21swapper/121:49:101
242482375,22sleep324291-21/usr/sbin/munin19:09:353
25274993615,11cyclictest4029-21sh22:39:243
25274993615,11cyclictest4029-21sh22:39:243
25274993614,14cyclictest381ktimersoftd/322:18:313
2527499361,23cyclictest3725-21latency_hist21:14:123
2527499361,23cyclictest3725-21latency_hist21:14:123
25274993612,11cyclictest0-21swapper/322:39:113
2527399365,18cyclictest0-21swapper/221:04:222
2527399362,23cyclictest0-21swapper/220:29:212
2527399362,13cyclictest30588-21ssh23:14:352
2527399361,21cyclictest27143-21ls20:49:102
2527299366,22cyclictest7950irq/56-dwc_otg23:34:291
2527299366,13cyclictest0-21swapper/123:39:291
2527299361,9cyclictest16178-21cron20:19:101
25272993618,5cyclictest25357-21wget19:14:241
25272993615,4cyclictest8409-21kernelversion21:19:311
2527499356,14cyclictest0-21swapper/320:14:253
2527499355,10cyclictest8150irq/56-dwc_otg_23:39:393
25274993512,21cyclictest15567-21ssh21:29:253
2527399354,23cyclictest8150irq/56-dwc_otg_22:39:222
2527399354,23cyclictest8150irq/56-dwc_otg_22:39:222
2527399353,21cyclictest8150irq/56-dwc_otg_23:59:402
2527399353,21cyclictest8150irq/56-dwc_otg_23:59:402
2527399353,20cyclictest0-21swapper/219:29:112
2527399351,24cyclictest15603-21missed_timers22:54:372
25273993511,11cyclictest13396-21latency_hist22:09:112
2527299354,18cyclictest7950irq/56-dwc_otg20:40:581
2527299353,18cyclictest7950irq/56-dwc_otg22:26:581
2527299351,26cyclictest4467-21ls21:14:291
163462350,2sleep20-21swapper/220:19:112
258522340,14sleep22527399cyclictest19:14:312
2527499345,20cyclictest30712-21ssh21:49:223
2527499345,15cyclictest8050irq/56-dwc_otg_21:34:443
2527499341,5cyclictest8050irq/56-dwc_otg_00:04:263
2527499341,23cyclictest28683-21latency_hist19:24:103
2527499341,21cyclictest5618-21timerwakeupswit23:24:403
2527399347,15cyclictest12908-21aten_rbpower_po20:09:242
2527399347,11cyclictest8050irq/56-dwc_otg_22:04:102
2527399346,16cyclictest0-21swapper/222:59:212
2527399343,10cyclictest0-21swapper/220:49:222
25273993418,9cyclictest27619-21latency19:19:312
25273993411,6cyclictest0-21swapper/200:27:442
2527299349,11cyclictest1911-21modprobe21:09:101
2527299348,6cyclictest23-21ksoftirqd/122:19:381
2527299347,16cyclictest12078-21grep20:04:401
2527299344,21cyclictest8050irq/56-dwc_otg_23:44:401
25272993415,4cyclictest1207-21unixbench_singl19:34:371
2527299341,4cyclictest0-21swapper/123:04:201
2527299341,27cyclictest0-21swapper/100:13:441
2527499337,6cyclictest0-21swapper/300:24:423
2527499337,5cyclictest7950irq/56-dwc_otg23:29:373
2527499337,5cyclictest7950irq/56-dwc_otg23:29:373
25274993314,5cyclictest1292-21runrttasks20:53:583
2527499331,16cyclictest28869-21munin-run23:59:113
25274993310,4cyclictest0-21swapper/323:59:373
25274993310,4cyclictest0-21swapper/323:59:373
2527399337,23cyclictest0-21swapper/221:44:232
2527399337,23cyclictest0-21swapper/219:59:222
2527399335,18cyclictest25615-21ls00:39:382
2527399332,22cyclictest0-21swapper/222:44:302
2527399332,20cyclictest0-21swapper/220:44:112
25273993316,8cyclictest3733-21diskmemload00:34:282
2527299336,18cyclictest3056-21grep19:39:371
2527299334,6cyclictest11569-21munin-run23:34:111
2527299334,6cyclictest11569-21munin-run23:34:111
2527299332,4cyclictest7950irq/56-dwc_otg21:14:111
2527299332,4cyclictest7950irq/56-dwc_otg21:14:111
2527299332,4cyclictest7950irq/56-dwc_otg20:29:201
2527299331,7cyclictest0-21swapper/119:19:221
25272993310,4cyclictest0-21swapper/123:49:271
2527499328,11cyclictest8050irq/56-dwc_otg_23:44:403
2527499326,5cyclictest7950irq/56-dwc_otg21:39:293
2527499326,5cyclictest7950irq/56-dwc_otg21:39:293
2527499322,7cyclictest31118-21fschecks_count20:59:253
2527499322,5cyclictest26052-21proc_pri19:14:353
2527499322,4cyclictest8050irq/56-dwc_otg_19:34:333
2527499322,19cyclictest0-21swapper/322:24:213
2527499321,2cyclictest0-21swapper/321:24:203
2527499321,21cyclictest18900-21uniq22:59:253
25274993210,5cyclictest8050irq/56-dwc_otg_22:59:103
2527399325,12cyclictest8150irq/56-dwc_otg_00:29:262
2527399325,12cyclictest8150irq/56-dwc_otg_00:29:262
2527399323,17cyclictest22432-21grep20:34:312
2527399323,17cyclictest22432-21grep20:34:312
25273993213,10cyclictest31-21ksoftirqd/221:49:232
2527399321,25cyclictest25787-21fschecks_count23:54:252
2527399321,25cyclictest0-21swapper/200:13:542
2527399321,22cyclictest5152-21timerwakeupswit21:14:382
2527399321,22cyclictest0-21swapper/200:19:212
2527499318,9cyclictest0-21swapper/300:13:253
2527499315,6cyclictest8050irq/56-dwc_otg_23:04:233
2527499312,4cyclictest0-21swapper/323:34:363
2527499311,2cyclictest0-21swapper/321:54:243
25274993112,4cyclictest7020-21seq21:59:363
2527399317,15cyclictest0-21swapper/222:29:232
2527399316,17cyclictest0-21swapper/220:59:212
2527399314,5cyclictest31-21ksoftirqd/222:19:382
2527399313,3cyclictest7950irq/56-dwc_otg21:19:332
2527399311,18cyclictest16932-21ls20:19:312
2527399311,12cyclictest0-21swapper/220:28:422
2527399311,12cyclictest0-21swapper/219:32:112
25273993111,11cyclictest7690-21fschecks_time19:54:282
2527299317,12cyclictest7950irq/56-dwc_otg20:25:031
2527299317,11cyclictest207332diskmemload22:18:431
2527499309,5cyclictest4801-21ntp_kernel_err21:14:333
2527499305,12cyclictest11366-21/usr/sbin/munin00:19:263
25274993015,4cyclictest30802-21aten_rbpower_po19:29:243
2527499301,4cyclictest7950irq/56-dwc_otg20:39:233
2527399308,14cyclictest0-21swapper/221:24:242
2527399304,6cyclictest8050irq/56-dwc_otg_21:29:252
2527399301,19cyclictest0-21swapper/219:34:372
2527299309,7cyclictest0-21swapper/119:24:241
2527299308,12cyclictest8160-21ps19:54:351
2527299306,10cyclictest7950irq/56-dwc_otg20:19:311
25272993015,4cyclictest13644-21sensors20:09:351
2527499296,14cyclictest0-21swapper/319:54:233
2527499295,4cyclictest7950irq/56-dwc_otg20:19:243
2527499294,7cyclictest13138-21fschecks_time20:09:273
2527499293,6cyclictest29748-21ntp_kernel_err20:54:333
25274992919,7cyclictest10923-21ssh22:04:383
2527499291,17cyclictest4673-21switchtime19:44:393
2527499291,17cyclictest4673-21switchtime19:44:393
2527399291,11cyclictest0-21swapper/219:53:042
249042293,16sleep10-21swapper/119:12:571
2527499281,20cyclictest0-21swapper/321:05:243
2527299286,14cyclictest8050irq/56-dwc_otg_20:34:291
2527299286,14cyclictest8050irq/56-dwc_otg_20:34:291
2527299285,16cyclictest221ktimersoftd/120:54:321
2527399271,14cyclictest0-21swapper/220:05:172
239652275,15sleep224012-21memory19:09:302
2527499261,22cyclictest0-21swapper/319:49:253
2527499261,13cyclictest8150irq/56-dwc_otg_20:04:103
2527499252,5cyclictest8050irq/56-dwc_otg_19:24:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional