You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-01 - 02:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackfslot7s.osadl.org (updated Sat Aug 31, 2024 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
296554999315015,3131cyclictest3050173-21kworker/3:1+events23:32:353
296554999284312,2827cyclictest3053374-21kworker/3:2+events23:55:013
296554999281712,2800cyclictest3024590-21kworker/3:2+events22:25:123
296554999280713,2790cyclictest3067588-21kworker/3:0+events00:30:113
296554999279713,2781cyclictest2994461-21kworker/3:1+events21:05:013
296554999279513,2778cyclictest3014792-21kworker/3:1+events21:50:173
296554999279011,2776cyclictest3058442-21kworker/3:0+events00:10:213
296554999275713,2742cyclictest3010024-21kworker/3:2+events21:30:013
296554999274413,2728cyclictest2971970-21kworker/3:2+events19:32:333
296554999257012,2554cyclictest3022744-21kworker/3:0+events22:10:013
29655499925307,2519cyclictest2968771-21kworker/3:0+events19:25:163
296554999252212,2506cyclictest2968771-21kworker/3:0+events19:58:033
296554999251512,2501cyclictest3024590-21kworker/3:2+events22:20:143
296554999250311,2489cyclictest3050173-21kworker/3:1+events23:43:173
296554999248912,2473cyclictest3037055-21kworker/3:2+events23:05:113
296554999248512,2469cyclictest3032258-21kworker/3:2+events22:42:163
296554999247412,2458cyclictest3005243-21kworker/3:1+events21:19:133
296554999247312,2457cyclictest2968771-21kworker/3:0+events19:40:113
296554999247110,2457cyclictest2968771-21kworker/3:0+events19:47:093
296554999246412,2448cyclictest2991292-21kworker/3:2+events20:41:343
296554999245311,2438cyclictest3022744-21kworker/3:0+events22:45:153
296554999243912,2424cyclictest3045135-21kworker/3:1+events23:25:123
296554999243812,2423cyclictest3060531-21kworker/3:1+events00:15:483
296554999243712,2423cyclictest3010024-21kworker/3:2+events21:35:023
296554999243412,2418cyclictest3005243-21kworker/3:1+events21:12:523
296554999243311,2419cyclictest3053374-21kworker/3:2+events23:45:513
296554999242811,2414cyclictest2991292-21kworker/3:2+events21:05:223
296554999242612,2412cyclictest3060531-21kworker/3:1+events00:25:093
296554999242612,2411cyclictest2984937-21kworker/3:0+events20:17:353
296554999242611,2412cyclictest3022744-21kworker/3:0+events22:10:453
296554999242512,2410cyclictest3017957-21kworker/3:2+events22:03:143
296554999242512,2409cyclictest2968771-21kworker/3:0+events19:36:103
296554999242412,2409cyclictest3003662-21kworker/3:0+events21:21:033
296554999241912,2403cyclictest3037055-21kworker/3:2+events22:56:363
296554999241612,2400cyclictest2989713-21kworker/3:1+events20:31:233
296554999241410,2401cyclictest2994461-21kworker/3:1+events20:46:433
296554999240610,2393cyclictest2994461-21kworker/3:1+events20:58:363
296554999240412,2388cyclictest3022744-21kworker/3:0+events22:53:333
296554999240310,2392cyclictest3010024-21kworker/3:2+events21:33:453
296554999240210,2389cyclictest2968771-21kworker/3:0+events19:20:073
296554999240012,2385cyclictest2989713-21kworker/3:1+events20:24:553
296554999238412,2364cyclictest2968771-21kworker/3:0+events20:01:013
296554999238112,2366cyclictest2983084-21kworker/3:2+events20:07:563
296554999237712,2362cyclictest3032258-21kworker/3:2+events22:39:473
296554999237511,2360cyclictest2968771-21kworker/3:0+events19:51:523
296554999237511,2360cyclictest2968771-21kworker/3:0+events19:51:513
296554999235911,2345cyclictest2984937-21kworker/3:0+events20:14:523
296554999233012,2315cyclictest3037055-21kworker/3:2+events23:12:443
296554999232011,2305cyclictest3069192-21kworker/3:1+events00:36:453
296554999231911,2305cyclictest3055811-21kworker/3:3+events00:00:503
296554999227410,2261cyclictest3064385-21kworker/3:0+events00:20:353
29655499922658,2254cyclictest2989713-21kworker/3:1+events20:25:393
296554999226510,2252cyclictest3022744-21kworker/3:0+events22:15:203
296554999226411,2250cyclictest3037055-21kworker/3:2+events23:02:583
296554999225910,2246cyclictest3017957-21kworker/3:2+events21:55:123
296554999224911,2234cyclictest2968771-21kworker/3:0+events19:18:403
29655499922448,2233cyclictest3022744-21kworker/3:0+events22:30:513
296554999224411,2229cyclictest3043553-21kworker/3:0+events23:23:153
296554999224211,2228cyclictest3010024-21kworker/3:2+events21:41:483
29655499922399,2227cyclictest3014792-21kworker/3:1+events21:48:483
296554999223910,2226cyclictest3050173-21kworker/3:1+events23:39:023
296554999223811,2224cyclictest2991292-21kworker/3:2+events20:35:413
296554999223811,2223cyclictest3060531-21kworker/3:1+events00:07:573
296554999223710,2223cyclictest2991292-21kworker/3:2+events20:54:003
296554999223311,2219cyclictest3055811-21kworker/3:3+events23:55:233
296554999223211,2218cyclictest3043553-21kworker/3:0+events23:17:093
296554999223211,2217cyclictest2953719-21kworker/3:2+events19:13:123
29650772181151,26sleep20-21swapper/219:06:412
29652852142111,26sleep00-21swapper/019:09:130
29652982141109,25sleep10-21swapper/119:09:221
29652632140104,29sleep30-21swapper/319:08:553
296554399106103,2cyclictest3052090-21kworker/u8:1+flush-259:023:50:001
2965543991050,104cyclictest1-21systemd23:05:001
2965543996259,3cyclictest2963815-21kworker/u8:3+flush-259:019:45:011
2965543995856,2cyclictest3058703-21kworker/u8:4+flush-259:000:35:011
2965543995449,3cyclictest298-21systemd-journal20:20:011
2965541995044,4cyclictest3040767-21latency_hist23:05:010
2965543994944,3cyclictest2981463-21kworker/u8:4+events_unbound20:00:011
2965541994938,8cyclictest3058354-21latency_hist00:00:010
2965547994642,3cyclictest3046009-21kworker/u8:3+flush-259:023:34:592
2965543994440,3cyclictest3016843-21kworker/u8:1+flush-259:022:45:161
2965543994340,2cyclictest2963815-21kworker/u8:3+flush-259:019:20:161
2965543994240,2cyclictest3029498-21kworker/u8:0+flush-259:022:50:111
2965543994236,2cyclictest3046009-21kworker/u8:3+events_unbound23:40:131
2965547994037,2cyclictest2983703-21kworker/u8:3+events_unbound20:30:192
2965543994039,1cyclictest2986386-21kworker/u8:0+events_unbound20:20:161
2965541994033,5cyclictest3022903-21apt-get22:09:590
2965541993935,3cyclictest2981463-21kworker/u8:4+events_unbound20:01:040
2965541993927,7cyclictest3026593-21apt-get22:20:120
2965543993834,3cyclictest298-21systemd-journal21:25:401
2965543993735,2cyclictest2988229-21kworker/u8:2+events_unbound21:20:011
2965543993735,1cyclictest3046009-21kworker/u8:3+flush-259:023:40:001
2965543993734,2cyclictest2994530-21kworker/u8:1+events_unbound21:05:181
2965543993733,3cyclictest3016843-21kworker/u8:1+flush-259:022:20:161
2965541993733,3cyclictest3046009-21kworker/u8:3+flush-259:023:45:000
2965541993732,3cyclictest3038922-21apt-get23:00:000
296554199370,35cyclictest678-21gmain23:45:160
2965541993628,7cyclictest2993178-21apt-get20:35:100
2965547993530,3cyclictest3052050-21latency_hist23:40:012
2965541993533,2cyclictest3018414-21kworker/u8:4+flush-259:022:05:010
2965547993432,2cyclictest2981463-21kworker/u8:4+flush-259:020:00:002
2965547993329,3cyclictest3043933-21kworker/u8:2+flush-259:023:25:002
2965543993331,2cyclictest3055245-21kworker/u8:2+flush-259:023:50:151
2965543993330,2cyclictest3004033-21kworker/u8:0+events_unbound21:35:201
2965543993330,2cyclictest2988229-21kworker/u8:2+flush-259:020:52:451
2965547993228,3cyclictest3029734-21apt-get22:30:102
2965547993225,4cyclictest2971546-21python319:25:172
2965543993230,2cyclictest3018414-21kworker/u8:4+events_unbound22:00:201
2965543993230,2cyclictest2983703-21kworker/u8:3+events_unbound20:30:011
2965541993227,4cyclictest298-21systemd-journal19:50:000
2965547993126,3cyclictest3004661-21meminfo21:10:152
2965547993025,4cyclictest298-21systemd-journal19:30:112
2965543993029,1cyclictest2975580-21kworker/u8:1+events_unbound19:50:111
2965543993029,1cyclictest2975580-21kworker/u8:1+events_unbound19:50:111
2965541993025,3cyclictest3047340-21latency_hist23:25:000
2965547992921,5cyclictest581-21rs:main2
2965541992923,4cyclictest3032615-21latency_hist22:40:010
2965547992825,2cyclictest3016843-21kworker/u8:1+flush-259:022:45:002
2965543992827,1cyclictest3010200-21kworker/u8:3+flush-259:021:45:001
2965543992826,2cyclictest3016843-21kworker/u8:1+events_unbound21:52:091
2965543992826,2cyclictest2963815-21kworker/u8:3+events_unbound19:12:501
2965543992825,2cyclictest3046009-21kworker/u8:3+flush-259:000:20:201
2965543992825,2cyclictest3043933-21kworker/u8:2+events_unbound23:30:111
2965547992725,2cyclictest3020966-21kworker/u8:2+flush-259:022:19:452
2965547992722,3cyclictest3027895-21latency_hist22:25:002
2965543992724,2cyclictest3058703-21kworker/u8:4+flush-259:000:25:161
2965543992723,3cyclictest2981462-21kworker/u8:2+flush-259:020:05:151
2965541992725,2cyclictest3063211-21kworker/u8:0+flush-259:000:21:460
2965541992720,5cyclictest298-21systemd-journal23:25:170
2965547992620,3cyclictest3013464-21systemctl21:40:002
2965543992624,2cyclictest3029498-21kworker/u8:0+events_unbound23:05:141
2965543992624,2cyclictest2983703-21kworker/u8:3+flush-259:020:10:151
2965543992623,2cyclictest2994530-21kworker/u8:1+events_unbound20:57:471
2965547992523,2cyclictest2988229-21kworker/u8:2+events_unbound21:45:212
2965547992522,2cyclictest2988229-21kworker/u8:2+events_unbound21:15:182
296554799250,24cyclictest1-21systemd21:10:002
2965543992523,2cyclictest3046009-21kworker/u8:3+flush-259:000:00:161
2965543992523,2cyclictest3029498-21kworker/u8:0+events_unbound22:55:161
2965543992523,2cyclictest2994530-21kworker/u8:1+events_unbound20:47:531
2965543992523,2cyclictest2986386-21kworker/u8:0+flush-259:020:40:201
2965543992523,2cyclictest2963815-21kworker/u8:3+events_unbound19:15:021
2965543992522,2cyclictest2961994-21kworker/u8:2+flush-259:019:29:541
2965543992521,3cyclictest3020966-21kworker/u8:2+flush-259:022:20:001
2965543992519,4cyclictest2972813-21meminfo19:30:151
2965541992523,2cyclictest3027535-21kworker/u8:4+flush-259:022:30:010
2965541992521,3cyclictest3016843-21kworker/u8:1+flush-259:022:20:010
2965547992422,2cyclictest2961994-21kworker/u8:2+events_unbound19:15:192
2965547992421,2cyclictest3016843-21kworker/u8:1+events_unbound21:50:182
2965547992419,3cyclictest3035776-21latency_hist22:50:002
2965547992419,3cyclictest2988181-21latency_hist20:20:002
2965547992418,4cyclictest3020535-21ntpq22:00:152
2965543992422,2cyclictest2988229-21kworker/u8:2+events_unbound21:21:401
2965543992422,2cyclictest2981463-21kworker/u8:4+events_unbound20:00:151
2965543992422,1cyclictest3010200-21kworker/u8:3+flush-259:021:33:571
2965543992421,2cyclictest3046009-21kworker/u8:3+events_unbound00:10:191
2965543992420,3cyclictest3024730-21latency_hist22:15:001
2965541992421,3cyclictest2988229-21kworker/u8:2+flush-259:021:08:380
2965541992419,3cyclictest2989769-21latency_hist20:25:010
296554199240,23cyclictest581-21rs:main0
296554199240,23cyclictest1-21systemd20:05:000
2965547992321,2cyclictest3058703-21kworker/u8:4+flush-259:000:03:482
2965541992320,2cyclictest3002077-21apt-get21:01:070
2965541992311,10cyclictest3018949-21ntpq21:55:150
296554199230,13cyclictest3066360-21perf00:29:550
296554199230,13cyclictest3063424-21/usr/sbin/munin00:15:170
2965543992220,2cyclictest3063211-21kworker/u8:0+flush-259:000:19:541
2965543992220,2cyclictest3055245-21kworker/u8:2+events_unbound23:55:141
2965543992220,2cyclictest3024367-21kworker/u8:3+events_unbound22:35:131
2965543992220,2cyclictest3016843-21kworker/u8:1+events_unbound22:40:141
2965543992220,2cyclictest2986386-21kworker/u8:0+events_unbound20:32:091
296554199226,14cyclictest298-21systemd-journal21:19:590
2965541992220,2cyclictest2994530-21kworker/u8:1+flush-259:020:50:010
2965541992217,3cyclictest2980026-21latency_hist19:55:010
2965541992217,3cyclictest2980026-21latency_hist19:55:000
296554199220,0cyclictest0-21swapper/023:10:150
2965547992116,3cyclictest2996072-21latency_hist20:45:002
2965547992116,3cyclictest2967008-21latency_hist19:15:012
2965543992119,2cyclictest3016843-21kworker/u8:1+events_unbound23:25:051
2965543992119,2cyclictest2994530-21kworker/u8:1+events_unbound21:13:191
296554199215,14cyclictest2975318-21awk19:40:000
2965541992119,2cyclictest2961994-21kworker/u8:2+flush-259:019:25:000
2965541992116,4cyclictest2986386-21kworker/u8:0+flush-259:020:35:000
296554199211,3cyclictest3052046-21cron23:40:000
2965547992016,3cyclictest3022926-21http22:10:012
2965543992018,2cyclictest3043933-21kworker/u8:2+events_unbound23:20:191
2965543992018,2cyclictest3024367-21kworker/u8:3+events_unbound22:31:181
2965543992018,2cyclictest2988229-21kworker/u8:2+flush-259:020:35:211
2965543992018,2cyclictest2963815-21kworker/u8:3+events_unbound19:47:231
296554199201,6cyclictest3049405-21hddtemp_smartct23:30:130
296554199201,3cyclictest3013454-21apt21:40:000
296554199201,2cyclictest2991349-21munin-run20:30:000
2965547991917,2cyclictest3023419-21kworker/u8:0+flush-259:022:10:162
2965547991916,2cyclictest1203-21snmpd23:07:482
2965547991913,4cyclictest298-21systemd-journal23:59:592
2965543991917,2cyclictest3046009-21kworker/u8:3+events_unbound00:08:411
2965543991917,2cyclictest3016843-21kworker/u8:1+flush-259:023:10:171
2965543991917,2cyclictest3004033-21kworker/u8:0+flush-259:021:55:111
2965543991917,2cyclictest2994530-21kworker/u8:1+events_unbound21:00:101
296554199195,13cyclictest579-21in:imuxsock23:10:000
296554199191,3cyclictest2981610-21munin-run20:00:010
296554199191,2cyclictest3000796-21systemctl21:00:000
296554199190,5cyclictest3034211-21munin-run22:45:010
2965547991814,3cyclictest3039373-21cat23:00:022
2965547991814,3cyclictest2996314-21/usr/sbin/munin20:45:142
2965547991813,3cyclictest3010303-21latency_hist21:30:002
2965543991816,2cyclictest2988229-21kworker/u8:2+events_unbound21:48:171
296554199185,11cyclictest3070305-21ntpq00:35:170
296554199184,2cyclictest2988184-21sendmail20:20:010
2965541991815,2cyclictest3061880-21apt-get00:10:110
296554199181,4cyclictest3045498-21dump-pmu-power23:20:000
296554199181,4cyclictest2968997-21sendmail19:19:590
296554199181,3cyclictest0-21swapper/022:14:590
296554199181,2cyclictest3060056-21awk00:05:000
296554199181,2cyclictest3031045-21idleruntime-cro22:35:000
2965543991715,2cyclictest3058703-21kworker/u8:4+events_unbound00:38:111
2965543991715,2cyclictest3029498-21kworker/u8:0+events_unbound23:18:411
296554199171,2cyclictest3011886-21awk21:35:000
296554199171,2cyclictest3008744-21munin-run21:24:590
2965547991614,2cyclictest2988229-21kworker/u8:2+flush-259:021:25:002
2965547991611,3cyclictest3000776-21latency_hist21:00:012
296554799160,3cyclictest3061271-21taskset00:05:582
296554799160,15cyclictest3069310-21seq00:32:382
2965543991614,2cyclictest2961994-21kworker/u8:2+events_unbound19:35:171
2965543991612,2cyclictest0-21swapper/122:30:011
296554199161,2cyclictest2967025-21idleruntime-cro19:14:590
2965541991611,4cyclictest3034681-21diskstats22:45:130
2965547991512,3cyclictest2986386-21kworker/u8:0+flush-259:020:29:492
2965547991511,3cyclictest401ktimers/220:50:122
2965547991511,3cyclictest3067655-21chrt00:26:342
2965547991511,3cyclictest3043591-21taskset23:11:132
2965547991511,3cyclictest2986386-21kworker/u8:0+flush-259:020:35:152
2965547991511,2cyclictest2983966-21hddtemp_smartct20:05:142
296554799150,2cyclictest3011585-21runrttasks21:31:332
296554799150,15cyclictest0-21swapper/221:00:142
296554799150,13cyclictest3044569-21runrttasks23:15:142
2965543991512,2cyclictest13155-21runrttasks22:06:231
296554199159,5cyclictest1203-21snmpd23:54:180
296554199151,3cyclictest0-21swapper/019:29:590
296554199151,2cyclictest3061620-21perf00:10:000
296554199151,2cyclictest2999219-21perf20:55:000
296554199151,2cyclictest2976896-21munin-run19:44:590
296554199151,2cyclictest2973087-21ps19:30:190
2965541991511,3cyclictest3004471-21/usr/sbin/munin21:10:120
296554199150,2cyclictest2996088-21idleruntime-cro20:45:010
296554199150,2cyclictest1203-21snmpd00:30:490
296554799149,4cyclictest3066118-21chrt00:22:212
2965547991412,2cyclictest3046009-21kworker/u8:3+flush-259:023:55:002
2965547991412,1cyclictest3028875-21/usr/sbin/munin22:25:212
2965547991411,2cyclictest45943-21sshd00:13:422
2965547991411,2cyclictest3014374-21pmu-power21:40:192
2965547991410,3cyclictest3063930-21ntpq00:15:162
2965547991410,2cyclictest3032658-21true22:40:012
2965547991410,2cyclictest2970583-21idleruntime-cro19:25:002
296554799140,3cyclictest3038628-21sshd22:56:272
296554799140,13cyclictest2989465-21chrt20:21:252
296554799140,13cyclictest2982857-21sshd20:01:042
296554199148,5cyclictest2985756-21/usr/sbin/munin20:10:140
296554199144,3cyclictest3014262-21ld21:40:160
296554199143,3cyclictest3036308-21cat22:50:130
296554199140,3cyclictest3010240-21sshd21:29:180
296554799139,3cyclictest2975227-21sshd19:39:162
296554799139,2cyclictest751-21mta-sts-daemon23:40:172
2965547991311,2cyclictest2961994-21kworker/u8:2+flush-259:019:50:012
2965547991311,1cyclictest3069764-21/usr/sbin/munin00:35:162
2965547991311,1cyclictest3054115-21/usr/sbin/munin23:45:122
296554199130,12cyclictest3016852-21/usr/sbin/munin21:50:120
296554799129,2cyclictest2979641-21taskset19:50:242
296554799129,2cyclictest2979641-21taskset19:50:232
296554799128,3cyclictest3018416-21chrt21:55:062
296554799127,4cyclictest3048762-21taskset23:28:142
296554799120,1cyclictest2975557-21/usr/sbin/munin19:40:182
296554799120,12cyclictest0-21swapper/222:52:092
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional