You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 08:24
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack9slot2.osadl.org (updated Fri Nov 22, 2024 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24802316278,25sleep20-21swapper/22
27122308271,24sleep00-21swapper/00
25162279241,26sleep10-21swapper/11
292999265257,5cyclictest7420-21munin-run1
26552241205,24sleep30-21swapper/33
292599205199,4cyclictest7458-21sendmail0
293999183178,3cyclictest7457-21sendmail3
3174821540,3sleep0292599cyclictest0
2459421350,5sleep2293499cyclictest2
293499124118,4cyclictest7432-21apt-get2
147132460,3sleep30-21swapper/33
243402420,5sleep2293499cyclictest2
57692400,2sleep30-21swapper/33
324122380,3sleep215748-1kworker/2:1H2
292599319,13cyclictest111rcu_preempt0
2925993113,4cyclictest9-21ksoftirqd/00
308802280,3sleep10-21swapper/11
248262280,1sleep30-21swapper/33
293499272,20cyclictest20349-21apt-get2
292599277,7cyclictest9-21ksoftirqd/00
2925992715,10cyclictest7141-21idleruntime-cro0
293499265,19cyclictest0-21swapper/22
2934992620,4cyclictest17025-21df2
292599256,9cyclictest9-21ksoftirqd/00
293999246,2cyclictest111rcu_preempt3
293499243,18cyclictest28346-21cron2
2925992421,2cyclictest101rcuc/00
2925992419,3cyclictest101rcuc/00
292599240,8cyclictest9-21ksoftirqd/00
292599240,6cyclictest9-21ksoftirqd/00
188622240,2sleep00-21swapper/00
188622240,2sleep00-21swapper/00
293999232,4cyclictest35-21ksoftirqd/33
2939992316,4cyclictest11763-21sh3
293499231,5cyclictest23512-40ogrotate2
293499231,4cyclictest10333-21gdbus2
292599233,7cyclictest9-21ksoftirqd/00
292599232,7cyclictest9-21ksoftirqd/00
2925992319,2cyclictest101rcuc/00
2925992319,2cyclictest101rcuc/00
2925992318,3cyclictest9-21ksoftirqd/00
2939992216,4cyclictest15334-21cut3
2939992215,5cyclictest14088-21wc3
2939992214,5cyclictest3386-21cat3
293999221,18cyclictest23531-21dump-pmu-power3
293499224,16cyclictest3230-21cat2
2934992217,3cyclictest9967-21munin-run2
2934992216,4cyclictest32095-21sh2
2934992214,5cyclictest28-21ksoftirqd/22
293499221,19cyclictest7951-21perf2
293499221,18cyclictest24485-21date2
293499221,18cyclictest20338-21munin-run2
293499221,18cyclictest17109-21apt-get2
293499221,18cyclictest17109-21apt-get2
2934992210,9cyclictest2632-21cat2
293499220,4cyclictest10333-21gdbus2
292599224,4cyclictest9-21ksoftirqd/00
292599223,10cyclictest9-21ksoftirqd/00
2925992218,3cyclictest101rcuc/00
292599221,7cyclictest9-21ksoftirqd/00
2925992217,3cyclictest9-21ksoftirqd/00
292599221,6cyclictest9-21ksoftirqd/00
292599221,6cyclictest9-21ksoftirqd/00
292599221,6cyclictest9-21ksoftirqd/00
2925992213,2cyclictest9-21ksoftirqd/00
292599220,4cyclictest9-21ksoftirqd/00
292599220,20cyclictest17509-21seq0
293999218,5cyclictest111rcu_preempt3
2939992115,4cyclictest29661-21sed3
2939992115,4cyclictest11395-21ntpq3
2939992115,3cyclictest2082-21systemd-journal3
293499214,16cyclictest8007-21perf2
2934992115,3cyclictest28686-21ntp_offset2
293499211,3cyclictest4824-21munin-run2
293499211,3cyclictest22281-21munin-run2
293499211,18cyclictest8794-21munin-run2
293499211,18cyclictest7073-1kworker/2:0H2
293499211,18cyclictest5052-21apt-get2
293499211,18cyclictest18283-21cron2
293499210,19cyclictest9062-21find2
293499210,18cyclictest18471-21diskstats2
293499210,18cyclictest18471-21diskstats2
293499210,18cyclictest14325-21apt-config2
292599214,8cyclictest9-21ksoftirqd/00
292599213,8cyclictest9-21ksoftirqd/00
292599213,12cyclictest9-21ksoftirqd/00
292599212,6cyclictest0-21swapper/00
292599211,7cyclictest9-21ksoftirqd/00
292599211,6cyclictest9-21ksoftirqd/00
2925992115,4cyclictest9835-21seq0
2925992115,3cyclictest6272-21sh0
262772210,1sleep20-21swapper/22
2939992015,3cyclictest35-21ksoftirqd/33
2939992015,3cyclictest25385-21processes3
2939992014,4cyclictest9803-21sed3
2939992014,4cyclictest10333-21gdbus3
293499202,16cyclictest15769-21latency_hist2
2934992015,3cyclictest32708-21ntp_states2
2934992015,3cyclictest32708-21ntp_states2
2934992015,3cyclictest31897-21switchtime2
2934992015,3cyclictest15502-21pmu-power2
2934992015,3cyclictest1229-21swap2
2934992014,4cyclictest23004-21df2
293499201,3cyclictest3961-21sendmail-mta2
293499201,3cyclictest12849-1kworker/2:3H2
293499201,2cyclictest31543-21munin-run2
293499201,17cyclictest9760-21idleruntime-cro2
293499201,17cyclictest5918-21cat2
293499201,17cyclictest5144-21perf2
293499201,17cyclictest30701-21cat2
293499201,17cyclictest17804-21perf2
293499201,17cyclictest15748-1kworker/2:1H2
293499200,19cyclictest0-21swapper/22
293499200,19cyclictest0-21swapper/22
293499200,18cyclictest12823-21munin-run2
2929992017,2cyclictest0-21swapper/11
2929992016,2cyclictest16348-21dump-pmu-power1
2929992015,3cyclictest31807-21/usr/sbin/munin1
2929992015,3cyclictest31807-21/usr/sbin/munin1
2929992015,3cyclictest3073-21/usr/sbin/munin1
2929992015,3cyclictest29975-21sed1
2929992015,3cyclictest22820-21swap1
2929992015,3cyclictest22574-21/usr/sbin/munin1
2929992014,4cyclictest31983-21sed1
2929992014,4cyclictest2488-21dump-pmu-power1
2929992014,4cyclictest22306-21basename1
2929992014,4cyclictest19632-21sed1
2929992013,4cyclictest8002-21cat1
2929992013,4cyclictest30884-21cpuspeed_turbos1
2929992013,4cyclictest25784-21sed1
2929992013,4cyclictest15258-21grep1
2929992011,5cyclictest2735-21apt-get1
292999200,17cyclictest0-21swapper/11
292599207,4cyclictest9-21ksoftirqd/00
292599205,7cyclictest111rcu_preempt0
292599203,9cyclictest9-21ksoftirqd/00
292599203,6cyclictest9-21ksoftirqd/00
292599202,9cyclictest9-21ksoftirqd/00
292599202,8cyclictest9-21ksoftirqd/00
292599201,6cyclictest9-21ksoftirqd/00
292599201,6cyclictest9-21ksoftirqd/00
292599201,6cyclictest9-21ksoftirqd/00
292599201,6cyclictest0-21swapper/00
2925992015,3cyclictest101rcuc/00
2925992014,4cyclictest19882-21awk0
2925992014,4cyclictest16907-21expr0
2925992014,4cyclictest14169-21seq0
2925992014,3cyclictest21573-21proc_pri0
292599200,8cyclictest9-21ksoftirqd/00
292599200,6cyclictest9-21ksoftirqd/00
6012190,4sleep3293999cyclictest3
6012190,4sleep3293999cyclictest3
293999198,9cyclictest9955-21grep3
2939991915,2cyclictest35-21ksoftirqd/33
2939991915,2cyclictest30918-21timerandwakeup3
2939991915,2cyclictest29437-21switchtime3
2939991914,3cyclictest7997-21/usr/sbin/munin3
2939991914,3cyclictest7015-21ntp_kernel_err3
2939991914,3cyclictest3939-21sed3
2939991914,3cyclictest30718-21expr3
2939991914,3cyclictest27810-21find3
2939991914,3cyclictest26781-21df3
2939991914,3cyclictest20349-21apt-get3
2939991914,3cyclictest16985-21pmu-power3
2939991914,3cyclictest15705-21wc3
2939991914,3cyclictest10241-21/usr/sbin/munin3
2939991913,4cyclictest365-21cat3
2939991913,3cyclictest18438-21grep3
2939991913,3cyclictest18438-21grep3
293999191,16cyclictest0-21swapper/33
2939991911,5cyclictest27575-21basename3
2939991910,3cyclictest35-21ksoftirqd/33
293499192,2cyclictest28-21ksoftirqd/22
2934991914,3cyclictest773-21ntp_kernel_pll_2
2934991914,3cyclictest28110-21sh2
2934991914,3cyclictest26028-21cpuspeed2
2934991914,3cyclictest21300-21latency2
2934991913,4cyclictest8707-21fw_packets2
2934991913,4cyclictest29224-21ntpq2
2934991913,4cyclictest20628-21ntp_kernel_pll_2
293499191,2cyclictest26303-21sh2
293499190,17cyclictest16318-21sendmail-msp2
293499190,17cyclictest14280-21perf2
2929991915,2cyclictest8190-21/usr/sbin/munin1
2929991915,2cyclictest32611-21/usr/sbin/munin1
292999191,4cyclictest0-21swapper/11
2929991914,3cyclictest665-21runrttasks1
2929991914,3cyclictest6482-21expr1
2929991914,3cyclictest3923-21/usr/sbin/munin1
2929991914,3cyclictest3679-21sh1
2929991914,3cyclictest24540-21/usr/sbin/munin1
2929991914,3cyclictest16556-21apt-get1
2929991914,3cyclictest10965-21awk1
2929991913,4cyclictest347-21awk1
2929991913,4cyclictest20368-21basename1
2929991913,4cyclictest16862-21tr1
2929991912,4cyclictest9750-21latency1
2929991912,4cyclictest17744-21uname1
2929991912,4cyclictest17744-21uname1
2929991912,4cyclictest12140-21cat1
292999191,15cyclictest8833-21idleruntime-cro1
292599198,3cyclictest111rcu_preempt0
292599192,6cyclictest9-21ksoftirqd/00
292599191,6cyclictest9-21ksoftirqd/00
2925991914,3cyclictest25816-21apt-get0
2925991914,3cyclictest24880-21cat0
2925991913,3cyclictest29982-21uptime0
2925991912,4cyclictest32597-21cron0
2925991910,2cyclictest111rcu_preempt0
292599190,6cyclictest9-21ksoftirqd/00
292599190,6cyclictest9-21ksoftirqd/00
292599190,6cyclictest9-21ksoftirqd/00
292599190,6cyclictest9-21ksoftirqd/00
292599190,6cyclictest0-21swapper/00
2939991814,2cyclictest24089-21cron3
2939991814,2cyclictest18460-21df3
2939991814,2cyclictest18460-21df3
2939991814,2cyclictest11056-21/usr/sbin/munin3
2939991813,3cyclictest8390-21aten2.4-expect3
2939991813,3cyclictest7432-21cat3
2939991813,3cyclictest32376-21seq3
2939991813,3cyclictest31031-21diskstats3
2939991813,3cyclictest29390-21missed_timers3
2939991813,3cyclictest2082-21systemd-journal3
2939991813,3cyclictest18340-21cron3
2939991813,3cyclictest15652-21seq3
2939991812,4cyclictest6872-21cat3
2939991812,4cyclictest16848-21grep3
2939991812,4cyclictest13195-21cut3
2939991812,3cyclictest2082-21systemd-journal3
2939991811,4cyclictest505-21dbus-daemon3
2939991811,4cyclictest20934-21expr3
293999180,15cyclictest26838-21sort3
293499188,8cyclictest665-21runrttasks2
2934991814,2cyclictest5380-21apt-get2
293499180,3cyclictest15748-1kworker/2:1H2
292999182,10cyclictest22539-21apt-get1
2929991816,1cyclictest0-21swapper/11
2929991816,1cyclictest0-21swapper/11
2929991814,2cyclictest19755-21/usr/sbin/munin1
2929991813,3cyclictest6284-21sed1
2929991813,3cyclictest5481-21aten2_r9power_v1
2929991813,3cyclictest3038-21ntp_kernel_pll_1
2929991813,3cyclictest28935-21grep1
2929991813,3cyclictest21380-21gzip1
2929991813,3cyclictest10503-21/usr/sbin/munin1
2929991812,4cyclictest665-21runrttasks1
2929991812,4cyclictest665-21runrttasks1
2929991812,4cyclictest29392-21gzip1
2929991812,4cyclictest21073-21grep1
2929991811,4cyclictest505-21dbus-daemon1
2929991811,4cyclictest18284-21cpuspeed_turbos1
2929991811,4cyclictest18284-21cpuspeed_turbos1
2925991812,3cyclictest32381-21cat0
2925991812,3cyclictest32381-21cat0
2939991715,1cyclictest0-21swapper/33
2939991714,2cyclictest35-21ksoftirqd/33
2939991712,3cyclictest3772-21latency3
2939991712,3cyclictest2680-21awk3
2939991711,4cyclictest19821-21cat3
2939991710,4cyclictest665-21runrttasks3
2934991711,4cyclictest15023-21cut2
292999172,9cyclictest14516-21apt-get1
2929991713,2cyclictest779-21sendmail-mta1
2929991713,2cyclictest26338-21/usr/sbin/munin1
2929991713,2cyclictest12001-21/usr/sbin/munin1
2929991712,3cyclictest665-21runrttasks1
2929991712,3cyclictest665-21runrttasks1
2929991712,3cyclictest665-21runrttasks1
2929991712,3cyclictest27461-21ntpq1
2929991711,4cyclictest4587-21cat1
2929991711,3cyclictest7388-21timerandwakeup1
2929991711,3cyclictest665-21runrttasks1
2929991710,4cyclictest10312-21cut1
2939991611,3cyclictest2833-21sed3
293999160,1cyclictest0-21swapper/33
293999160,15cyclictest0-21swapper/33
293499163,8cyclictest10259-21apt-get2
292999160,1cyclictest0-21swapper/11
2929991510,3cyclictest23938-21cut1
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional