OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rackeslot4s.osadl.org (updated Fri Apr 11, 2025 12:44:05)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
450174	99	48	41,4	cyclictest	581694	-21	latency_hist	09:15:00	2
450174	99	39	32,4	cyclictest	539617	-21	latency_hist	08:35:01	2
450179	99	37	31,4	cyclictest	712931	-21	latency_hist	11:55:00	3
450171	99	36	33,2	cyclictest	691050	-21	kworker/u8:4+flush-259:0	11:30:00	1
450179	99	33	30,2	cyclictest	647754	-21	kworker/u8:3+flush-259:0	10:40:16	3
450179	99	31	24,5	cyclictest	652712	-21	latency_hist	10:40:01	3
450174	99	31	25,4	cyclictest	598143	-21	/usr/sbin/munin	09:35:16	2
450166	99	29	23,4	cyclictest	740892	-21	apt-get	12:30:07	0
450171	99	28	24,3	cyclictest	669212	-21	apt-get	11:00:13	1
450174	99	27	22,3	cyclictest	565905	-21	apt-get	09:00:08	2
450179	99	26	22,3	cyclictest	465846	-21	kworker/u8:0+flush-259:0	08:25:18	3
450174	99	26	20,5	cyclictest	481388	-21	apt-get	07:40:00	2
450166	99	26	21,4	cyclictest	3453141	-21	kworker/u8:3+events_unbound	07:25:27	0
450166	99	26	20,4	cyclictest	729148	-21	latency_hist	12:15:01	0
450171	99	25	21,3	cyclictest	702110	-21	apt-get	11:40:12	1
450166	99	25	21,3	cyclictest	647754	-21	kworker/u8:3+flush-259:0	10:40:17	0
450166	99	25	19,4	cyclictest	454949	-21	kworker/u8:4+events_unbound	07:15:19	0
450174	99	24	19,3	cyclictest	652952	-21	/usr/sbin/munin	10:40:14	2
450171	99	24	20,3	cyclictest	471279	-21	apt-get	07:30:15	1
450166	99	24	20,3	cyclictest	570828	-21	kworker/u8:0+events_unbound	09:25:00	0
450166	99	24	17,5	cyclictest	3000860	-21	systemd-journal	07:40:00	0
450174	99	23	17,4	cyclictest	3000063	-21	rs:main		2
450171	99	23	18,3	cyclictest	3000860	-21	systemd-journal	08:11:26	1
450166	99	23	1,20	cyclictest	1	-21	systemd	09:50:00	0
450174	99	22	16,4	cyclictest	3000860	-21	systemd-journal	10:23:26	2
450171	99	22	18,3	cyclictest	576050	-21	http	09:10:00	1
450171	99	22	18,2	cyclictest	734778	-21	latency_hist	12:20:00	1
450179	99	21	15,4	cyclictest	592101	-21	latency_hist	09:25:00	3
450174	99	21	14,5	cyclictest	533482	-21	ntpq	08:25:21	2
450171	99	21	17,3	cyclictest	636274	-21	/usr/sbin/munin	10:20:14	1
450171	99	21	14,5	cyclictest	630819	-21	latency_hist	10:15:01	1
450166	99	21	17,3	cyclictest	534420	-21	kworker/u8:1+events_unbound	08:45:01	0
450166	99	21	16,4	cyclictest	502738	-21	kworker/u8:2+events_unbound	08:05:12	0
450166	99	21	16,2	cyclictest	647754	-21	kworker/u8:3+events_unbound	10:50:14	0
450166	99	21	15,5	cyclictest	465846	-21	kworker/u8:0+events_unbound	08:25:14	0
450166	99	21	14,4	cyclictest	3000860	-21	systemd-journal	12:27:26	0
450166	99	21	13,5	cyclictest	689968	-21	ntpq	11:20:23	0
450179	99	20	16,3	cyclictest	625210	-21	apt-get	10:10:06	3
450166	99	20	16,3	cyclictest	3453141	-21	kworker/u8:3+flush-259:0	07:31:31	0
450166	99	20	16,3	cyclictest	299417	-21	kworker/u8:1+events_unbound	07:20:20	0
450179	99	19	14,3	cyclictest	739985	-21	latency_hist	12:25:00	3
450179	99	19	14,3	cyclictest	570828	-21	kworker/u8:0+flush-259:0	10:10:00	3
450179	99	19	10,8	cyclictest	492494	-21	kworker/u8:3+flush-259:0	07:50:26	3
450174	99	19	14,3	cyclictest	3453141	-21	kworker/u8:3+flush-259:0	07:25:00	2
450174	99	19	13,4	cyclictest	3000063	-21	rs:main		2
450171	99	19	16,2	cyclictest	619220	-21	kworker/u8:1+flush-259:0	11:00:02	1
450171	99	19	16,2	cyclictest	3453141	-21	kworker/u8:3+flush-259:0	07:35:25	1
450166	99	19	16,2	cyclictest	550237	-21	kworker/u8:3+events_unbound	09:28:28	0
450166	99	19	15,3	cyclictest	619220	-21	kworker/u8:1+events_unbound	11:05:01	0
450166	99	19	15,3	cyclictest	534420	-21	kworker/u8:1+flush-259:0	08:45:16	0
450166	99	19	15,3	cyclictest	476200	-21	kworker/u8:2+events_unbound	07:40:22	0
450166	99	19	15,3	cyclictest	465846	-21	kworker/u8:0+events_unbound	08:15:18	0
450166	99	19	13,4	cyclictest	729658	-21	cat	12:15:00	0
450166	99	19	13,4	cyclictest	625212	-21	kworker/u8:2+events_unbound	10:35:21	0
450179	99	18	14,3	cyclictest	576029	-21	apt-get	09:10:00	3
450179	99	18	12,4	cyclictest	668941	-21	latency_hist	11:00:01	3
450174	99	18	15,2	cyclictest	647754	-21	kworker/u8:3+events_unbound	11:15:16	2
450174	99	18	12,4	cyclictest	707949	-21	kworker/u8:1+flush-259:0	11:55:27	2
450174	99	18	12,4	cyclictest	3000860	-21	systemd-journal	11:14:26	2
450171	99	18	14,3	cyclictest	647766	-21	/usr/sbin/munin	10:35:18	1
450171	99	18	12,4	cyclictest	603050	-21	latency_hist	09:40:01	1
450171	99	18	12,4	cyclictest	555241	-21	latency_hist	08:50:00	1
450166	99	18	14,3	cyclictest	691050	-21	kworker/u8:4+events_unbound	12:04:29	0
450166	99	18	14,3	cyclictest	691050	-21	kworker/u8:4+events_unbound	12:04:28	0
450166	99	18	14,3	cyclictest	603278	-21	kworker/u8:2+events_unbound	09:50:10	0
450166	99	18	14,3	cyclictest	465846	-21	kworker/u8:0+events_unbound	08:40:01	0
450166	99	18	14,2	cyclictest	534420	-21	kworker/u8:1+flush-259:0	08:50:27	0
450179	99	17	14,2	cyclictest	534420	-21	kworker/u8:1+flush-259:0	08:50:22	3
450179	99	17	14,2	cyclictest	465846	-21	kworker/u8:0+flush-259:0	07:55:24	3
450179	99	17	12,4	cyclictest	691050	-21	kworker/u8:4+flush-259:0	11:30:25	3
450179	99	17	12,3	cyclictest	3000860	-21	systemd-journal	11:42:27	3
450179	99	17	11,4	cyclictest	466060	-21	/usr/sbin/munin	07:25:14	3
450174	99	17	11,4	cyclictest	641856	-21	apt-get	10:30:01	2
450171	99	17	14,2	cyclictest	619220	-21	kworker/u8:1+flush-259:0	11:15:25	1
450171	99	17	12,3	cyclictest	3000860	-21	systemd-journal	11:40:01	1
450166	99	17	14,2	cyclictest	647754	-21	kworker/u8:3+events_unbound	11:10:18	0
450166	99	17	14,2	cyclictest	613743	-21	kworker/u8:4+flush-259:0	10:10:10	0
450166	99	17	14,2	cyclictest	465846	-21	kworker/u8:0+events_unbound	08:00:27	0
450166	99	17	13,3	cyclictest	707949	-21	kworker/u8:1+events_unbound	11:59:27	0
450166	99	17	13,3	cyclictest	576686	-21	kworker/u8:4+events_unbound	09:10:20	0
450166	99	17	13,3	cyclictest	550237	-21	kworker/u8:3+flush-259:0	09:30:20	0
450179	99	16	12,3	cyclictest	534420	-21	kworker/u8:1+flush-259:0	08:45:26	3
450179	99	16	12,3	cyclictest	465846	-21	kworker/u8:0+flush-259:0	08:10:21	3
450179	99	16	11,3	cyclictest	674794	-21	http	11:10:01	3
450174	99	16	13,2	cyclictest	492942	-21	irqcore	07:50:19	2
450174	99	16	12,3	cyclictest	534420	-21	kworker/u8:1+flush-259:0	08:50:26	2
450171	99	16	12,3	cyclictest	518973	-21	gpgv	08:15:06	1
450171	99	16	11,4	cyclictest	3000860	-21	systemd-journal	10:08:26	1
450166	99	16	14,1	cyclictest	606	50	irq/133-enp0s29f1:rx-0	11:00:17	0
450166	99	16	13,2	cyclictest	723359	-21	kworker/u8:2+events_unbound	12:35:21	0
450166	99	16	13,2	cyclictest	697359	-21	irqcore	11:35:18	0
450166	99	16	12,3	cyclictest	613743	-21	kworker/u8:4+events_unbound	10:22:27	0
450166	99	16	12,3	cyclictest	570828	-21	kworker/u8:0+events_unbound	10:05:27	0
450166	99	16	12,3	cyclictest	570828	-21	kworker/u8:0+events_unbound	09:55:14	0
450166	99	16	12,3	cyclictest	465846	-21	kworker/u8:0+flush-259:0	07:46:31	0
450166	99	16	12,3	cyclictest	299417	-21	kworker/u8:1+events_unbound	07:53:27	0
450166	99	16	10,4	cyclictest	706248	-21	ntpq	11:40:22	0
450179	99	15	9,4	cyclictest	564803	-21	ntpq	08:55:21	3
450179	99	15	12,2	cyclictest	690846	-21	kworker/u8:3+flush-259:0	11:46:31	3
450179	99	15	12,2	cyclictest	576686	-21	kworker/u8:4+flush-259:0	09:17:31	3
450179	99	15	12,2	cyclictest	465846	-21	kworker/u8:0+flush-259:0	07:34:32	3
450174	99	15	1,8	cyclictest	502696	-21	cron	08:00:00	2
450174	99	15	11,3	cyclictest	508217	-21	apt-get	08:05:11	2
450174	99	15	11,3	cyclictest	299417	-21	kworker/u8:1+flush-259:0	07:29:59	2
450174	99	15	11,2	cyclictest	576056	-21	gpgv	09:10:00	2
450174	99	15	10,4	cyclictest	647754	-21	kworker/u8:3+flush-259:0	11:00:18	2
450171	99	15	9,4	cyclictest	642167	-21	base64	10:30:01	1
450171	99	15	12,2	cyclictest	691050	-21	kworker/u8:4+flush-259:0	11:55:18	1
450171	99	15	12,2	cyclictest	662864	-21	ntp_kernel_pll_	10:50:18	1
450171	99	15	12,2	cyclictest	591238	-21	ntp_kernel_pll_	09:20:23	1
450166	99	15	13,1	cyclictest	606	50	irq/133-enp0s29f1:rx-0	09:38:23	0
450166	99	15	12,2	cyclictest	614	50	irq/134-enp0s29f1:tx-0	09:05:15	0
450166	99	15	12,2	cyclictest	606	50	irq/133-enp0s29f1:rx-0	11:48:45	0
450166	99	15	11,3	cyclictest	707979	-21	apt-get	11:50:08	0
450166	99	15	11,3	cyclictest	641173	-21	kworker/u8:0+events_unbound	10:30:17	0
450166	99	15	11,3	cyclictest	550237	-21	kworker/u8:3+events_unbound	08:55:11	0
450179	99	14	9,4	cyclictest	685496	-21	kworker/u8:2+flush-259:0	11:20:25	3
450179	99	14	11,2	cyclictest	523551	-21	unixbench_multi	08:15:25	3
450179	99	14	10,2	cyclictest	3000860	-21	systemd-journal	09:45:00	3
450179	99	14	10,2	cyclictest	3000860	-21	systemd-journal	09:45:00	3
450174	99	14	9,3	cyclictest	3000860	-21	systemd-journal	07:10:25	2
450174	99	14	11,2	cyclictest	735458	-21	irqcore	12:20:19	2
450174	99	14	10,3	cyclictest	582361	-21	irqcore	09:15:16	2
450174	99	14	10,3	cyclictest	486543	-21	unixbench_multi	07:40:26	2
450171	99	14	9,3	cyclictest	508192	-21	/usr/sbin/munin	08:05:14	1
450171	99	14	11,2	cyclictest	723962	-21	idleruntime-cro	12:10:00	1
450171	99	14	11,2	cyclictest	707949	-21	kworker/u8:1+flush-259:0	12:10:25	1
450166	99	14	12,1	cyclictest	606	50	irq/133-enp0s29f1:rx-0	07:13:55	0
450166	99	14	11,2	cyclictest	692752	-21	kthreadcore	11:30:22	0
450166	99	14	11,2	cyclictest	647754	-21	kworker/u8:3+events_unbound	10:55:17	0
450166	99	14	11,2	cyclictest	621927	-21	kthreadcore	10:00:18	0
450166	99	14	11,2	cyclictest	606	50	irq/133-enp0s29f1:rx-0	12:06:08	0
450166	99	14	11,2	cyclictest	606	50	irq/133-enp0s29f1:rx-0	09:02:32	0
450166	99	14	10,3	cyclictest	690846	-21	kworker/u8:3+events_unbound	11:26:28	0
450179	99	13	9,3	cyclictest	701433	-21	unixbench_singl	11:35:25	3
450179	99	13	10,2	cyclictest	713265	-21	cpuspeed_turbos	11:55:15	3
450179	99	13	10,2	cyclictest	691050	-21	kworker/u8:4+flush-259:0	12:15:24	3
450179	99	13	10,2	cyclictest	673700	-21	unixbench_singl	11:00:27	3
450179	99	13	10,2	cyclictest	642957	-21	irqcore	10:30:12	3
450179	99	13	10,2	cyclictest	547375	-21	kthreadcore	08:40:20	3
450179	99	13	10,2	cyclictest	481394	-21	sh	07:40:00	3
450174	99	13	10,2	cyclictest	603543	-21	fschecks_time	09:40:21	2
450174	99	13	10,2	cyclictest	603543	-21	fschecks_time	09:40:21	2
450174	99	13	10,2	cyclictest	491839	-21	unixbench_singl	07:45:25	2
450171	99	13	9,3	cyclictest	593006	-21	apt-get	09:30:16	1
450171	99	13	9,3	cyclictest	3000860	-21	systemd-journal	10:49:55	1
450171	99	13	11,1	cyclictest	0	-21	swapper/1	09:00:00	1
450166	99	13	9,3	cyclictest	641173	-21	kworker/u8:0+events_unbound	10:25:02	0
450166	99	13	9,3	cyclictest	465846	-21	kworker/u8:0+events_unbound	08:22:28	0
450166	99	13	8,4	cyclictest	723359	-21	kworker/u8:2+events_unbound	12:20:26	0
450166	99	13	11,1	cyclictest	606	50	irq/133-enp0s29f1:rx-0	10:15:19	0
450166	99	13	11,1	cyclictest	606	50	irq/133-enp0s29f1:rx-0	09:40:24	0
450166	99	13	11,1	cyclictest	606	50	irq/133-enp0s29f1:rx-0	09:40:24	0
450179	99	12	9,2	cyclictest	726544	-21	tr	12:10:26	3
450179	99	12	9,2	cyclictest	602625	-21	unixbench_singl	09:35:27	3
450179	99	12	7,4	cyclictest	492251	-21	cron	07:50:00	3
450179	99	12	6,4	cyclictest	455321	-21	latency_hist	07:15:01	3
450174	99	12	9,2	cyclictest	648212	-21	sed	10:35:20	2
450174	99	12	9,2	cyclictest	465846	-21	kworker/u8:0+flush-259:0	08:21:56	2
450171	99	12	9,2	cyclictest	635563	-21	unixbench_multi	10:15:28	1
450171	99	12	1,9	cyclictest	460570	-21	awk	07:20:00	1
450171	99	12	1,3	cyclictest	608905	-21	awk	09:50:00	1
450171	99	12	10,1	cyclictest	544802	-21	cron	08:40:01	1
450171	99	12	10,1	cyclictest	506976	-21	needreboot	08:00:19	1
450166	99	12	9,2	cyclictest	465846	-21	kworker/u8:0+flush-259:0	07:55:20	0
450179	99	11	9,1	cyclictest	0	-21	swapper/3	09:34:59	3
450179	99	11	8,2	cyclictest	524701	-21	irqcore	08:20:17	3
450179	99	11	8,2	cyclictest	465846	-21	kworker/u8:0+flush-259:0	08:35:01	3
450179	99	11	8,2	cyclictest	0	-21	swapper/3	10:00:14	3
450179	99	11	7,3	cyclictest	707949	-21	kworker/u8:1+flush-259:0	12:00:30	3
450179	99	11	7,3	cyclictest	707949	-21	kworker/u8:1+flush-259:0	12:00:30	3
450179	99	11	7,2	cyclictest	662818	-21	ntpq	10:50:16	3
450179	99	11	6,4	cyclictest	691549	-21	sed	11:30:00	3
450179	99	11	0,10	cyclictest	740629	-21	apt	12:30:00	3
450174	99	11	6,4	cyclictest	3000860	-21	systemd-journal	09:00:00	2
450174	99	11	6,3	cyclictest	3000860	-21	systemd-journal	08:15:00	2
450171	99	11	9,1	cyclictest	0	-21	swapper/1	12:25:00	1
450171	99	11	8,2	cyclictest	618533	-21	ntpq	09:55:21	1
450171	99	11	7,2	cyclictest	3000860	-21	systemd-journal	08:25:01	1
450171	99	11	6,4	cyclictest	497474	-21	awk	07:54:59	1
450166	99	11	8,2	cyclictest	619220	-21	kworker/u8:1+flush-259:0	11:15:24	0
450166	99	11	6,3	cyclictest	550237	-21	kworker/u8:3+flush-259:0	09:15:20	0
450179	99	10	8,1	cyclictest	0	-21	swapper/3	09:55:15	3
450179	99	10	7,2	cyclictest	640391	-21	ntpq	10:20:21	3
450174	99	10	7,2	cyclictest	723971	-21	awk	12:10:00	2
450174	99	10	7,2	cyclictest	664120	-21	cpuspeed_turbos	10:55:14	2
450174	99	10	7,2	cyclictest	0	-21	swapper/2	08:05:00	2
450171	99	10	6,3	cyclictest	476200	-21	kworker/u8:2+flush-259:0	07:40:26	1
450166	99	10	0,2	cyclictest	658524	-21	latency_hist	10:50:01	0
450179	99	9	5,2	cyclictest	507949	-21	latency_hist	08:05:01	3
450179	99	9	1,6	cyclictest	674780	-21	munin-run	11:09:59	3
450174	99	9	5,3	cyclictest	457116	-21	tr	07:15:19	2
450174	99	9	0,8	cyclictest	625207	-21	latency_hist	10:10:01	2
450174	99	9	0,8	cyclictest	608901	-21	latency_hist	09:50:00	2
450174	99	8	2,2	cyclictest	691472	-21	awk	11:30:01	2
450174	99	8	0,6	cyclictest	592746	-21	munin-run	09:30:00	2
450171	99	8	1,5	cyclictest	3000860	-21	systemd-journal	11:09:59	1
450179	99	7	5,1	cyclictest	0	-21	swapper/3	10:30:00	3
450174	99	7	1,4	cyclictest	707068	-21	cron	11:45:00	2
450174	99	7	1,4	cyclictest	674765	-21	apt-get	11:09:59	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:34