You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-11 - 12:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Tue Mar 11, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
281126599753748,4cyclictest2507838-21kworker/2:1+events19:54:422
281126799746741,4cyclictest2951931-21kworker/3:2+events21:37:103
281126599743738,4cyclictest2507838-21kworker/2:1+events23:56:062
281126599742738,3cyclictest2507838-21kworker/2:1+events00:03:352
281126599741736,4cyclictest2507838-21kworker/2:1+events19:13:052
281126599741735,5cyclictest2507838-21kworker/2:1+events00:09:342
281126799737733,3cyclictest2951931-21kworker/3:2+events22:01:593
281126799737732,4cyclictest2908008-21kworker/3:1+events21:00:233
281126799737732,4cyclictest2766994-21kworker/3:1+events19:56:583
281126799732729,2cyclictest2876788-21kworker/3:2+events20:33:003
281126799732727,4cyclictest3048726-21kworker/3:2+events22:58:133
281126799729725,3cyclictest2908008-21kworker/3:1+events20:41:593
281126799728724,3cyclictest2951931-21kworker/3:2+events22:12:203
281126799725721,3cyclictest2951931-21kworker/3:2+events21:14:233
281126799724720,3cyclictest2951931-21kworker/3:2+events21:33:463
281126799724719,4cyclictest2951931-21kworker/3:2+events21:42:403
281126799724719,4cyclictest2876788-21kworker/3:2+events20:06:383
281126799723719,3cyclictest3067680-21kworker/3:1+events00:38:573
281126799723719,3cyclictest2908008-21kworker/3:1+events20:49:293
281126799722719,2cyclictest3067680-21kworker/3:1+events23:39:283
281126799720713,6cyclictest3067680-21kworker/3:1+events00:30:203
281126799719716,2cyclictest2766994-21kworker/3:1+events19:31:053
281126799718707,9cyclictest3067680-21kworker/3:1+events23:20:223
281126799717714,2cyclictest3067680-21kworker/3:1+events23:26:463
281126799717714,2cyclictest2951931-21kworker/3:2+events22:05:453
281126799717714,2cyclictest2951931-21kworker/3:2+events21:16:063
281126799717713,3cyclictest2951931-21kworker/3:2+events22:26:223
281126799717713,3cyclictest2951931-21kworker/3:2+events21:45:213
281126799717713,3cyclictest2908008-21kworker/3:1+events21:05:433
281126799717713,3cyclictest2908008-21kworker/3:1+events21:05:433
281126799717713,3cyclictest2876788-21kworker/3:2+events20:27:403
281126799717713,3cyclictest2766994-21kworker/3:1+events19:49:203
281126799717709,5cyclictest2951931-21kworker/3:2+events21:27:433
281126799717708,5cyclictest2951931-21kworker/3:2+events21:51:363
281126799716713,2cyclictest3067680-21kworker/3:1+events00:25:233
281126799716712,3cyclictest3036144-21kworker/3:0+events22:44:353
281126799716712,3cyclictest2951931-21kworker/3:2+events22:31:263
281126799716712,3cyclictest2951931-21kworker/3:2+events22:22:093
281126799716712,3cyclictest2766994-21kworker/3:1+events20:00:543
281126799715711,3cyclictest2908008-21kworker/3:1+events20:39:103
281126799715711,3cyclictest2766994-21kworker/3:1+events19:27:423
281126799715705,3cyclictest3067680-21kworker/3:1+events23:51:263
281126799714711,2cyclictest3067680-21kworker/3:1+events23:06:503
281126799714711,2cyclictest3048726-21kworker/3:2+events22:51:253
281126799714710,3cyclictest2908008-21kworker/3:1+events20:51:503
281126799714710,3cyclictest2766994-21kworker/3:1+events19:15:433
281126799713710,2cyclictest3067680-21kworker/3:1+events23:02:593
281126799713710,2cyclictest2766994-21kworker/3:1+events19:38:473
281126799713709,3cyclictest2876788-21kworker/3:2+events20:16:553
281126799713708,4cyclictest3067680-21kworker/3:1+events23:30:223
281126799712709,2cyclictest3067680-21kworker/3:1+events23:44:293
281126799712708,3cyclictest3067680-21kworker/3:1+events23:18:283
281126799712708,3cyclictest3067680-21kworker/3:1+events23:18:283
281126799712708,3cyclictest3067680-21kworker/3:1+events00:23:513
281126799711708,2cyclictest2908008-21kworker/3:1+events20:59:303
281126799710707,2cyclictest3067680-21kworker/3:1+events00:13:153
281126799710707,2cyclictest2766994-21kworker/3:1+events19:42:363
281126799709706,2cyclictest3067680-21kworker/3:1+events00:19:513
281126799709706,2cyclictest2951931-21kworker/3:2+events22:18:053
281126799709706,2cyclictest2876788-21kworker/3:2+events20:11:273
281126799709703,5cyclictest3036144-21kworker/3:0+events22:38:523
281126799708705,2cyclictest3067680-21kworker/3:1+events23:11:073
281126799708703,4cyclictest3067680-21kworker/3:1+events23:48:163
281126799707703,3cyclictest2951931-21kworker/3:2+events21:55:423
281126799707703,3cyclictest2951931-21kworker/3:2+events21:20:233
281126799707703,3cyclictest2876788-21kworker/3:2+events20:21:053
281126799706702,3cyclictest2766994-21kworker/3:1+events19:24:093
281126799706701,4cyclictest3048726-21kworker/3:2+events22:47:303
281126799701696,4cyclictest2811125-21turbostat19:14:553
281126799692687,4cyclictest3125831-21turbostat00:09:553
281126799691688,2cyclictest2851952-21turbostat19:54:563
281126599683679,3cyclictest3107180-21turbostat23:54:552
281126599660657,2cyclictest2851322-21turbostat19:49:562
31072982310,1sleep03107301-21sort23:50:010
2811265992219,2cyclictest3029529-21latency_hist22:25:002
2811260992219,2cyclictest3614154-21systemd-journal23:20:001
2811260992219,2cyclictest3614154-21systemd-journal23:20:001
2811260992216,4cyclictest2990831-21cron21:50:001
2811260992118,2cyclictest0-21swapper/100:00:001
2811257992117,3cyclictest506-21in:imuxsock19:45:010
2811260992017,2cyclictest3128178-21kthreadcore00:10:121
281125799209,9cyclictest0-21swapper/020:30:000
2811257992016,3cyclictest2889502-21cron20:20:000
2811265991916,2cyclictest493-21dbus-daemon20:40:002
2811265991916,2cyclictest3039575-21kthreadcore22:35:192
2811265991916,2cyclictest2970923-21sh21:25:002
2811260991915,2cyclictest0-21swapper/121:35:001
2811257991916,2cyclictest3022951-21turbostat22:20:010
2811257991916,2cyclictest2851952-21turbostat19:50:010
2811265991815,2cyclictest3036285-21munin-run22:35:002
2811265991814,3cyclictest2958985-21gltestperf21:15:142
2811265991813,4cyclictest0-21swapper/221:10:002
2811265991813,4cyclictest0-21swapper/221:10:002
2811265991812,4cyclictest2897033-21sshd20:25:182
2811260991816,1cyclictest0-21swapper/123:00:001
2811260991815,2cyclictest3138877-21idleruntime-cro00:20:011
2811257991815,2cyclictest2946033-21cron21:05:000
2811257991815,2cyclictest2811251-21cyclictest19:15:010
2811257991814,3cyclictest3144653-21unixbench_multi00:20:250
2811257991814,3cyclictest3028747-21unixbench_singl22:20:230
2811265991714,2cyclictest3094505-21cpuspeed_turbos23:35:122
2811265991714,2cyclictest3016212-21unixbench_multi22:10:192
2811260991715,1cyclictest0-21swapper/121:20:001
2811260991714,2cyclictest2873-21cron22:05:011
281126099171,15cyclictest0-21swapper/123:10:001
2811257991714,2cyclictest3058896-21kthreadcore22:55:190
2811257991714,2cyclictest2985705-21tr21:40:150
2811257991714,2cyclictest0-21swapper/021:30:000
2811257991714,2cyclictest0-21swapper/020:14:590
2811257991713,3cyclictest3120005-21cpuspeed_turbos00:00:130
2811265991613,2cyclictest3067593-21unixbench_multi23:00:252
2811265991613,2cyclictest3003561-21sendmail22:00:012
2811265991613,2cyclictest2957973-21unixbench_multi21:10:262
2811265991613,2cyclictest2939803-21sendmail21:00:002
2811265991612,3cyclictest3016923-21find22:15:142
2811265991612,3cyclictest2990200-21cron21:45:002
2811265991610,4cyclictest1-21systemd19:30:012
2811260991613,2cyclictest3113361-21sh23:55:001
2811260991613,2cyclictest3095340-21irqrtprio23:35:151
2811260991612,3cyclictest3036625-21cpuspeed_turbos22:35:141
2811260991612,3cyclictest2831556-21unixbench_singl19:20:251
281126099161,13cyclictest3023223-21apt-get22:20:111
281126099160,15cyclictest2916630-21cut20:40:201
2811257991613,2cyclictest3110434-21kthreadcore23:50:130
2811257991613,2cyclictest3078160-21kthreadcore23:15:180
2811257991613,2cyclictest3078160-21kthreadcore23:15:180
2811257991613,2cyclictest0-21swapper/023:35:000
2811257991613,2cyclictest0-21swapper/020:45:000
2811257991612,3cyclictest2977358-21unixbench_singl21:30:190
2811257991612,3cyclictest2845085-21cron19:40:000
2811257991610,5cyclictest0-21swapper/000:35:210
30500562150,1sleep23050639-21sed22:50:122
281126599159,4cyclictest3106955-21sshd23:47:192
281126599153,2cyclictest0-21swapper/223:05:242
2811265991512,2cyclictest2915405-21irqcore20:40:192
2811265991512,2cyclictest0-21swapper/223:35:002
2811265991511,2cyclictest0-21swapper/223:20:002
2811265991511,2cyclictest0-21swapper/223:20:002
2811260991512,2cyclictest3122885-21kthreadcore00:00:181
2811260991512,2cyclictest2970455-21unixbench_multi21:20:251
2811260991512,2cyclictest0-21swapper/122:55:001
2811260991511,3cyclictest3144047-21grep00:20:201
2811260991511,3cyclictest3002558-21needreboot21:55:201
2811260991511,3cyclictest2978475-21irqcore21:35:161
2811257991512,2cyclictest3010780-21cpuspeed_turbos22:10:090
2811257991512,2cyclictest0-21swapper/020:55:160
2811257991512,2cyclictest0-21swapper/000:10:000
2811257991511,3cyclictest0-21swapper/020:50:000
2811257991510,4cyclictest3614154-21systemd-journal20:24:180
2811267991410,2cyclictest0-21swapper/300:00:003
281126599149,4cyclictest3151211-21memory00:30:212
281126599149,4cyclictest2870558-21sshd20:04:172
281126599149,3cyclictest2971936-21grep21:30:092
2811265991412,1cyclictest499-21polkitd21:35:212
2811265991412,1cyclictest2889514-21idleruntime-cro20:20:002
2811265991411,2cyclictest64-21kcompactd020:50:242
2811265991411,2cyclictest3043143-21gltestperf22:40:182
2811265991411,2cyclictest2817755-21gltestperf19:15:012
2811265991410,3cyclictest2889878-21cpuspeed_turbos20:20:172
2811265991410,2cyclictest3061391-21grep22:55:252
2811265991410,2cyclictest0-21swapper/200:30:002
281126099146,1cyclictest0-21swapper/119:20:001
2811260991411,2cyclictest502-21systemd-logind21:00:001
2811260991411,2cyclictest3072057-21kthreadcore23:10:121
2811260991411,2cyclictest2958037-21unixbench_singl21:10:251
2811260991411,2cyclictest2811622-21fschecks_count19:10:151
2811260991411,2cyclictest0-21swapper/121:45:001
2811260991410,3cyclictest3107167-21cron23:50:001
2811260991410,3cyclictest3081488-21cpuspeed_turbos23:20:171
2811260991410,3cyclictest2896079-21cpuspeed_turbos20:25:141
2811260991410,3cyclictest2838445-21unixbench_multi19:30:201
281126099140,2cyclictest2920921-21apt-get20:45:011
281125799149,4cyclictest2864402-21cron20:00:010
2811257991411,2cyclictest0-21swapper/000:15:010
2811257991410,3cyclictest2965054-21cpuspeed_turbos21:20:140
281125799140,13cyclictest3116900-21grep23:55:180
281126599139,3cyclictest2926681-21python320:45:242
281126599138,4cyclictest3614154-21systemd-journal19:45:002
281126599138,4cyclictest3158237-21seq00:37:022
281126599138,4cyclictest3074021-21awk23:10:162
281126599138,4cyclictest2839311-21df19:35:132
2811265991310,2cyclictest502-21systemd-logind20:00:012
2811265991310,2cyclictest3081533-21cpuspeed_turbos23:20:172
2811265991310,2cyclictest0-21swapper/221:30:002
281126099139,3cyclictest311ktimers/120:30:141
281126099138,3cyclictest2889103-21grep20:15:271
281126099138,3cyclictest0-21swapper/121:00:141
281126099136,5cyclictest523-21gdbus22:50:001
281126099133,2cyclictest295798rtkit-daemon23:05:001
2811260991310,2cyclictest2991081-21/usr/sbin/munin21:50:131
2811260991310,2cyclictest0-21swapper/122:45:001
2811260991310,2cyclictest0-21swapper/120:05:011
2811260991310,2cyclictest0-21swapper/119:50:011
2811260991310,2cyclictest0-21swapper/100:40:001
281125799139,3cyclictest0-21swapper/021:55:010
281125799138,4cyclictest2952241-21cron21:10:000
281125799138,4cyclictest2952241-21cron21:10:000
281125799137,5cyclictest21750irq/159-ahci[0000:00:17.0]19:55:010
2811257991311,1cyclictest2876874-21kworker/0:3+kacpid23:20:220
2811257991310,2cyclictest509-21rs:main0
2811257991310,2cyclictest3042901-21cpuspeed_turbos22:40:160
2811257991310,2cyclictest2811009-21kworker/0:2+kacpid19:15:140
2811257991310,2cyclictest0-21swapper/023:40:000
2811257991310,2cyclictest0-21swapper/020:05:180
2811257991310,2cyclictest0-21swapper/020:05:010
281125799130,11cyclictest0-21swapper/000:25:110
28052102132,6sleep20-21swapper/219:05:112
295798120,3rtkit-daemon728-21snmpd19:05:081
281126599129,2cyclictest3133801-21irqcore00:15:172
281126599129,2cyclictest2871631-21irqcore20:05:192
281126599129,2cyclictest1-21systemd23:29:192
281126599128,2cyclictest2940360-21tail21:00:142
281126599126,4cyclictest3008486-21tr22:00:182
281126599121,3cyclictest2956-21rtkit-daemon20:32:142
2811265991210,1cyclictest0-21swapper/223:45:002
2811265991210,1cyclictest0-21swapper/219:30:192
281126099129,2cyclictest2914403-21sendmail20:39:591
281126099129,2cyclictest2854699-21cat19:50:141
281126099129,2cyclictest2844698-21grep19:35:241
281126099129,2cyclictest0-21swapper/122:30:111
281126099128,3cyclictest2845565-21fschecks_count19:40:151
281126099127,4cyclictest2877299-21sed20:10:011
281126099126,4cyclictest493-21dbus-daemon21:10:001
281126099126,4cyclictest493-21dbus-daemon21:10:001
281126099121,3cyclictest2956-21rtkit-daemon20:50:201
281126099121,2cyclictest730-21ntpd22:15:241
281125799129,2cyclictest3133243-21gltestperf00:15:160
281125799129,2cyclictest2876874-21kworker/0:3+kacpid22:35:190
281125799129,2cyclictest2833150-21fschecks_time19:30:110
281125799129,2cyclictest2804599-21kworker/0:0+kacpi_notify19:20:160
281125799129,2cyclictest0-21swapper/022:50:000
281125799129,2cyclictest0-21swapper/022:30:120
281125799129,2cyclictest0-21swapper/021:15:240
281125799127,4cyclictest3614154-21systemd-journal21:40:000
281125799127,4cyclictest2952881-21gltestperf21:10:170
281125799126,4cyclictest0-21swapper/023:10:080
281125799122,8cyclictest3614154-21systemd-journal20:40:000
2811257991210,1cyclictest2811009-21kworker/0:2+kacpid22:00:180
2811257991210,1cyclictest2811009-21kworker/0:2+kacpid20:52:340
281125799120,5cyclictest3063483-21irqrtprio23:00:170
281125799120,4cyclictest730-21ntpd22:25:010
2811267991110,1cyclictest0-21swapper/300:03:233
281126599118,1cyclictest0-21swapper/219:20:132
281126599116,4cyclictest3139170-21apt00:20:132
281126599116,4cyclictest3128330-21cut00:10:122
281126599116,4cyclictest2990835-21cron21:50:002
281126599116,4cyclictest2881482-21cut20:10:202
281126599115,4cyclictest2994420-21tr21:50:132
281126599110,1cyclictest295798rtkit-daemon22:46:372
281126099119,1cyclictest0-21swapper/123:30:191
281126099119,1cyclictest0-21swapper/120:11:191
281126099119,1cyclictest0-21swapper/100:31:321
281126099118,2cyclictest3030041-21cron22:30:001
281126099118,2cyclictest2858557-21cpuspeed_turbos19:55:171
281126099116,4cyclictest3101920-21sed23:40:131
281126099116,4cyclictest3010805-21cut22:10:091
281126099114,5cyclictest1-21systemd19:30:011
281126099110,9cyclictest0-21swapper/100:25:351
281126099110,5cyclictest2890273-21tail20:20:201
281125799117,3cyclictest0-21swapper/020:30:150
281125799117,2cyclictest0-21swapper/023:40:110
281125799110,3cyclictest3054899-21mailstats22:50:180
281126599107,2cyclictest0-21swapper/222:05:192
281126599106,3cyclictest3029670-21sshd22:25:202
281126099109,1cyclictest0-21swapper/123:25:131
281126099100,1cyclictest0-21swapper/122:07:381
281126099100,10cyclictest0-21swapper/121:26:101
281125799109,1cyclictest0-21swapper/023:05:410
281125799107,2cyclictest3146623-21gltestperf00:30:160
281125799107,2cyclictest0-21swapper/022:10:010
281125799105,4cyclictest733-21lldpd21:48:130
281125799100,10cyclictest0-21swapper/019:26:330
28112609994,4cyclictest3126202-21seq00:06:521
28112579997,1cyclictest0-21swapper/019:10:010
28112579991,3cyclictest2956-21rtkit-daemon23:28:280
2811097296,2sleep30-21swapper/319:09:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional