You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 07:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8s.osadl.org (updated Wed Apr 02, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
300578199747741,5cyclictest3038918-21kworker/3:0+events20:12:543
300577799746741,3cyclictest3287939-21kworker/2:2+events23:38:022
300578199743740,2cyclictest3038918-21kworker/3:0+events20:22:553
300578199740733,6cyclictest3229664-21turbostat22:39:553
300578199738734,3cyclictest3184169-21kworker/3:1+events23:49:223
300578199734728,5cyclictest3184169-21kworker/3:1+events21:48:033
300578199732726,5cyclictest3096440-21turbostat20:34:553
300577699732725,6cyclictest3019968-21kworker/1:0+events23:00:221
300578199731728,2cyclictest3038918-21kworker/3:0+events21:33:173
300577699731727,3cyclictest3019968-21kworker/1:0+events19:21:161
300578199730726,3cyclictest3032670-21turbostat19:34:553
300577699729725,3cyclictest3019968-21kworker/1:0+events22:53:501
300578199727721,5cyclictest3204175-21turbostat22:14:553
300578199727721,5cyclictest3204175-21turbostat22:14:553
300577699727723,3cyclictest3019968-21kworker/1:0+events19:31:451
300577799726721,4cyclictest3012157-21kworker/2:3+events19:56:272
300578199725719,5cyclictest3184169-21kworker/3:1+events23:55:243
300578199725719,5cyclictest3184169-21kworker/3:1+events23:55:233
300578199724721,2cyclictest3038918-21kworker/3:0+events19:36:063
300577699723716,4cyclictest3019968-21kworker/1:0+events23:09:291
300577699721717,3cyclictest2961696-21kworker/1:2+events19:15:411
300578199720714,5cyclictest3184169-21kworker/3:1+events00:14:223
300577699720716,3cyclictest3320969-21kworker/1:1+events00:39:221
300577699720716,3cyclictest3320969-21kworker/1:1+events00:22:521
300577699719716,2cyclictest3320969-21kworker/1:1+events00:28:261
300577699719716,2cyclictest3019968-21kworker/1:0+events22:57:121
300577699719715,3cyclictest3019968-21kworker/1:0+events19:29:501
300578199718715,2cyclictest3184169-21kworker/3:1+events00:16:183
300578199718714,3cyclictest3184169-21kworker/3:1+events23:41:393
300577699718714,3cyclictest3320969-21kworker/1:1+events00:34:491
300577699717713,3cyclictest3019968-21kworker/1:0+events23:13:151
300577699717713,3cyclictest2961696-21kworker/1:2+events19:14:461
300578199716712,3cyclictest3184169-21kworker/3:1+events22:41:073
300578199715711,3cyclictest3184169-21kworker/3:1+events23:17:293
300578199714709,4cyclictest3038918-21kworker/3:0+events20:07:113
300578199714709,4cyclictest3038918-21kworker/3:0+events20:07:113
300577799710707,2cyclictest3184317-21kworker/2:1+events23:26:322
300578199709705,3cyclictest3038918-21kworker/3:0+events21:10:223
300577199709702,6cyclictest3038760-21kworker/0:3+events21:55:220
300578199705702,2cyclictest3038918-21kworker/3:0+events21:28:573
300578199705700,4cyclictest3038918-21kworker/3:0+events21:40:203
300578199705699,5cyclictest3184169-21kworker/3:1+events00:00:233
300578199704700,3cyclictest3038918-21kworker/3:0+events19:52:123
300578199702698,3cyclictest3083385-21turbostat20:19:553
300578199701698,2cyclictest3038918-21kworker/3:0+events20:52:433
300577799701694,6cyclictest3184317-21kworker/2:1+events23:20:202
300577799699694,4cyclictest3184317-21kworker/2:1+events22:47:082
300578199698694,3cyclictest3038918-21kworker/3:0+events20:36:273
300578199696693,2cyclictest3184169-21kworker/3:1+events21:51:393
300578199696693,2cyclictest3038918-21kworker/3:0+events20:55:223
300578199696690,5cyclictest3152571-21turbostat21:19:563
300578199695691,3cyclictest3038918-21kworker/3:0+events21:09:313
300578199694690,3cyclictest3184169-21kworker/3:1+events22:16:483
300577199694687,6cyclictest3184907-21turbostat21:54:550
300578199693690,2cyclictest3184169-21kworker/3:1+events00:05:223
300578199693689,3cyclictest3038918-21kworker/3:0+events20:04:003
300578199692689,2cyclictest3184169-21kworker/3:1+events22:34:523
300578199691688,2cyclictest3184169-21kworker/3:1+events22:23:503
300577799691688,2cyclictest3236171-21turbostat22:44:552
300578199690684,3cyclictest3184169-21kworker/3:1+events22:27:103
300578199688685,2cyclictest3038918-21kworker/3:0+events19:41:483
300578199688684,3cyclictest3038918-21kworker/3:0+events20:40:223
300578199685681,3cyclictest3038918-21kworker/3:0+events21:38:123
300577699685679,5cyclictest3327143-21turbostat00:19:551
300578199684681,2cyclictest3184169-21kworker/3:1+events23:52:423
300578199684681,2cyclictest3038918-21kworker/3:0+events21:01:183
300578199684681,2cyclictest3038918-21kworker/3:0+events20:26:273
300578199684679,4cyclictest3038918-21kworker/3:0+events19:45:213
300578199683680,2cyclictest3038918-21kworker/3:0+events21:23:323
300578199682679,2cyclictest3038918-21kworker/3:0+events20:46:043
300577199679675,3cyclictest3197054-21kworker/0:0+events22:00:430
300577199676672,3cyclictest913995-21kworker/0:2+events22:08:230
300577799656654,1cyclictest3057508-21turbostat19:54:552
300578199653650,2cyclictest3203547-21turbostat22:09:553
300577799650646,3cyclictest3184317-21kworker/2:1+events23:30:202
300578199649645,3cyclictest3262536-21turbostat23:14:553
300578199646643,2cyclictest3288272-21turbostat23:39:553
300578199645641,3cyclictest3063756-21turbostat19:59:553
300577799621619,1cyclictest3268737-21turbostat23:19:552
300577699597593,3cyclictest3242378-21turbostat22:49:551
2957983480,344rtkit-daemon2961696-21kworker/1:2+events19:09:021
3005776991830,181cyclictest3039153-21apt-get19:35:121
3005771991450,143cyclictest3303381-21kthreadcore23:50:130
3005776992724,2cyclictest3314043-21sendmail00:00:001
3005776992724,2cyclictest3314043-21sendmail00:00:001
3005771992522,2cyclictest3127741-21idleruntime-cro20:55:010
3005781992418,4cyclictest3026458-21sh19:25:003
300578199240,13cyclictest3339651-21latency_hist00:25:003
300578199230,12cyclictest3250446-21cut22:55:163
300578199230,12cyclictest3246399-21grep22:50:143
300578199230,12cyclictest3031314-21grep19:25:193
300578199220,11cyclictest3347920-21sed00:35:183
300578199220,11cyclictest3284513-21grep23:30:123
300578199220,11cyclictest3260769-21grep23:00:183
300578199220,11cyclictest3202840-21python322:00:213
300578199220,11cyclictest3012158-21munin-node21:55:273
300578199220,11cyclictest3010188-21grep19:10:203
3005777992219,2cyclictest3249216-21munin-plugin-st22:55:002
300578199211,9cyclictest753-21runrttasks23:05:443
300578199210,10cyclictest3343314-21tr00:30:183
300578199200,9cyclictest3014231-21/usr/sbin/munin19:15:153
300578199200,11cyclictest753-21runrttasks00:29:583
3005777992017,2cyclictest3340244-21basename00:30:002
3005777991916,2cyclictest3153294-21irqcore21:15:162
3005777991916,2cyclictest0-21swapper/223:15:002
3005777991916,2cyclictest0-21swapper/200:00:002
3005777991916,2cyclictest0-21swapper/200:00:002
3005776991916,2cyclictest3283808-21kthreadcore23:30:121
3005776991916,2cyclictest0-21swapper/121:40:001
3005776991915,3cyclictest3216654-21sendmail22:20:001
3005771991915,3cyclictest3317350-21kthreadcore00:00:190
3005771991915,2cyclictest491-21avahi-daemon23:50:000
3005771991914,3cyclictest3012339-21grep19:15:020
3005771991913,4cyclictest3346410-21turbostat00:35:000
3005771991911,7cyclictest0-21swapper/019:55:000
3005777991814,3cyclictest3184265-21cron21:45:012
3005776991814,3cyclictest3102460-21unixbench_multi20:30:201
3005771991813,3cyclictest31467412chrt21:10:160
3005771991813,3cyclictest3078960-21kthreadcore20:10:160
3005777991714,2cyclictest523-21gdbus20:50:012
3005777991713,3cyclictest3299742-21needreboot23:40:182
3005777991713,3cyclictest3172382-21fschecks_time21:35:152
3005777991710,5cyclictest523-21gdbus23:50:002
3005776991713,3cyclictest493-21dbus-daemon21:20:011
3005776991711,4cyclictest3191677-21grep21:55:171
300577199178,6cyclictest3614154-21systemd-journal00:00:000
300577199178,6cyclictest3614154-21systemd-journal00:00:000
3005771991714,2cyclictest3261913-21cut23:05:010
3005771991713,3cyclictest493-21dbus-daemon23:20:000
3005771991713,3cyclictest3146347-21cut21:10:010
3005771991713,3cyclictest3052023-21irqcore19:45:160
3005771991713,2cyclictest0-21swapper/020:30:000
3005771991713,2cyclictest0-21swapper/000:20:000
3005771991711,4cyclictest3133900-21cron21:00:000
300578199164,10cyclictest3614154-21systemd-journal23:25:233
300578199160,8cyclictest0-21swapper/322:45:053
300577799169,5cyclictest493-21dbus-daemon00:40:012
300577799164,4cyclictest728-21snmpd20:14:422
3005777991613,2cyclictest0-21swapper/219:40:002
3005777991612,3cyclictest3171450-21unixbench_singl21:30:252
3005776991613,2cyclictest506-21in:imuxsock21:25:001
3005776991613,2cyclictest3275361-21latency_hist23:20:001
3005776991612,3cyclictest3204161-21cron22:10:001
3005776991610,4cyclictest3294476-21cat23:40:001
3005771991612,3cyclictest3236143-21cron22:40:000
3005771991612,3cyclictest3089630-21sendmail20:20:000
3005771991611,3cyclictest3262523-21cron23:10:000
3005771991611,3cyclictest3249199-21cron22:55:000
300577199160,15cyclictest0-21swapper/020:00:240
300577799159,4cyclictest1-21systemd19:25:002
300577799157,6cyclictest493-21dbus-daemon20:05:012
300577799154,2cyclictest0-21swapper/222:55:192
3005777991512,2cyclictest3211025-21gltestperf22:15:152
3005777991512,2cyclictest3115322-21sh20:45:002
3005777991512,2cyclictest3027683-21irqrtprio19:25:172
3005777991510,3cyclictest0-21swapper/222:35:002
3005777991510,3cyclictest0-21swapper/200:15:002
300577699159,4cyclictest0-21swapper/123:25:001
3005776991513,1cyclictest0-21swapper/123:45:001
3005776991512,2cyclictest3133908-21idleruntime-cro21:00:001
3005776991511,3cyclictest3171463-21unixbench_singl21:30:251
300577699151,12cyclictest3064618-21gltestperf19:55:151
300577199159,4cyclictest3045114-21sh19:40:010
300577199157,6cyclictest493-21dbus-daemon19:25:000
3005771991512,2cyclictest3204509-21cpuspeed_turbos22:10:090
3005771991512,2cyclictest3204509-21cpuspeed_turbos22:10:090
3005771991512,2cyclictest3097286-21gltestperf20:30:110
3005771991512,2cyclictest171ktimers/021:41:580
3005771991511,2cyclictest913995-21kworker/0:2+kacpid20:05:020
3005771991511,2cyclictest913995-21kworker/0:2+kacpid20:05:020
3005771991511,2cyclictest3297783-21kthreadcore23:40:160
3005771991511,2cyclictest0-21swapper/020:45:000
300578199141,11cyclictest3275987-21sed23:20:193
300578199140,12cyclictest3005850-21cut19:10:013
300577799143,5cyclictest391rcuc/220:50:252
3005777991411,2cyclictest3342314-21kthreadcore00:30:182
3005777991411,2cyclictest3333359-21munin-plugin-st00:20:002
3005777991411,2cyclictest3093101-21kthreadcore20:20:182
3005777991411,2cyclictest0-21swapper/221:10:012
3005777991410,3cyclictest401ktimers/222:20:122
3005777991410,3cyclictest3614154-21systemd-journal20:55:242
3005777991410,3cyclictest3314383-21cpuspeed_turbos00:00:142
3005777991410,3cyclictest3146715-21cpuspeed_turbos21:10:162
3005777991410,3cyclictest3033145-21fschecks_count19:30:182
300577699149,4cyclictest0-21swapper/120:25:001
300577699148,4cyclictest3614154-21systemd-journal20:50:231
300577699143,5cyclictest3045864-21irqcore19:40:171
3005776991410,3cyclictest3058743-21irqrtprio19:50:231
300577699140,1cyclictest0-21swapper/100:10:001
300577199147,5cyclictest3139292-21ntpq21:00:220
300577199144,9cyclictest3160535-21cat21:20:180
300577199142,2cyclictest3152844-21apt-get21:15:120
3005771991411,2cyclictest913995-21kworker/0:2+kacpid20:47:500
3005771991411,2cyclictest506-21in:imuxsock00:30:000
3005771991411,2cyclictest3275625-21cpuspeed_turbos23:20:160
300577199141,12cyclictest3216626-21idleruntime-cro22:20:000
3005771991411,2cyclictest0-21swapper/023:35:180
300577799139,3cyclictest3083972-21gltestperf20:15:162
300577799139,3cyclictest0-21swapper/219:45:172
300577799136,5cyclictest523-21gdbus23:05:012
300577799134,6cyclictest391rcuc/221:20:172
3005777991311,1cyclictest391rcuc/221:50:132
3005777991311,1cyclictest0-21swapper/220:35:182
3005777991310,2cyclictest3197833-21fschecks_time22:00:122
3005777991310,2cyclictest3134276-21cpuspeed_turbos21:00:132
3005777991310,2cyclictest3012485-21kthreadcore19:15:032
300577799130,4cyclictest312-21jbd2/sda2-819:40:182
300577699139,3cyclictest3103255-21cpuspeed_turbos20:35:141
300577699139,2cyclictest3057505-21turbostat.cron19:50:001
300577699138,4cyclictest0-21swapper/122:30:001
3005776991310,2cyclictest3314766-21sed00:00:171
3005776991310,2cyclictest3071020-21sed20:00:171
3005776991310,2cyclictest0-21swapper/122:40:001
3005776991310,2cyclictest0-21swapper/121:30:001
3005776991310,2cyclictest0-21swapper/120:10:141
300577699130,12cyclictest0-21swapper/121:50:111
300577199139,3cyclictest3282532-21fschecks_count23:30:090
300577199139,3cyclictest3026806-21cpuspeed_turbos19:25:130
300577199139,2cyclictest0-21swapper/020:25:010
300577199136,5cyclictest3321213-21apt-get00:10:090
300577199131,3cyclictest523-21gdbus21:40:000
3005771991310,2cyclictest745-21cups-browsed19:35:000
3005771991310,2cyclictest3197054-21kworker/0:0+kacpid22:31:110
300577199130,11cyclictest0-21swapper/000:20:090
30054972134,1sleep20-21swapper/219:08:322
300577799129,2cyclictest3333726-21cpuspeed_turbos00:20:132
300577799129,2cyclictest3303537-21kthreadcore23:50:122
300577799129,2cyclictest3209756-21switchtime22:10:182
300577799129,2cyclictest3209756-21switchtime22:10:182
300577799129,2cyclictest3191456-21cpuspeed_turbos21:55:142
300577799129,2cyclictest0-21swapper/220:30:002
300577799128,2cyclictest0-21swapper/219:10:172
300577799127,4cyclictest3165621-21cron21:30:002
300577799127,3cyclictest3614154-21systemd-journal21:45:232
300577799124,2cyclictest191rcu_preempt22:35:202
300577799122,1cyclictest191rcu_preempt20:30:102
300577699129,2cyclictest0-21swapper/120:30:011
300577699128,2cyclictest0-21swapper/121:15:011
300577699126,4cyclictest493-21dbus-daemon20:20:001
300577699123,2cyclictest191rcu_preempt20:10:001
300577699123,2cyclictest191rcu_preempt20:10:001
3005776991210,1cyclictest0-21swapper/122:43:321
300577699120,4cyclictest3224680-21tr22:30:131
300577699120,1cyclictest0-21swapper/120:40:161
300577199129,2cyclictest3011521-21unixbench_singl19:10:250
300577199127,4cyclictest3221975-21ntpq22:20:210
300577199122,2cyclictest3103190-21apt-get20:35:120
300577199121,9cyclictest3184914-21munin-run21:50:000
3005771991210,1cyclictest913995-21kworker/0:2+kacpid19:56:480
3005771991210,1cyclictest3038760-21kworker/0:3+kacpid23:25:260
300577199120,11cyclictest0-21swapper/022:58:380
300577799114,2cyclictest3077206-21apt20:10:002
300577799114,2cyclictest3077206-21apt20:10:002
3005777991110,1cyclictest0-21swapper/200:07:402
300577799110,1cyclictest0-21swapper/222:09:052
300577699118,2cyclictest6950irq/9-acpi22:20:081
300577699118,2cyclictest3321510-21gltestperf00:10:121
300577699118,2cyclictest0-21swapper/120:45:161
300577699117,3cyclictest3178634-21gltestperf21:40:141
300577699116,4cyclictest3137121-21cut21:00:181
300577699113,2cyclictest295798rtkit-daemon21:47:241
300577699110,1cyclictest0-21swapper/123:47:191
300577699110,10cyclictest0-21swapper/123:25:091
300577199118,2cyclictest3045522-21cpuspeed_turbos19:40:140
300577199118,2cyclictest0-21swapper/021:30:000
300577199117,2cyclictest3236452-21apt-get22:40:130
300577199110,4cyclictest728-21snmpd22:29:490
300577199110,1cyclictest0-21swapper/000:36:210
300577199110,11cyclictest0-21swapper/023:14:300
300577799100,3cyclictest0-21swapper/222:27:072
300577699107,2cyclictest6950irq/9-acpi22:11:561
300577699107,2cyclictest6950irq/9-acpi22:11:561
300577699107,2cyclictest6950irq/9-acpi22:00:171
300577699107,2cyclictest3302107-21fschecks_time23:50:091
300577699107,2cyclictest3140686-21gltestperf21:05:151
300577199107,2cyclictest171ktimers/022:47:580
300577199105,4cyclictest3170493-21tr21:30:200
300577199100,3cyclictest0-21swapper/000:05:480
30055402106,2sleep00-21swapper/019:09:050
30057779991,2cyclictest3262542-21cron23:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional