You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 08:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Nov 22, 2024 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
314535921110,9sleep11753-21gnome-shell23:10:281
162399910,8rtkit-daemon1622-21rtkit-daemon00:04:143
305977428771,10sleep20-21swapper/219:09:032
305964328566,14sleep30-21swapper/319:07:243
162399820,9rtkit-daemon26-21ksoftirqd/121:49:461
30768892800,6sleep00-21swapper/019:55:130
305965027660,9sleep00-21swapper/019:07:300
162399710,5rtkit-daemon2197014-21PK-Backend21:21:193
162399690,9rtkit-daemon1622-21rtkit-daemon22:19:402
162399680,9rtkit-daemon1622-21rtkit-daemon22:09:282
305974826638,22sleep10-21swapper/119:08:421
162399660,8rtkit-daemon1622-21rtkit-daemon21:27:552
162399660,4rtkit-daemon2197014-21PK-Backend22:26:373
3059993996422,15cyclictest2179-21gvfsd-metadata23:15:023
30928232610,8chrt1753-21gnome-shell20:40:221
162399610,9rtkit-daemon1622-21rtkit-daemon19:18:151
31029712600,5sleep0161rcuc/021:10:190
3059993996016,15cyclictest3116081-21cat21:45:293
162399600,9rtkit-daemon1622-21rtkit-daemon23:21:242
3059993995918,13cyclictest860997-21snmpd20:05:573
162399590,8rtkit-daemon1622-21rtkit-daemon21:24:212
3059993995820,12cyclictest3110135-21cat21:30:193
3059993995820,12cyclictest3110135-21cat21:30:183
3059993995818,14cyclictest3242255-21gdbus20:30:013
3059993995818,14cyclictest3242255-21gdbus20:30:013
3059993995630,7cyclictest1753-21gnome-shell20:48:393
3059993995622,8cyclictest1753-21gnome-shell21:11:143
3059993995618,11cyclictest3162961-21cut00:05:153
305999399561,14cyclictest3086115-21python320:20:293
305999399552,9cyclictest1753-21gnome-shell21:02:143
305999399552,25cyclictest1753-21gnome-shell20:11:213
3059993995519,11cyclictest3076303-21sh19:55:003
3059993995518,12cyclictest3124392-21ntpq22:10:263
305999399542,25cyclictest1753-21gnome-shell19:25:303
305999399542,25cyclictest1753-21gnome-shell19:25:303
305999399542,25cyclictest1753-21gnome-shell00:21:463
3059993995421,8cyclictest1753-21gnome-shell23:19:453
305999399532,25cyclictest1753-21gnome-shell00:26:103
305999399532,25cyclictest1753-21gnome-shell00:17:453
3059993995319,9cyclictest3070335-21cut19:35:223
3059993995318,11cyclictest42-21ksoftirqd/322:49:173
305999399531,26cyclictest3133208-21python322:35:293
305999399531,26cyclictest3133208-21python322:35:283
162399530,8rtkit-daemon1622-21rtkit-daemon22:00:352
305999399522,9cyclictest1753-21gnome-shell21:37:213
305999399522,8cyclictest3063086-21hddtemp_smartct19:15:203
305999399522,8cyclictest1753-21gnome-shell22:00:103
305999399522,8cyclictest1753-21gnome-shell20:16:493
305999399522,25cyclictest1753-21gnome-shell20:34:133
305998299522,46cyclictest1622-21rtkit-daemon19:42:000
31313762510,8sleep11753-21gnome-shell22:30:301
30789712510,5chrt0-21swapper/020:00:240
305999399512,9cyclictest3092763-21memory20:40:213
305999399512,7cyclictest854-21avahi-daemon00:10:093
3059993995122,8cyclictest1753-21gnome-shell23:38:103
3059993995122,8cyclictest1753-21gnome-shell23:38:103
305999399512,27cyclictest1753-21gnome-shell22:55:113
305999399512,27cyclictest1753-21gnome-shell22:55:103
305999399512,26cyclictest1753-21gnome-shell23:54:353
305999399510,46cyclictest1622-21rtkit-daemon22:05:363
3059982995124,23cyclictest3104989-40lsb_release21:15:200
305999399502,26cyclictest1753-21gnome-shell19:57:023
3059993995021,8cyclictest1753-21gnome-shell22:42:093
30746612490,22sleep01753-21gnome-shell19:50:030
30746612490,22sleep01753-21gnome-shell19:50:020
305999399492,25cyclictest1753-21gnome-shell19:10:393
305999399472,8cyclictest1753-21gnome-shell21:27:443
3059986994728,14cyclictest3242253-21nautilus22:25:001
31540662440,6sleep02197014-21PK-Backend23:35:210
31540662440,6sleep02197014-21PK-Backend23:35:210
3059990994426,13cyclictest855-21dbus-daemon19:35:002
3059986994419,21cyclictest3123385-21cron22:10:011
305999099433,21cyclictest141rcu_preempt00:20:032
3059990994312,12cyclictest141rcu_preempt23:35:012
3059986994331,8cyclictest3085794-21grep20:20:211
3059986994330,9cyclictest26-21ksoftirqd/123:16:141
3059982994328,11cyclictest3133517-21sshd22:36:360
3059982994328,11cyclictest3133517-21sshd22:36:360
3059982994315,6cyclictest141rcu_preempt22:52:360
3059982994310,7cyclictest17531gnome-shell20:18:450
31327352420,4chrt2197014-21PK-Backend22:35:182
31327352420,4chrt2197014-21PK-Backend22:35:182
3059990994228,10cyclictest3156700-21sh23:45:002
3059990994226,12cyclictest3094033-21grep20:45:132
3059986994217,20cyclictest3065931-21munin-run19:25:001
305998699421,13cyclictest3164495-21cat00:10:021
3059982994229,9cyclictest3160884-21systemd00:00:020
3059982994225,12cyclictest855-21dbus-daemon23:22:540
3059982994224,13cyclictest438-20systemd-journal23:31:560
3059982994224,13cyclictest438-20systemd-journal21:59:590
162399420,35rtkit-daemon1775-21gvfs-afc-volume22:28:472
162399420,32rtkit-daemon3059974-21cyclictest00:05:022
31362792410,6chrt0-21swapper/222:45:202
30631892410,7sleep21753-21gnome-shell19:15:232
305999099415,5cyclictest141rcu_preempt20:55:242
3059990994127,10cyclictest3175100-21sshd00:37:072
3059990994127,10cyclictest3175100-21sshd00:37:062
3059990994124,13cyclictest3061758-21rm19:15:012
3059990994122,14cyclictest34-21ksoftirqd/221:40:002
305999099411,23cyclictest3147154-21needreboot23:15:242
3059986994117,19cyclictest26-21ksoftirqd/123:30:291
305998299412,9cyclictest1753-21gnome-shell21:45:490
3059982994126,11cyclictest3121675-21/usr/sbin/munin22:05:000
3059982994123,12cyclictest438-20systemd-journal20:46:360
305998299412,18cyclictest3169517-21python300:20:300
162399410,34rtkit-daemon3061942-21munin-node22:11:182
3059993994025,11cyclictest42-21ksoftirqd/323:47:293
3059982994023,13cyclictest3070015-21df19:35:130
3059982994022,13cyclictest438-20systemd-journal00:10:010
3059982994022,13cyclictest13-21ksoftirqd/021:27:310
3059982994021,6cyclictest13-21ksoftirqd/019:22:040
3059982994021,14cyclictest13-21ksoftirqd/022:12:430
305998299401,21cyclictest3146850-40lsb_release23:15:190
162399400,26rtkit-daemon0-21swapper/221:10:332
31172612390,5sleep32197014-21PK-Backend21:50:243
305999399392,9cyclictest1753-21gnome-shell22:33:253
305999399391,13cyclictest860997-21snmpd00:40:023
305999399391,13cyclictest860997-21snmpd00:40:023
305999099398,11cyclictest34-21ksoftirqd/220:10:002
3059990993923,12cyclictest3145906-21cron23:15:002
3059990993921,13cyclictest3143434-21awk23:05:242
3059990993911,6cyclictest141rcu_preempt19:28:162
3059990993911,6cyclictest141rcu_preempt19:28:162
305999099391,12cyclictest17531gnome-shell23:45:112
3059986993923,12cyclictest3156985-21sshd23:45:361
3059986993923,11cyclictest1-21systemd00:00:011
3059986993921,13cyclictest438-20systemd-journal21:09:361
3059986993920,14cyclictest3079065-21ntp_states20:00:271
3059982993929,7cyclictest1069-21runrttasks21:22:430
3059982993929,6cyclictest3114741-21turbostat21:45:000
305998299392,33cyclictest1753-21gnome-shell22:22:140
305998299392,33cyclictest1753-21gnome-shell00:33:170
3059982993923,11cyclictest855-21dbus-daemon23:05:000
305998299392,19cyclictest1753-21gnome-shell21:35:240
305998299391,34cyclictest1753-21gnome-shell22:31:180
3059982993910,6cyclictest141rcu_preempt20:10:110
162399390,31rtkit-daemon3061942-21munin-node00:13:452
30994482380,4chrt0-21swapper/121:00:141
3059993993822,12cyclictest3153042-21sshd23:32:363
305999399381,15cyclictest3064861-21smartctl19:20:183
305999099382,8cyclictest1753-21gnome-shell20:22:382
305999099382,7cyclictest1753-21gnome-shell00:26:332
305999099382,32cyclictest1753-21gnome-shell19:49:482
305999099382,32cyclictest1753-21gnome-shell19:49:482
305998699382,7cyclictest1753-21gnome-shell23:03:491
3059986993825,9cyclictest1827-21JS14:31:441
305998699382,32cyclictest1753-21gnome-shell23:36:401
305998699382,32cyclictest1753-21gnome-shell23:36:391
305998699382,32cyclictest1753-21gnome-shell19:53:221
3059986993822,11cyclictest438-20systemd-journal22:50:001
3059986993821,13cyclictest3168580-21gdbus00:20:031
3059986993819,14cyclictest438-20systemd-journal19:45:001
305998699381,10cyclictest0-21swapper/120:36:141
3059986993810,22cyclictest3167515-21smartctl00:15:281
3059986993810,12cyclictest855-21dbus-daemon22:05:001
305998299388,11cyclictest141rcu_preempt21:30:230
305998299388,11cyclictest141rcu_preempt21:30:230
305998299385,10cyclictest13-21ksoftirqd/021:05:570
305998299384,16cyclictest1753-21gnome-shell20:55:100
305998299383,17cyclictest1753-21gnome-shell21:00:290
305998299382,32cyclictest1753-21gnome-shell23:51:590
305998299382,32cyclictest1753-21gnome-shell00:03:330
3059982993822,12cyclictest3080297-21ls20:05:130
3059982993822,12cyclictest3076287-21seq19:54:540
305998299382,18cyclictest3129618-21ntp_states22:25:250
3059982993821,13cyclictest13-21ksoftirqd/020:53:010
3059982993820,13cyclictest3125763-21df22:15:140
3059982993819,6cyclictest13-21ksoftirqd/019:31:110
162399380,30rtkit-daemon3059974-21cyclictest21:58:422
31079252370,7chrt0-21swapper/121:25:021
3059993993724,9cyclictest1753-21gnome-shell23:20:213
3059993993712,7cyclictest1753-21gnome-shell21:15:103
305999099375,11cyclictest34-21ksoftirqd/220:54:112
305999099372,8cyclictest1753-21gnome-shell22:57:252
305999099372,8cyclictest1753-21gnome-shell22:57:252
3059990993727,7cyclictest331rcuc/219:52:262
3059990993726,7cyclictest3072853-21turbostat.cron19:45:012
305999099372,31cyclictest1753-21gnome-shell23:35:352
305999099372,31cyclictest1753-21gnome-shell23:35:352
3059990993720,13cyclictest3159106-21systemd23:55:012
3059990993711,7cyclictest3135538-21turbostat22:45:012
3059986993727,7cyclictest1069-21runrttasks19:59:401
3059986993726,7cyclictest3068092-21tr19:30:121
305998699372,31cyclictest1069-21runrttasks22:25:471
3059986993721,12cyclictest3158715-21sshd23:50:361
3059986993720,12cyclictest3175220-21sshd00:38:331
3059986993720,12cyclictest3175220-21sshd00:38:331
3059986993715,17cyclictest3104880-21expr21:15:151
305998699371,12cyclictest3134217-21taskset22:40:131
305998699371,10cyclictest3164850-21/usr/sbin/munin00:10:171
305998299372,8cyclictest1753-21gnome-shell21:52:310
3059982993725,8cyclictest3086723-21sleep20:25:000
3059982993722,11cyclictest3151494-21cron23:30:000
305998299372,18cyclictest3090822-21diskstats20:35:160
305998299372,18cyclictest1753-21gnome-shell20:40:290
3059982993720,13cyclictest3165655-21awk00:10:330
3059982993720,12cyclictest438-20systemd-journal19:25:360
3059982993720,12cyclictest438-20systemd-journal19:25:350
3059982993720,12cyclictest438-20systemd-journal00:38:020
3059982993720,12cyclictest438-20systemd-journal00:38:020
3059982993719,13cyclictest3062407-21irqstats19:15:060
3059982993718,6cyclictest13-21ksoftirqd/019:12:070
3059993993620,12cyclictest3097233-21cut20:55:013
3059993993619,14cyclictest3073108-21basename19:45:123
3059993993619,14cyclictest3073108-21basename19:45:123
3059993993619,13cyclictest855-21dbus-daemon00:00:023
3059993993616,6cyclictest141rcu_preempt20:55:113
305999399361,13cyclictest3114036-21open_files21:40:253
3059990993627,6cyclictest0-21swapper/221:03:542
3059990993626,7cyclictest331rcuc/220:17:532
3059990993625,8cyclictest3173690-21cat00:35:012
3059990993620,12cyclictest860997-21snmpd19:55:302
3059990993618,14cyclictest3242255-21gdbus20:35:002
305999099361,21cyclictest175350gnome-shell00:20:262
305999099361,14cyclictest1753-21gnome-shell22:54:012
3059986993624,9cyclictest231irq_work/120:18:441
3059986993623,10cyclictest3149788-21awk23:25:001
3059986993622,10cyclictest26-21ksoftirqd/119:10:061
3059986993621,11cyclictest3109555-21/usr/sbin/munin21:30:011
3059986993619,12cyclictest3082417-21cat20:10:241
3059986993615,10cyclictest3133397-21sed22:35:301
3059986993615,10cyclictest3133397-21sed22:35:301
305998299369,5cyclictest141rcu_preempt00:25:140
305998299368,13cyclictest141rcu_preempt22:55:290
305998299368,13cyclictest141rcu_preempt22:55:280
305998299364,11cyclictest13-21ksoftirqd/022:45:380
305998299362,9cyclictest1753-21gnome-shell23:48:320
3059982993622,10cyclictest3143984-21sshd23:07:360
3059982993620,12cyclictest1-21systemd20:30:010
3059982993620,12cyclictest1-21systemd20:30:010
3059982993618,5cyclictest13-21ksoftirqd/000:16:470
162399360,11rtkit-daemon2197014-21PK-Backend20:36:072
3059993993523,8cyclictest1753-21gnome-shell19:32:163
305999399352,16cyclictest1753-21gnome-shell21:55:593
3059993993521,10cyclictest3079762-21cron20:05:003
3059993993518,12cyclictest3138630-21awk22:50:353
3059993993517,13cyclictest1632-21dbus-daemon21:10:023
305999099357,5cyclictest141rcu_preempt21:49:232
305999099356,9cyclictest34-21ksoftirqd/222:22:592
305999099354,11cyclictest3104576-40apt-check21:15:142
305999099351,12cyclictest1358-1kworker/u9:121:05:342
3059986993523,7cyclictest26-21ksoftirqd/120:55:521
3059986993521,10cyclictest3172898-21sh00:30:291
3059986993521,10cyclictest3119963-21systemctl22:00:001
3059986993520,11cyclictest3067654-21dump-pmu-power19:30:011
3059986993520,11cyclictest3067654-21dump-pmu-power19:30:001
3059986993519,12cyclictest3113039-21pool-org.gnome.21:40:001
3059986993519,12cyclictest3104208-21pool-org.gnome.21:15:021
3059986993518,13cyclictest3097134-21pool-org.gnome.20:55:001
3059986993518,12cyclictest3117429-21netstat21:50:261
3059986993517,13cyclictest438-20systemd-journal19:39:541
3059986993515,17cyclictest1753-21gnome-shell22:55:241
3059986993515,17cyclictest1753-21gnome-shell22:55:241
305998299352,9cyclictest1753-21gnome-shell23:40:230
3059982993520,11cyclictest3088896-21expr20:30:130
3059993993422,8cyclictest1753-21gnome-shell22:19:413
3059993993419,11cyclictest42-21ksoftirqd/320:38:243
3059993993419,11cyclictest3155616-21sh23:40:163
3059990993422,9cyclictest34-21ksoftirqd/219:22:372
305999099342,28cyclictest1753-21gnome-shell20:40:112
3059990993419,11cyclictest3162569-21grep00:04:592
3059990993419,11cyclictest3109820-21dpkg21:30:162
3059990993419,11cyclictest3109820-21dpkg21:30:152
3059990993411,8cyclictest141rcu_preempt00:00:032
305999099341,13cyclictest2197014-21PK-Backend23:25:302
305999099341,12cyclictest141rcu_preempt21:40:122
3059986993423,7cyclictest1753-21gnome-shell20:28:041
3059986993423,7cyclictest1753-21gnome-shell20:28:031
305998699342,29cyclictest1069-21runrttasks21:34:341
305998699342,29cyclictest1069-21runrttasks21:34:341
305998699342,26cyclictest854-21avahi-daemon00:01:211
3059986993422,6cyclictest26-21ksoftirqd/120:05:271
3059986993419,11cyclictest3124436-21awk22:10:281
3059986993419,11cyclictest3073569-21cat19:45:221
3059986993419,11cyclictest3073569-21cat19:45:211
3059986993412,5cyclictest141rcu_preempt22:55:031
3059986993411,19cyclictest3156865-21latency_hist23:45:011
3059982993420,10cyclictest13-21ksoftirqd/023:12:560
3059993993317,12cyclictest3127706-21sed22:20:223
305999399331,29cyclictest2197014-21PK-Backend19:42:403
305999399331,28cyclictest2197014-21PK-Backend23:02:303
305999099337,11cyclictest141rcu_preempt21:55:022
3059990993325,5cyclictest0-21swapper/223:00:152
3059990993321,8cyclictest1753-21gnome-shell20:29:562
3059990993321,8cyclictest1753-21gnome-shell20:29:552
3059990993321,8cyclictest1753-21gnome-shell19:36:082
3059986993322,6cyclictest26-21ksoftirqd/123:10:001
3059986993321,8cyclictest3126300-21ls22:15:301
3059986993317,12cyclictest3090218-21grep20:35:011
3059986993314,14cyclictest3150797-21ntp_states23:25:261
305998299338,5cyclictest141rcu_preempt22:41:520
3059982993319,10cyclictest3122922-21wc22:05:310
3059993993221,7cyclictest3150408-21df23:25:153
3059993993219,9cyclictest1753-21gnome-shell23:06:113
3059990993220,9cyclictest1753-21gnome-shell20:01:552
3059990993214,13cyclictest438-20systemd-journal20:14:592
305999099321,12cyclictest141rcu_preempt22:30:342
3059986993221,6cyclictest26-21ksoftirqd/121:40:111
305999399312,9cyclictest3173235-21users00:30:353
3059986993121,6cyclictest26-21ksoftirqd/120:45:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional