You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-11 - 22:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Apr 11, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
37893322350,22sleep30-21swapper/307:05:133
1623991200,9rtkit-daemon1622-21rtkit-daemon07:14:290
44482521140,6sleep22197014-21PK-Backend10:15:292
1623991140,9rtkit-daemon1622-21rtkit-daemon11:16:200
1623991030,9rtkit-daemon1622-21rtkit-daemon12:36:453
1623991010,9rtkit-daemon1622-21rtkit-daemon08:35:093
1623991000,9rtkit-daemon1622-21rtkit-daemon08:32:463
162399990,9rtkit-daemon1622-21rtkit-daemon10:13:133
162399960,9rtkit-daemon1622-21rtkit-daemon08:00:413
162399950,8rtkit-daemon1622-21rtkit-daemon12:13:213
162399940,9rtkit-daemon1622-21rtkit-daemon10:53:073
162399940,9rtkit-daemon1622-21rtkit-daemon10:53:063
162399940,9rtkit-daemon1622-21rtkit-daemon09:54:273
162399930,9rtkit-daemon1622-21rtkit-daemon09:35:383
162399930,8rtkit-daemon1622-21rtkit-daemon11:39:593
3957192920,8sleep32197014-21PK-Backend07:50:043
3957192920,8sleep32197014-21PK-Backend07:50:043
162399920,6rtkit-daemon1622-21rtkit-daemon09:49:053
4088422900,8sleep21753-21gnome-shell08:25:292
162399890,9rtkit-daemon1622-21rtkit-daemon10:29:163
162399880,8rtkit-daemon1622-21rtkit-daemon10:19:073
37991928258,16sleep00-21swapper/007:07:000
38030599760,71cyclictest1622-21rtkit-daemon08:07:493
37908627155,9sleep20-21swapper/207:05:172
162399710,54rtkit-daemon0-21swapper/107:05:151
38029799700,65cyclictest1622-21rtkit-daemon12:34:182
38029799680,63cyclictest1622-21rtkit-daemon11:52:282
38029799680,63cyclictest1622-21rtkit-daemon08:03:322
38029799660,62cyclictest1622-21rtkit-daemon09:32:162
38029799650,60cyclictest1622-21rtkit-daemon10:26:182
162399640,8rtkit-daemon1622-21rtkit-daemon08:15:112
162399630,9rtkit-daemon1622-21rtkit-daemon07:49:142
38029799620,58cyclictest1622-21rtkit-daemon09:19:192
38029799620,57cyclictest1622-21rtkit-daemon07:26:572
162399620,9rtkit-daemon1622-21rtkit-daemon11:21:092
162399620,9rtkit-daemon1622-21rtkit-daemon11:08:232
38029799610,56cyclictest1622-21rtkit-daemon12:16:312
4038672590,5sleep10-21swapper/108:10:351
380305995919,12cyclictest426896-21systemd-udevd09:18:223
380305995912,23cyclictest1753-21gnome-shell09:24:303
162399590,9rtkit-daemon1622-21rtkit-daemon12:04:052
162399590,9rtkit-daemon1622-21rtkit-daemon11:13:243
380305995826,7cyclictest42-21ksoftirqd/308:50:093
380305995712,21cyclictest0-21swapper/311:30:263
162399560,9rtkit-daemon1622-21rtkit-daemon11:27:043
162399560,9rtkit-daemon1622-21rtkit-daemon11:00:472
162399560,8rtkit-daemon1622-21rtkit-daemon12:14:032
380305995520,8cyclictest1753-21gnome-shell12:25:023
162399550,6rtkit-daemon1622-21rtkit-daemon09:22:592
38030599542,27cyclictest1753-21gnome-shell10:36:463
38030599542,26cyclictest1753-21gnome-shell11:05:443
380305995421,8cyclictest1753-21gnome-shell08:11:113
380305995421,8cyclictest1753-21gnome-shell07:22:473
4697982530,6sleep10-21swapper/111:40:211
4072852530,9sleep31753-21gnome-shell08:20:313
38030599532,26cyclictest1753-21gnome-shell09:40:083
38030599532,25cyclictest1753-21gnome-shell10:20:263
380305995320,8cyclictest1753-21gnome-shell10:02:203
38030599531,6cyclictest1753-21gnome-shell09:12:203
38029799530,50cyclictest1622-21rtkit-daemon10:33:372
38029799530,50cyclictest1622-21rtkit-daemon10:33:362
38030599522,8cyclictest1753-21gnome-shell12:06:193
38030599522,25cyclictest1753-21gnome-shell10:40:113
380297995228,20cyclictest1753-21gnome-shell11:10:172
38029799520,47cyclictest1622-21rtkit-daemon07:39:432
3983282510,5sleep3411rcuc/307:55:283
38030599511,22cyclictest456662-21ls10:55:323
38029799510,47cyclictest1622-21rtkit-daemon10:22:592
162399510,6rtkit-daemon1622-21rtkit-daemon09:40:402
4043892500,6sleep2331rcuc/208:15:022
38030599502,25cyclictest1753-21gnome-shell09:08:343
380305995020,9cyclictest1753-21gnome-shell09:57:183
380305995016,13cyclictest438-20systemd-journal10:10:003
38029799500,46cyclictest1622-21rtkit-daemon10:14:292
38029799500,46cyclictest1622-21rtkit-daemon07:41:002
38029799500,46cyclictest1622-21rtkit-daemon07:14:292
38030599492,9cyclictest1753-21gnome-shell08:19:503
38030599492,25cyclictest1753-21gnome-shell12:33:483
38030599492,25cyclictest1753-21gnome-shell11:41:563
38029799481,18cyclictest453876-21/usr/sbin/munin10:50:232
38029799481,18cyclictest453876-21/usr/sbin/munin10:50:232
380291994823,20cyclictest437187-21ntp_states09:50:220
162399480,40rtkit-daemon381996-21munin-node11:18:143
380297994726,16cyclictest3242253-21nautilus11:35:022
38029799470,43cyclictest1622-21rtkit-daemon11:16:202
380305994620,21cyclictest1753-21gnome-shell07:25:463
38029799460,42cyclictest1622-21rtkit-daemon11:37:222
380291994627,14cyclictest457732-21df11:00:170
380305994531,10cyclictest431894-21mktemp09:35:013
380305994525,16cyclictest1753-21gnome-shell07:44:273
38029599455,10cyclictest26-21ksoftirqd/109:25:131
380295994526,15cyclictest3242253-21nautilus08:50:021
380295994526,15cyclictest3242253-21nautilus08:50:021
4866702440,8chrt1753-21gnome-shell12:35:322
3902862440,8sleep01753-21gnome-shell07:35:230
380291994427,12cyclictest426893-21systemd-udevd09:18:240
4578292430,7chrt2197014-21PK-Backend11:00:223
38029799430,39cyclictest1622-21rtkit-daemon11:58:132
380295994329,10cyclictest485150-21apt-get12:35:011
380295994329,10cyclictest1617-10pipewire-media-08:30:171
380295994322,17cyclictest1069-21runrttasks07:50:011
380291994324,7cyclictest13-21ksoftirqd/010:45:010
380291994320,11cyclictest457083-21cron11:00:010
162399430,8rtkit-daemon141rcu_preempt08:40:202
380297994222,14cyclictest470809-21pool-org.gnome.11:45:022
380297994219,20cyclictest453655-21awk10:50:002
380295994222,15cyclictest433530-21sshd09:36:341
380291994230,7cyclictest1753-21gnome-shell10:45:520
380291994224,7cyclictest13-21ksoftirqd/011:30:120
380291994223,7cyclictest13-21ksoftirqd/008:35:200
380305994125,12cyclictest395217-21apt-get07:50:003
380305994123,14cyclictest1-21systemd07:15:013
380305994118,19cyclictest42-21ksoftirqd/310:48:413
380297994123,13cyclictest418008-21users08:50:342
380295994123,14cyclictest428221-21swap09:20:311
38029199414,11cyclictest13-21ksoftirqd/011:43:580
380291994131,6cyclictest13-21ksoftirqd/007:50:570
380291994131,6cyclictest13-21ksoftirqd/007:50:570
380291994128,9cyclictest13-21ksoftirqd/012:35:080
380291994126,11cyclictest420060-21sshd08:59:340
380291994123,7cyclictest13-21ksoftirqd/008:05:350
380291994123,6cyclictest13-21ksoftirqd/010:15:230
380291994112,7cyclictest3991002cat08:00:010
380291994111,6cyclictest4079292latency_hist08:25:020
38030599401,9cyclictest2197014-21PK-Backend11:20:373
38029799401,9cyclictest2197014-21PK-Backend08:33:312
380295994024,12cyclictest441988-21sh10:10:001
380295994022,14cyclictest1-21systemd10:00:011
380295994013,10cyclictest26-21ksoftirqd/108:21:411
380291994024,11cyclictest436160-21cron09:50:000
380291994023,6cyclictest13-21ksoftirqd/012:06:270
380291994022,13cyclictest438-20systemd-journal10:25:340
380291994022,13cyclictest438-20systemd-journal10:06:330
380291994021,7cyclictest13-21ksoftirqd/011:12:190
380291994020,15cyclictest1-21systemd12:35:040
380291994015,6cyclictest141rcu_preempt07:33:300
3869812390,13chrt0-21swapper/107:25:271
38029599392,6cyclictest38435150journal-offline07:20:021
380295993923,11cyclictest449562-21/usr/sbin/munin10:35:011
380295993923,11cyclictest449562-21/usr/sbin/munin10:35:011
380295993921,13cyclictest438-20systemd-journal07:11:341
380295993918,18cyclictest389894-21acpi07:35:121
38029199397,13cyclictest141rcu_preempt10:35:070
380291993928,7cyclictest1753-21gnome-shell12:14:300
380291993928,7cyclictest1069-21runrttasks08:02:400
380291993927,8cyclictest416185-21cat08:45:330
380291993927,8cyclictest416185-21cat08:45:320
380291993925,10cyclictest421638-21sshd09:02:340
380291993922,6cyclictest13-21ksoftirqd/011:55:170
380291993922,6cyclictest13-21ksoftirqd/008:25:120
380291993922,5cyclictest13-21ksoftirqd/009:11:520
380291993922,13cyclictest438902-21ntpq09:55:230
380291993921,6cyclictest13-21ksoftirqd/007:46:030
380291993919,16cyclictest1753-21gnome-shell10:10:180
380291993914,6cyclictest141rcu_preempt10:04:230
4684602380,4sleep0468464-21python311:35:290
38030599381,9cyclictest2197014-21PK-Backend07:18:043
38029799382,8cyclictest1753-21gnome-shell07:31:172
38029799382,13cyclictest1753-21gnome-shell11:47:582
38029799382,12cyclictest1753-21gnome-shell09:29:442
38029799381,20cyclictest437169-21ntp_kernel_pll_09:50:192
38029799380,34cyclictest1622-21rtkit-daemon12:20:252
38029599385,22cyclictest417092-21grep08:50:161
380295993824,9cyclictest446407-21cut10:20:221
380295993824,10cyclictest477404-21sshd12:08:341
380295993822,12cyclictest855-21dbus-daemon12:15:011
380295993822,12cyclictest452956-21sshd10:44:331
380295993822,12cyclictest445094-21sshd10:15:341
380295993820,13cyclictest26-21ksoftirqd/109:30:241
380295993819,14cyclictest461218-40apt-check11:12:171
380295993819,13cyclictest1-21systemd09:05:011
380295993817,17cyclictest414463-21timerwakeupswit08:40:321
380295993812,22cyclictest884-21gdbus09:18:231
38029199386,13cyclictest141rcu_preempt09:42:230
380291993823,11cyclictest447056-21sshd10:23:340
380291993820,13cyclictest429517-21pkexec09:27:270
380291993819,6cyclictest13-21ksoftirqd/007:20:100
380291993813,6cyclictest141rcu_preempt09:38:050
4732112370,5chrt0-21swapper/111:55:011
380297993725,8cyclictest859-21NetworkManager08:07:342
380297993720,13cyclictest452723-21sshd10:41:342
38029599379,9cyclictest0-21swapper/110:57:021
38029599378,9cyclictest437639-21runrttasks09:51:491
380295993726,7cyclictest384568-21cpuspeed_turbos07:20:111
380295993721,11cyclictest464372-21swap11:20:301
380295993721,11cyclictest450909-21vmstat10:35:351
380295993720,13cyclictest855-21dbus-daemon11:04:591
38029599371,32cyclictest443020-21sh10:10:301
380295993710,23cyclictest468090-21sed11:35:221
38029199374,5cyclictest181irq_work/008:31:540
380291993730,4cyclictest2197014-21PK-Backend10:54:050
380291993730,4cyclictest2197014-21PK-Backend10:54:050
38029199372,12cyclictest414650-40apt-check08:42:200
380291993721,12cyclictest471358-21sshd11:49:340
380291993720,12cyclictest884-21gdbus07:40:030
380291993720,12cyclictest438-20systemd-journal08:54:550
380291993719,13cyclictest3242255-21gdbus11:25:010
380291993713,6cyclictest141rcu_preempt10:33:240
380291993713,6cyclictest141rcu_preempt10:33:240
162399370,28rtkit-daemon2197014-21PK-Backend10:31:513
162399370,28rtkit-daemon2197014-21PK-Backend10:31:503
162399370,27rtkit-daemon2197014-21PK-Backend12:20:250
38030599363,10cyclictest141rcu_preempt12:16:043
38030599362,18cyclictest42-21ksoftirqd/311:47:593
38029799362,14cyclictest1753-21gnome-shell09:35:322
380297993613,6cyclictest141rcu_preempt08:47:532
380297993613,6cyclictest141rcu_preempt08:47:532
38029599368,23cyclictest453995-21cpuspeed_turbos10:50:101
38029599368,23cyclictest453995-21cpuspeed_turbos10:50:101
38029599364,5cyclictest141rcu_preempt11:10:001
380295993622,9cyclictest485910-21cpuspeed_turbos12:35:161
380295993621,11cyclictest440327-21basename10:00:191
380295993619,13cyclictest465534-21sshd11:29:331
380295993619,12cyclictest473758-21expr11:55:161
380295993618,13cyclictest970-21ntpd07:55:251
380295993615,17cyclictest26-21ksoftirqd/107:51:501
380295993615,17cyclictest26-21ksoftirqd/107:51:501
380295993612,7cyclictest462350-21missed_timers11:15:231
380291993625,8cyclictest13-21ksoftirqd/009:35:020
38029199362,30cyclictest1753-21gnome-shell09:21:390
380291993621,11cyclictest483153-21sshd12:27:340
380291993620,5cyclictest13-21ksoftirqd/011:50:100
380291993619,13cyclictest475908-21awk12:00:290
380291993618,6cyclictest13-21ksoftirqd/011:25:110
380291993616,17cyclictest1753-21gnome-shell07:15:310
38030599359,21cyclictest0-21swapper/309:00:193
380305993523,8cyclictest1753-21gnome-shell08:27:223
380305993522,10cyclictest42-21ksoftirqd/312:03:043
38030599351,11cyclictest2197014-21PK-Backend09:30:023
38029799352,10cyclictest1753-21gnome-shell10:57:012
380297993518,13cyclictest2181-21gdbus07:25:012
38029799351,19cyclictest175350gnome-shell07:55:202
38029599358,6cyclictest26-21ksoftirqd/109:40:351
38029599354,11cyclictest26-21ksoftirqd/108:36:111
380295993525,7cyclictest1-21systemd12:04:471
380295993520,11cyclictest466180-21expr11:30:151
380295993520,11cyclictest419080-21basename08:55:191
380295993519,12cyclictest855-21dbus-daemon08:03:351
380295993519,12cyclictest424853-21cat09:10:351
380295993519,12cyclictest408083-21ls08:25:141
380295993519,11cyclictest482143-21python312:20:321
380295993518,13cyclictest388658-21ntp_states07:30:301
380295993510,13cyclictest0-21swapper/109:05:131
380291993528,4cyclictest2197014-21PK-Backend09:08:460
380291993527,5cyclictest2197014-21PK-Backend08:13:300
38029199352,30cyclictest0-21swapper/012:17:220
380291993520,5cyclictest13-21ksoftirqd/007:26:350
162399350,28rtkit-daemon0-21swapper/209:08:532
162399350,27rtkit-daemon1622-21rtkit-daemon08:35:232
380305993422,8cyclictest1753-21gnome-shell11:55:533
380305993422,8cyclictest1753-21gnome-shell08:56:113
380305993418,11cyclictest42-21ksoftirqd/307:35:313
380305993410,20cyclictest1753-21gnome-shell08:50:013
380305993410,20cyclictest1753-21gnome-shell08:50:013
380297993420,10cyclictest441670-21seq10:06:012
380297993417,12cyclictest438-20systemd-journal12:25:032
38029799341,4cyclictest0-21swapper/209:14:412
380297993412,9cyclictest141rcu_preempt08:56:452
380295993419,11cyclictest480251-21ntp_kernel_pll_12:15:241
380295993417,12cyclictest970-21ntpd07:40:251
38029199341,29cyclictest2197014-21PK-Backend08:17:540
38029199341,28cyclictest2197014-21PK-Backend11:09:140
38030599332,27cyclictest1753-21gnome-shell08:40:183
380305993320,9cyclictest42-21ksoftirqd/307:32:263
380305993315,14cyclictest1753-21gnome-shell12:28:423
380297993325,5cyclictest2197014-21PK-Backend08:20:232
380297993322,7cyclictest1753-21gnome-shell12:06:292
380297993322,7cyclictest1069-21runrttasks10:01:562
380297993319,11cyclictest34-21ksoftirqd/210:36:492
38029799331,15cyclictest141rcu_preempt11:25:162
380297993310,19cyclictest1753-21gnome-shell09:45:112
38029599332,28cyclictest1069-21runrttasks10:29:001
380295993319,10cyclictest436276-21sed09:50:021
380295993316,12cyclictest401731-21netstat08:05:241
380297993224,5cyclictest2197014-21PK-Backend09:57:192
38029799321,27cyclictest2197014-21PK-Backend07:17:132
380297993210,18cyclictest1753-21gnome-shell07:51:142
380297993210,18cyclictest1753-21gnome-shell07:51:142
380295993215,12cyclictest404621-21cpuspeed_turbos08:15:151
380295993211,17cyclictest1753-21gnome-shell10:45:231
380305993117,10cyclictest42-21ksoftirqd/311:50:133
38029799311,5cyclictest2197014-21PK-Backend09:01:492
38029599316,5cyclictest141rcu_preempt12:25:181
380295993117,10cyclictest471290-21NetworkManager11:48:411
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional