You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-25 - 14:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Mon Nov 25, 2024 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
64277921070,6sleep1639099-21kworker/1:222:10:011
5774902930,8sleep30-21swapper/319:05:203
6587332890,6sleep22197014-21PK-Backend22:55:142
162399810,11rtkit-daemon1753-21gnome-shell20:24:401
162399800,9rtkit-daemon869-21irqbalance19:55:341
162399800,8rtkit-daemon580362-21munin-node19:26:351
162399800,8rtkit-daemon580362-21munin-node19:26:341
162399800,11rtkit-daemon2196306-21snap-store22:51:211
162399790,9rtkit-daemon881143-21gmain00:36:001
57807527863,9sleep10-21swapper/119:07:171
162399780,7rtkit-daemon707415-21gmain20:29:181
162399770,8rtkit-daemon707415-21gmain23:02:111
162399770,8rtkit-daemon643930-21python322:10:331
162399770,8rtkit-daemon580362-21munin-node20:44:041
162399770,5rtkit-daemon854-21avahi-daemon20:50:091
162399770,5rtkit-daemon854-21avahi-daemon20:50:091
162399760,4rtkit-daemon0-21swapper/122:45:241
162399760,4rtkit-daemon0-21swapper/122:45:241
162399760,4rtkit-daemon0-21swapper/120:31:221
162399760,3rtkit-daemon0-21swapper/121:58:351
57809927548,17sleep00-21swapper/019:07:350
162399750,9rtkit-daemon2287-21gmain21:46:371
162399750,8rtkit-daemon856-21gmain22:19:381
162399750,8rtkit-daemon580362-21munin-node20:12:101
162399750,5rtkit-daemon2197014-21PK-Backend22:01:281
162399750,5rtkit-daemon2197014-21PK-Backend19:31:281
57797927459,9sleep20-21swapper/219:06:002
162399730,4rtkit-daemon0-21swapper/119:48:441
162399700,4rtkit-daemon0-21swapper/100:10:381
162399640,10rtkit-daemon1622-21rtkit-daemon21:07:071
162399630,9rtkit-daemon1622-21rtkit-daemon21:15:441
162399620,9rtkit-daemon1622-21rtkit-daemon00:36:480
6808542610,5sleep0161rcuc/000:00:200
162399600,9rtkit-daemon1622-21rtkit-daemon21:47:390
162399600,9rtkit-daemon1622-21rtkit-daemon19:18:461
162399580,8rtkit-daemon1622-21rtkit-daemon23:44:221
162399580,8rtkit-daemon1622-21rtkit-daemon23:44:211
162399570,9rtkit-daemon1622-21rtkit-daemon00:21:351
162399570,8rtkit-daemon1622-21rtkit-daemon22:55:161
162399570,8rtkit-daemon1622-21rtkit-daemon21:33:011
162399570,8rtkit-daemon1622-21rtkit-daemon00:35:001
162399570,22rtkit-daemon1622-21rtkit-daemon00:11:390
162399560,9rtkit-daemon1622-21rtkit-daemon20:17:061
57841899551,13cyclictest6278432/usr/sbin/munin21:25:231
162399550,8rtkit-daemon1622-21rtkit-daemon23:58:490
162399550,8rtkit-daemon1622-21rtkit-daemon22:37:111
162399550,8rtkit-daemon1622-21rtkit-daemon21:43:301
162399550,8rtkit-daemon1622-21rtkit-daemon19:20:250
162399550,39rtkit-daemon0-21swapper/019:11:520
6534142540,5sleep00-21swapper/022:40:020
578418995419,11cyclictest637082-21seq21:52:361
578418995415,13cyclictest617955-21apt-check20:57:171
578418995415,10cyclictest647727-21sshd22:21:511
578418995413,13cyclictest666162-21memory23:15:211
162399540,8rtkit-daemon1622-21rtkit-daemon20:03:381
162399540,21rtkit-daemon2197014-21PK-Backend23:43:160
162399540,21rtkit-daemon2197014-21PK-Backend23:43:160
578418995315,12cyclictest648629-21df_inode22:25:151
162399530,8rtkit-daemon1622-21rtkit-daemon19:10:401
162399530,7rtkit-daemon1622-21rtkit-daemon21:34:400
578418995217,12cyclictest3242254-21gmain20:50:001
162399520,8rtkit-daemon1622-21rtkit-daemon23:00:060
162399520,8rtkit-daemon1622-21rtkit-daemon21:26:440
162399520,8rtkit-daemon1622-21rtkit-daemon00:15:371
162399520,23rtkit-daemon1622-21rtkit-daemon21:14:510
6579602510,5sleep20-21swapper/222:50:352
6328732510,5sleep32197014-21PK-Backend21:40:203
578418995118,9cyclictest619294-21which21:00:271
578418995116,12cyclictest589955-21cpuspeed_turbos19:40:141
578418995114,12cyclictest589286-40lsb_release19:36:261
578418995112,16cyclictest3242254-21gmain00:00:001
162399510,6rtkit-daemon1622-21rtkit-daemon20:30:080
578418995013,12cyclictest650118-21grep22:30:181
578418995012,11cyclictest689816-21if_enp2s000:25:191
162399500,6rtkit-daemon1622-21rtkit-daemon23:35:120
57841899494,19cyclictest1826-21JS14:31:441
578418994917,10cyclictest26-21ksoftirqd/122:40:261
578418994915,11cyclictest662254-21cut23:05:121
578418994821,8cyclictest64850irq/126-i91523:10:351
578418994821,8cyclictest1753-21gnome-shell00:00:031
578418994814,10cyclictest1617-10pipewire-media-21:40:001
578414994832,12cyclictest687063-21apt-get00:20:000
6681862470,8chrt0-21swapper/123:20:291
578414994726,17cyclictest13-21ksoftirqd/019:36:290
5892522460,8sleep342-21ksoftirqd/319:36:203
57841899462,7cyclictest609909-21diskstats20:35:191
57841899462,7cyclictest609909-21diskstats20:35:181
578426994526,16cyclictest680152-40systemctl00:00:013
578414994533,8cyclictest1069-21runrttasks20:48:380
578414994530,11cyclictest670895-21cpuspeed_turbos23:30:180
578414994527,7cyclictest13-21ksoftirqd/021:00:000
6451752440,18chrt42-21ksoftirqd/322:15:183
5911442440,5sleep0161rcuc/019:42:530
57841899442,8cyclictest593729-21hddtemp_smartct19:50:231
57841899442,8cyclictest593729-21hddtemp_smartct19:50:231
578418994428,12cyclictest918-21gdbus21:21:571
578418994428,12cyclictest918-21gdbus21:21:571
57841899442,11cyclictest677777-21/usr/sbin/munin23:50:261
578418994419,6cyclictest623427-21latency_hist21:15:021
578418994417,12cyclictest670507-21latency_hist23:30:031
162399440,23rtkit-daemon1622-21rtkit-daemon21:52:430
162399440,21rtkit-daemon0-21swapper/019:26:470
162399440,21rtkit-daemon0-21swapper/019:26:470
578426994324,14cyclictest42-21ksoftirqd/321:59:163
578426994315,23cyclictest42-21ksoftirqd/323:00:193
57841899431,38cyclictest1753-21gnome-shell23:39:351
578414994311,12cyclictest141rcu_preempt22:25:120
162399430,23rtkit-daemon1622-21rtkit-daemon23:13:100
578426994225,13cyclictest623455-21pool-org.gnome.21:15:013
578426994210,28cyclictest1753-21gnome-shell21:18:063
57841899422,7cyclictest683004-21python300:05:291
57841499428,12cyclictest141rcu_preempt23:05:140
578414994216,12cyclictest13-21ksoftirqd/022:04:340
162399420,22rtkit-daemon2197014-21PK-Backend23:27:460
578426994113,8cyclictest1753-21gnome-shell23:41:533
578426994113,8cyclictest1753-21gnome-shell23:41:533
57842299417,5cyclictest141rcu_preempt00:15:162
578422994122,14cyclictest682499-21df_inode00:05:142
578422994118,19cyclictest580122-21cron19:14:592
57842299411,35cyclictest691972-21gunzip00:30:312
162399410,35rtkit-daemon13-21ksoftirqd/020:39:020
162399410,35rtkit-daemon13-21ksoftirqd/020:39:020
578426994025,11cyclictest42-21ksoftirqd/321:20:243
578426994025,11cyclictest42-21ksoftirqd/321:20:243
57842699402,33cyclictest667792-21irqstats23:20:193
578426994023,13cyclictest666440-21sensors_temp23:15:263
578426994014,8cyclictest1753-21gnome-shell21:48:323
57842699401,34cyclictest3242255-21gdbus21:35:023
57842299407,13cyclictest141rcu_preempt21:02:352
57842299406,8cyclictest141rcu_preempt20:45:122
578422994025,11cyclictest600040-21mktemp20:10:022
578422994023,6cyclictest34-21ksoftirqd/200:24:052
578422994023,13cyclictest34-21ksoftirqd/220:14:222
578422994022,13cyclictest34-21ksoftirqd/221:16:012
57842299401,7cyclictest854-21avahi-daemon21:38:092
57841899400,36cyclictest1622-21rtkit-daemon20:06:161
57841499409,11cyclictest141rcu_preempt22:45:150
57841499409,11cyclictest141rcu_preempt22:45:140
57841499402,8cyclictest1753-21gnome-shell00:27:550
578414994022,6cyclictest13-21ksoftirqd/023:50:010
578414994022,14cyclictest855-21dbus-daemon21:40:010
578414994022,13cyclictest13-21ksoftirqd/019:18:300
6801842390,9sleep2680141-40updatedb.mlocat00:00:012
57842699398,5cyclictest141rcu_preempt00:15:183
57842699397,11cyclictest42-21ksoftirqd/320:37:293
57842699397,11cyclictest42-21ksoftirqd/320:37:293
57842699393,17cyclictest1753-21gnome-shell23:05:083
578426993923,12cyclictest596513-21ls20:00:003
57842699392,13cyclictest1753-21gnome-shell20:03:093
578426993914,21cyclictest1753-21gnome-shell00:13:523
57842699391,35cyclictest616312-21dpkg20:55:023
57842699391,35cyclictest616312-21dpkg20:55:013
578426993911,9cyclictest859-21NetworkManager22:00:293
57842699391,11cyclictest0-21swapper/322:50:283
57842299398,5cyclictest141rcu_preempt22:10:122
57842299396,9cyclictest34-21ksoftirqd/223:00:192
57842299394,6cyclictest141rcu_preempt20:20:222
57842299392,7cyclictest1753-21gnome-shell21:45:052
578422993922,7cyclictest3242254-21gmain23:37:562
578422993922,12cyclictest34-21ksoftirqd/222:37:452
578422993921,6cyclictest34-21ksoftirqd/219:28:102
578422993921,6cyclictest34-21ksoftirqd/219:28:092
57842299392,13cyclictest2197014-21PK-Backend19:50:552
57842299392,13cyclictest2197014-21PK-Backend19:50:552
578422993921,13cyclictest438-20systemd-journal19:21:522
578422993913,22cyclictest641705-21/usr/sbin/munin22:05:172
57841499393,12cyclictest141rcu_preempt00:05:380
57841499392,9cyclictest642775-21cron22:10:000
57841499392,7cyclictest1753-21gnome-shell23:53:010
578414993925,10cyclictest601992-21sshd20:14:520
57841499392,33cyclictest1753-21gnome-shell23:21:360
578414993923,12cyclictest626161-21ntpq21:20:250
578414993923,12cyclictest626161-21ntpq21:20:250
578414993922,13cyclictest586834-21ntpq19:30:260
578414993917,13cyclictest13-21ksoftirqd/021:55:470
57841499391,15cyclictest2197014-21PK-Backend22:59:350
578414993911,13cyclictest141rcu_preempt21:00:550
6151052380,6chrt2197014-21PK-Backend20:50:172
6151052380,6chrt2197014-21PK-Backend20:50:162
57842699386,12cyclictest42-21ksoftirqd/300:04:373
57842699382,7cyclictest0-21swapper/320:05:313
578426993826,8cyclictest1753-21gnome-shell22:44:403
578426993823,11cyclictest587553-21sh19:35:003
578426993822,12cyclictest875-21polkitd19:12:173
578426993821,13cyclictest647606-21users22:20:323
578426993819,14cyclictest994-21lldpd22:08:493
578426993819,14cyclictest42-21ksoftirqd/319:45:283
578426993818,7cyclictest1753-21gnome-shell21:35:153
57842699381,19cyclictest1753-21gnome-shell22:37:533
578426993811,8cyclictest1617-10pipewire-media-19:20:253
578426993810,5cyclictest141rcu_preempt00:05:253
578426993810,11cyclictest42-21ksoftirqd/320:23:083
57842299387,5cyclictest141rcu_preempt23:21:262
57842299387,5cyclictest141rcu_preempt21:07:082
57842299387,5cyclictest141rcu_preempt19:44:592
57842299387,10cyclictest34-21ksoftirqd/220:30:522
57842299386,10cyclictest34-21ksoftirqd/223:34:102
578422993825,8cyclictest1826-21JS14:31:442
578422993824,10cyclictest34-21ksoftirqd/223:29:202
578422993821,13cyclictest34-21ksoftirqd/219:30:282
578422993821,12cyclictest692845-21apt00:35:142
578422993820,6cyclictest34-21ksoftirqd/220:36:482
578422993820,6cyclictest34-21ksoftirqd/220:36:482
578422993818,15cyclictest854-21avahi-daemon00:00:062
57842299381,13cyclictest2197014-21PK-Backend23:18:192
57842299381,13cyclictest2197014-21PK-Backend20:55:112
57842299381,12cyclictest141rcu_preempt21:20:262
57842299381,12cyclictest141rcu_preempt21:20:252
57842299381,12cyclictest141rcu_preempt19:35:212
57842299381,11cyclictest2197014-21PK-Backend22:15:592
57842299381,10cyclictest0-21swapper/221:10:142
57841499382,7cyclictest1753-21gnome-shell22:34:040
578414993824,10cyclictest687063-21apt-get00:20:030
578414993824,10cyclictest603766-21turbostat20:19:590
578414993822,12cyclictest644193-21sshd22:10:580
578414993822,12cyclictest633715-21sshd21:43:510
57841499382,12cyclictest141rcu_preempt20:50:280
57841499382,12cyclictest141rcu_preempt20:50:270
57841499382,12cyclictest141rcu_preempt19:45:120
578414993820,13cyclictest605492-21/usr/sbin/munin20:25:020
578414993819,13cyclictest438-20systemd-journal20:00:020
57841499381,13cyclictest647110-21chrt22:20:220
57841499381,12cyclictest141rcu_preempt20:25:060
57842699379,9cyclictest0-21swapper/321:01:283
57842699379,7cyclictest0-21swapper/320:56:123
57842699378,8cyclictest859-21NetworkManager19:16:373
578426993725,8cyclictest693327-21irqstats00:35:243
578426993723,10cyclictest42-21ksoftirqd/321:50:473
578426993721,6cyclictest593641-21expr19:50:173
578426993721,6cyclictest593641-21expr19:50:173
578426993721,12cyclictest2696378-21snapd22:34:473
578426993717,17cyclictest42-21ksoftirqd/320:28:393
578426993710,22cyclictest590197-21cat19:40:163
578426993710,10cyclictest42-21ksoftirqd/323:49:163
57842299378,5cyclictest141rcu_preempt23:53:022
57842299377,5cyclictest141rcu_preempt21:30:222
57842299376,5cyclictest141rcu_preempt21:58:102
57842299372,8cyclictest1753-21gnome-shell20:16:162
57842299372,7cyclictest0-21swapper/222:02:012
57842299372,14cyclictest683868-21apt00:10:122
578422993720,5cyclictest34-21ksoftirqd/220:28:422
578422993719,13cyclictest438-20systemd-journal20:04:592
578422993718,15cyclictest34-21ksoftirqd/219:48:222
57842299371,14cyclictest2197014-21PK-Backend22:25:182
578414993724,8cyclictest666474-21ls23:15:280
57841499372,12cyclictest141rcu_preempt22:15:260
578414993719,13cyclictest595839-21pluginstate19:55:280
6551882360,9sleep22197014-21PK-Backend22:45:112
6551882360,9sleep22197014-21PK-Backend22:45:112
57842699365,11cyclictest42-21ksoftirqd/323:53:393
578426993625,8cyclictest691770-21grep00:30:263
578426993624,8cyclictest655311-21tr22:45:143
578426993624,8cyclictest655311-21tr22:45:143
578426993621,10cyclictest42-21ksoftirqd/321:05:113
578426993619,12cyclictest994-21lldpd23:28:433
578426993615,17cyclictest42-21ksoftirqd/300:25:293
578426993610,7cyclictest42-21ksoftirqd/320:19:543
57842299366,5cyclictest141rcu_preempt22:31:582
57842299364,9cyclictest34-21ksoftirqd/222:20:462
57842299364,8cyclictest34-21ksoftirqd/221:50:122
57842299362,7cyclictest595453-21hddtemp_smartct19:55:192
57842299362,5cyclictest141rcu_preempt19:15:052
578422993616,15cyclictest34-21ksoftirqd/222:40:082
578422993610,6cyclictest141rcu_preempt00:25:132
578414993626,7cyclictest1069-21runrttasks21:19:440
578414993625,7cyclictest1069-21runrttasks21:09:550
578414993622,10cyclictest612545-21sshd20:42:520
578414993620,12cyclictest599393-21python320:05:280
578426993523,8cyclictest6643002chrt23:10:253
578426993523,8cyclictest411rcuc/323:32:173
578426993522,9cyclictest1753-21gnome-shell21:26:423
578426993521,10cyclictest42-21ksoftirqd/322:57:443
578426993520,11cyclictest688672-21cat00:20:303
578426993520,11cyclictest42-21ksoftirqd/320:45:163
57842299355,5cyclictest141rcu_preempt23:40:262
57842299355,5cyclictest141rcu_preempt23:40:252
578422993525,7cyclictest1753-21gnome-shell23:07:572
57842299351,7cyclictest854-21avahi-daemon21:40:492
57842299351,6cyclictest2197014-21PK-Backend23:10:282
57842299351,15cyclictest2197014-21PK-Backend20:40:392
578418993519,12cyclictest676267-21pkexec23:45:281
57841499358,6cyclictest141rcu_preempt22:42:420
578414993518,13cyclictest593856-21cat19:50:250
578414993518,13cyclictest593856-21cat19:50:240
57842699347,5cyclictest141rcu_preempt20:15:003
578426993422,8cyclictest643046-21acpi22:10:143
57842699342,11cyclictest585232-21python319:25:283
57842699342,11cyclictest585232-21python319:25:283
578426993419,11cyclictest612008-21grep20:40:253
57842699341,29cyclictest2197014-21PK-Backend20:34:363
578426993410,20cyclictest1753-21gnome-shell23:37:173
57842299341,28cyclictest676075-21chmod23:45:022
578418993419,11cyclictest584272-21cut19:25:021
57841499341,10cyclictest13-21ksoftirqd/000:34:330
578426993317,12cyclictest648726-21interrupts22:25:203
578414993321,8cyclictest1753-21gnome-shell22:50:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional