You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 03:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Apr 02, 2025 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
368276821280,7sleep10-21swapper/123:20:011
360154529378,9sleep30-21swapper/319:07:533
36510332820,5sleep20-21swapper/221:30:312
162399780,6rtkit-daemon2197014-21PK-Backend00:18:491
162399750,6rtkit-daemon2197014-21PK-Backend00:02:321
162399750,5rtkit-daemon0-21swapper/122:12:301
162399750,5rtkit-daemon0-21swapper/122:12:301
360141427447,17sleep10-21swapper/119:06:091
360135127459,9sleep20-21swapper/219:05:302
162399740,6rtkit-daemon2197014-21PK-Backend23:24:461
162399720,5rtkit-daemon2197014-21PK-Backend22:51:451
162399720,5rtkit-daemon2197014-21PK-Backend22:51:451
3601859996938,9cyclictest42-21ksoftirqd/323:25:003
3601859996921,23cyclictest42-21ksoftirqd/320:15:003
360156726949,14sleep00-21swapper/019:08:100
3601859996019,9cyclictest42-21ksoftirqd/300:36:573
36535922590,14sleep1251rcuc/121:40:111
162399590,9rtkit-daemon1622-21rtkit-daemon20:16:073
3601859995720,20cyclictest42-21ksoftirqd/300:00:033
162399570,8rtkit-daemon1622-21rtkit-daemon20:19:352
36024232560,7sleep30-21swapper/319:10:173
36500002550,7sleep20-21swapper/221:30:032
36500002550,7sleep20-21swapper/221:30:022
360184399559,19cyclictest13-21ksoftirqd/022:51:560
360184399559,19cyclictest13-21ksoftirqd/022:51:560
3601859995419,19cyclictest42-21ksoftirqd/319:29:593
3601859995415,20cyclictest42-21ksoftirqd/321:05:023
3601859995326,7cyclictest141rcu_preempt20:30:183
3601859995319,10cyclictest42-21ksoftirqd/320:10:023
360185999531,48cyclictest1622-21rtkit-daemon21:27:133
360185999531,48cyclictest1622-21rtkit-daemon21:27:123
360185999531,29cyclictest42-21ksoftirqd/322:05:003
360185599530,49cyclictest1622-21rtkit-daemon21:56:162
3601850995323,17cyclictest26-21ksoftirqd/123:30:021
3601850995323,17cyclictest26-21ksoftirqd/123:30:011
3601843995333,8cyclictest13-21ksoftirqd/023:30:180
3601843995322,18cyclictest13-21ksoftirqd/022:15:020
3601843995322,18cyclictest13-21ksoftirqd/022:15:010
3601843995321,16cyclictest13-21ksoftirqd/023:52:150
3601843995320,18cyclictest13-21ksoftirqd/022:30:010
162399530,8rtkit-daemon1622-21rtkit-daemon21:17:003
37034512520,7chrt2197014-21PK-Backend00:30:063
36027952520,14sleep013-21ksoftirqd/019:10:250
360185999529,22cyclictest42-21ksoftirqd/300:15:163
360185999524,10cyclictest42-21ksoftirqd/323:20:023
3601859995217,9cyclictest42-21ksoftirqd/322:32:553
3601859995217,9cyclictest42-21ksoftirqd/322:32:553
360185999521,20cyclictest42-21ksoftirqd/321:30:223
3601850995223,24cyclictest3641278-40lsb_release21:00:271
36778352510,12chrt421ksoftirqd/323:00:233
360185999513,8cyclictest42-21ksoftirqd/300:15:013
360185999513,8cyclictest42-21ksoftirqd/300:15:013
3601859995117,20cyclictest42-21ksoftirqd/321:55:003
360185599510,48cyclictest1622-21rtkit-daemon20:23:242
360185599510,46cyclictest1622-21rtkit-daemon22:00:042
360185999509,23cyclictest42-21ksoftirqd/321:45:003
360185999501,32cyclictest3687387-21/usr/sbin/munin23:35:183
3601850995022,12cyclictest855-21dbus-daemon21:25:001
3601850995018,23cyclictest26-21ksoftirqd/122:00:241
3601843995021,9cyclictest13-21ksoftirqd/022:43:120
360184399501,26cyclictest1795-21gmain20:15:190
360184399501,22cyclictest3649851-21cron21:30:000
360184399501,22cyclictest3649851-21cron21:30:000
360185999491,6cyclictest2197014-21PK-Backend21:20:463
3601859994914,18cyclictest42-21ksoftirqd/322:50:153
3601859994914,18cyclictest42-21ksoftirqd/322:50:143
3601859994911,18cyclictest42-21ksoftirqd/320:57:273
3601859994911,18cyclictest42-21ksoftirqd/320:57:273
3601843994923,8cyclictest13-21ksoftirqd/000:00:260
3601843994915,19cyclictest13-21ksoftirqd/020:30:010
3601843994914,11cyclictest13-21ksoftirqd/000:00:020
360184399491,30cyclictest13-21ksoftirqd/021:05:260
360184399491,30cyclictest13-21ksoftirqd/021:05:250
3601859994812,19cyclictest42-21ksoftirqd/319:56:113
360185999481,12cyclictest42-21ksoftirqd/322:12:143
360185999481,12cyclictest42-21ksoftirqd/322:12:133
3601859994811,19cyclictest42-21ksoftirqd/320:25:033
360184399484,12cyclictest13-21ksoftirqd/021:00:040
3601843994822,8cyclictest13-21ksoftirqd/022:21:380
3601843994819,9cyclictest13-21ksoftirqd/023:10:090
3601843994818,8cyclictest13-21ksoftirqd/023:25:030
3601843994818,8cyclictest13-21ksoftirqd/023:25:030
3601843994818,15cyclictest13-21ksoftirqd/023:39:560
3601859994714,19cyclictest42-21ksoftirqd/323:29:123
3601859994714,19cyclictest42-21ksoftirqd/323:29:123
3601859994714,10cyclictest42-21ksoftirqd/319:30:243
360185999471,24cyclictest860997-21snmpd23:30:073
360185999471,19cyclictest42-21ksoftirqd/321:10:183
360184399478,22cyclictest13-21ksoftirqd/000:39:460
360184399475,9cyclictest13-21ksoftirqd/022:50:020
3601843994718,16cyclictest13-21ksoftirqd/023:20:230
3601843994717,8cyclictest13-21ksoftirqd/021:36:330
360184399471,6cyclictest3647724-21chrt21:20:020
3601843994716,14cyclictest13-21ksoftirqd/021:47:490
3601843994714,9cyclictest13-21ksoftirqd/019:30:250
3601843994712,18cyclictest13-21ksoftirqd/020:25:020
360184399471,12cyclictest3661537-21taskset22:10:010
3601843994711,18cyclictest13-21ksoftirqd/020:06:330
3601843994710,11cyclictest13-21ksoftirqd/019:27:150
360185999469,25cyclictest141rcu_preempt19:36:203
360185999469,19cyclictest42-21ksoftirqd/322:46:353
360185999467,19cyclictest42-21ksoftirqd/300:01:323
360185999465,22cyclictest141rcu_preempt22:07:543
360185999463,13cyclictest42-21ksoftirqd/323:15:033
3601859994620,9cyclictest42-21ksoftirqd/320:25:023
3601859994615,8cyclictest42-21ksoftirqd/323:10:003
3601859994613,10cyclictest42-21ksoftirqd/321:50:023
3601859994612,19cyclictest42-21ksoftirqd/320:00:253
3601859994610,6cyclictest42-21ksoftirqd/300:20:513
3601850994619,16cyclictest26-21ksoftirqd/122:10:031
360184399468,19cyclictest13-21ksoftirqd/021:10:170
3601843994620,15cyclictest13-21ksoftirqd/020:00:340
3601843994612,19cyclictest13-21ksoftirqd/023:50:000
3601843994612,10cyclictest13-21ksoftirqd/019:45:160
3601843994611,18cyclictest13-21ksoftirqd/022:38:490
36921182450,7sleep32197014-21PK-Backend23:50:183
360185999458,16cyclictest1753-21gnome-shell21:56:293
360185999458,14cyclictest42-21ksoftirqd/320:46:313
360185999452,21cyclictest42-21ksoftirqd/319:50:353
3601859994516,6cyclictest141rcu_preempt21:07:313
3601859994516,6cyclictest141rcu_preempt21:07:303
3601859994515,9cyclictest42-21ksoftirqd/322:40:073
3601859994513,18cyclictest42-21ksoftirqd/322:19:363
3601859994511,7cyclictest42-21ksoftirqd/322:58:373
360185999451,16cyclictest42-21ksoftirqd/320:45:023
360185999451,12cyclictest42-21ksoftirqd/319:40:183
360185999451,12cyclictest42-21ksoftirqd/319:40:183
360185599452,15cyclictest1753-21gnome-shell23:55:022
3601855994514,10cyclictest141rcu_preempt23:28:472
3601855994514,10cyclictest141rcu_preempt23:28:472
360184399459,9cyclictest13-21ksoftirqd/020:50:260
360184399456,20cyclictest13-21ksoftirqd/020:55:380
360184399456,20cyclictest13-21ksoftirqd/020:55:370
360184399452,18cyclictest0-21swapper/021:22:230
3601843994521,10cyclictest13-21ksoftirqd/019:41:200
3601843994521,10cyclictest13-21ksoftirqd/019:41:200
3601843994516,17cyclictest13-21ksoftirqd/021:57:490
3601843994515,16cyclictest13-21ksoftirqd/000:10:590
3601843994515,16cyclictest13-21ksoftirqd/000:10:590
3601843994513,10cyclictest13-21ksoftirqd/023:15:120
3601843994512,10cyclictest13-21ksoftirqd/022:15:110
360184399451,12cyclictest13-21ksoftirqd/023:05:000
360185999447,19cyclictest42-21ksoftirqd/320:35:243
360185999447,17cyclictest42-21ksoftirqd/322:29:423
360185999444,18cyclictest42-21ksoftirqd/319:15:173
360185999441,8cyclictest42-21ksoftirqd/320:50:313
360185999441,20cyclictest42-21ksoftirqd/319:21:153
360185999441,19cyclictest42-21ksoftirqd/322:39:593
3601855994424,15cyclictest3242253-21nautilus21:00:032
3601850994428,11cyclictest3694698-21/usr/sbin/munin00:00:031
360184399448,18cyclictest13-21ksoftirqd/020:00:010
360184399447,21cyclictest13-21ksoftirqd/019:23:000
360184399442,21cyclictest13-21ksoftirqd/019:40:020
3601843994415,9cyclictest13-21ksoftirqd/022:00:140
3601843994413,8cyclictest13-21ksoftirqd/000:06:360
3601843994413,16cyclictest13-21ksoftirqd/021:44:050
360184399441,22cyclictest13-21ksoftirqd/022:32:080
360184399441,22cyclictest13-21ksoftirqd/022:32:080
3601843994411,8cyclictest13-21ksoftirqd/020:32:270
360184399441,11cyclictest2197014-21PK-Backend00:15:030
360185999438,20cyclictest42-21ksoftirqd/300:30:013
3601859994317,7cyclictest42-21ksoftirqd/300:06:093
3601859994312,8cyclictest42-21ksoftirqd/319:47:063
360185999431,21cyclictest42-21ksoftirqd/322:20:213
360185999431,17cyclictest2197014-21PK-Backend21:38:393
3601855994310,15cyclictest141rcu_preempt00:00:272
360185099434,11cyclictest141rcu_preempt20:27:221
360185099433,11cyclictest3625281-21ntp_states20:15:281
3601850994324,15cyclictest3670675-21sed22:35:311
3601850994322,12cyclictest26-21ksoftirqd/123:10:101
360184399439,17cyclictest13-21ksoftirqd/021:31:210
360184399432,18cyclictest13-21ksoftirqd/000:23:500
3601843994315,8cyclictest13-21ksoftirqd/000:30:220
3601843994310,17cyclictest3634746-21apt20:45:130
3601859994216,7cyclictest42-21ksoftirqd/323:41:433
3601859994212,7cyclictest42-21ksoftirqd/323:47:503
3601855994214,12cyclictest141rcu_preempt21:20:252
3601855994214,12cyclictest141rcu_preempt00:20:232
3601855994214,11cyclictest141rcu_preempt19:40:022
3601855994212,12cyclictest141rcu_preempt19:46:062
3601855994211,12cyclictest141rcu_preempt21:05:202
3601855994211,12cyclictest141rcu_preempt21:05:192
3601855994210,6cyclictest141rcu_preempt23:08:142
3601855994210,12cyclictest141rcu_preempt00:10:112
3601855994210,12cyclictest141rcu_preempt00:10:102
360185099421,7cyclictest3686243-21cut23:30:241
360185099421,38cyclictest1753-21gnome-shell00:30:451
360184399422,17cyclictest13-21ksoftirqd/020:36:230
360184399421,21cyclictest3656854-21sed21:50:240
360184399421,21cyclictest13-21ksoftirqd/023:05:070
360184399421,20cyclictest13-21ksoftirqd/000:29:150
360184399421,12cyclictest13-21ksoftirqd/020:40:300
3601855994114,12cyclictest141rcu_preempt22:35:222
3601855994113,10cyclictest141rcu_preempt23:30:062
3601855994113,10cyclictest141rcu_preempt19:55:012
3601855994112,12cyclictest141rcu_preempt19:20:182
3601855994112,11cyclictest141rcu_preempt20:45:022
3601855994111,12cyclictest141rcu_preempt23:20:212
3601855994111,12cyclictest141rcu_preempt21:38:332
3601855994111,12cyclictest141rcu_preempt20:50:182
3601855994111,12cyclictest141rcu_preempt19:34:142
3601855994111,12cyclictest141rcu_preempt19:10:312
3601855994110,14cyclictest141rcu_preempt21:16:362
3601855994110,14cyclictest141rcu_preempt00:06:412
3601850994127,10cyclictest3608909-21sshd19:26:301
3601850994122,14cyclictest3637967-21pool-org.gnome.20:55:001
360185099411,19cyclictest36334172smartctl20:40:201
3601843994116,8cyclictest13-21ksoftirqd/019:54:280
3601843994115,5cyclictest141rcu_preempt20:10:210
3601843994112,9cyclictest13-21ksoftirqd/023:41:430
360184399411,21cyclictest3605421-21chrt19:16:090
360184399411,10cyclictest2197014-21PK-Backend22:58:320
162399410,27rtkit-daemon2197014-21PK-Backend23:39:041
162399410,27rtkit-daemon2197014-21PK-Backend00:05:091
3601855994023,12cyclictest438-20systemd-journal21:40:302
360185599401,9cyclictest2197014-21PK-Backend00:38:412
3601855994014,6cyclictest141rcu_preempt19:55:262
3601855994013,7cyclictest141rcu_preempt20:35:022
3601855994013,6cyclictest141rcu_preempt22:10:092
3601855994013,6cyclictest141rcu_preempt22:10:082
3601855994013,12cyclictest141rcu_preempt22:46:302
3601855994012,12cyclictest141rcu_preempt22:40:282
3601855994012,11cyclictest141rcu_preempt23:16:452
3601855994012,11cyclictest141rcu_preempt20:02:422
360185599401,15cyclictest3605648-21taskset19:19:052
3601855994011,12cyclictest141rcu_preempt21:45:042
3601855994011,12cyclictest141rcu_preempt20:26:272
3601850994022,13cyclictest26-21ksoftirqd/120:30:201
3601850994021,14cyclictest3619151-21ntpq19:55:251
3601850994020,5cyclictest26-21ksoftirqd/119:35:141
3601850994019,17cyclictest26-21ksoftirqd/122:59:391
3601850994015,6cyclictest26-21ksoftirqd/100:10:211
3601850994015,6cyclictest26-21ksoftirqd/100:10:211
360185099401,35cyclictest3645756-21cron21:15:001
360185599399,12cyclictest141rcu_preempt22:30:182
360185599399,12cyclictest141rcu_preempt22:30:182
3601855993922,12cyclictest3693371-21grep23:55:142
3601855993913,6cyclictest141rcu_preempt22:20:152
3601855993912,6cyclictest141rcu_preempt20:55:212
3601855993912,6cyclictest141rcu_preempt20:55:212
3601855993911,12cyclictest141rcu_preempt23:36:562
3601855993911,11cyclictest141rcu_preempt22:54:102
3601855993911,11cyclictest141rcu_preempt22:54:092
3601855993910,13cyclictest141rcu_preempt22:26:312
3601855993910,13cyclictest141rcu_preempt00:32:552
3601855993910,12cyclictest141rcu_preempt23:00:132
3601855993910,11cyclictest141rcu_preempt23:11:102
360185099399,14cyclictest3665274-21munin-run22:20:001
360185099398,6cyclictest141rcu_preempt22:30:021
360185099398,4cyclictest141rcu_preempt20:36:301
3601850993927,8cyclictest1753-21gnome-shell21:35:201
3601850993926,10cyclictest3705207-21awk00:35:131
3601850993922,5cyclictest26-21ksoftirqd/120:10:021
3601850993920,14cyclictest3242253-21nautilus20:25:021
360185599389,12cyclictest141rcu_preempt00:15:272
360185599388,14cyclictest141rcu_preempt21:14:202
360185599388,12cyclictest141rcu_preempt20:35:082
360185599388,11cyclictest141rcu_preempt22:15:212
360185599383,14cyclictest1358-1kworker/u9:120:05:122
3601855993823,11cyclictest3676043-21memory22:55:242
3601855993810,12cyclictest141rcu_preempt20:45:052
3601855993810,11cyclictest141rcu_preempt19:25:142
3601855993810,11cyclictest141rcu_preempt00:25:042
360185099383,16cyclictest3610109-40lsb_release19:30:191
3601850993823,10cyclictest438-20systemd-journal20:13:311
3601850993821,13cyclictest3691878-21cat23:50:141
3601850993821,12cyclictest26-21ksoftirqd/120:45:281
162399380,27rtkit-daemon2197014-21PK-Backend19:41:352
162399380,27rtkit-daemon2197014-21PK-Backend19:41:342
3601855993713,6cyclictest141rcu_preempt22:05:152
360185099372,31cyclictest1753-21gnome-shell21:32:491
3601850993721,5cyclictest26-21ksoftirqd/123:00:161
3601850993721,11cyclictest26-21ksoftirqd/121:55:161
360185599364,10cyclictest0-21swapper/223:48:162
360185599362,18cyclictest1753-21gnome-shell20:11:272
3601850993623,9cyclictest3669381-21awk22:35:001
3601850993623,9cyclictest3669381-21awk22:35:001
3601850993620,12cyclictest3606633-21cat19:20:241
3601850993620,11cyclictest3620536-21forks20:00:201
3601850993619,13cyclictest3242255-21gdbus19:50:011
3601850993618,14cyclictest26-21ksoftirqd/119:42:101
3601850993618,14cyclictest26-21ksoftirqd/119:42:091
360185099361,10cyclictest2197014-21PK-Backend21:05:281
360185099361,10cyclictest2197014-21PK-Backend21:05:281
3601855993518,13cyclictest3656964-21wc21:50:262
3601850993520,11cyclictest3673423-21seq22:49:451
3601850993520,11cyclictest3602581-21expr19:10:221
3601850993513,6cyclictest141rcu_preempt22:21:311
36167582340,13sleep1251rcuc/119:50:191
3601850993423,6cyclictest26-21ksoftirqd/121:50:011
3601850993419,11cyclictest3701473-21expr00:20:151
3601850993418,12cyclictest855-21dbus-daemon23:45:001
3601850993417,13cyclictest3656961-21timerwakeupswit21:50:251
3601850993417,13cyclictest26-21ksoftirqd/100:30:011
3601850993414,16cyclictest1069-21runrttasks19:18:281
3601850993412,6cyclictest141rcu_preempt20:55:251
3601850993412,6cyclictest141rcu_preempt20:55:251
3601850993317,11cyclictest26-21ksoftirqd/121:15:151
3601850993315,13cyclictest2962348-21rs:main1
3601855993212,6cyclictest141rcu_preempt23:40:302
3601850993222,6cyclictest26-21ksoftirqd/121:30:011
3601850993222,6cyclictest26-21ksoftirqd/121:30:011
3601850993220,8cyclictest3671483-21cpuspeed_turbos22:40:141
3601850993016,10cyclictest3679066-21sshd23:07:301
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional