You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-01 - 04:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackfslot7s.osadl.org (updated Sun Sep 01, 2024 00:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
343759299290714,2888cyclictest3479472-21kworker/3:0+events21:35:013
343759299290714,2888cyclictest3479472-21kworker/3:0+events21:35:003
343759299282212,2805cyclictest3513086-21kworker/3:1+events23:10:163
343759299279313,2777cyclictest3435516-21kworker/3:0+events19:20:203
343759299274611,2731cyclictest3440751-21kworker/3:2+events19:45:223
343759299264412,2627cyclictest3533033-21kworker/3:1+events00:30:113
34375929925616,2551cyclictest3440751-21kworker/3:2+events19:42:373
343759299256013,2544cyclictest3533033-21kworker/3:1+events00:10:023
343759299248912,2474cyclictest3514922-21kworker/3:0+events23:17:033
343759299247912,2463cyclictest3440751-21kworker/3:2+events19:26:113
34375929924766,2466cyclictest3474702-21kworker/3:1+events21:04:113
343759299246512,2449cyclictest3474702-21kworker/3:1+events21:15:113
34375929924626,2452cyclictest3509897-21kworker/3:0+events22:53:413
343759299246212,2447cyclictest3514922-21kworker/3:0+events23:25:143
343759299243812,2422cyclictest3468129-21kworker/3:0+events20:48:263
343759299242412,2409cyclictest3524288-21kworker/3:0+events23:39:473
34375929924238,2412cyclictest3440751-21kworker/3:2+events19:53:033
343759299241312,2397cyclictest3464943-21kworker/3:2+events21:05:163
34375929924067,2395cyclictest3440751-21kworker/3:2+events19:38:483
343759299239712,2382cyclictest3492395-21kworker/3:2+events22:14:343
343759299239611,2381cyclictest3529348-21kworker/3:2+events23:59:273
343759299239412,2379cyclictest3464943-21kworker/3:2+events21:14:493
343759299239412,2378cyclictest3499953-21kworker/3:0+events22:23:463
343759299239311,2378cyclictest3492395-21kworker/3:2+events22:28:523
343759299238912,2374cyclictest3479472-21kworker/3:0+events21:21:263
343759299238911,2374cyclictest3458551-21kworker/3:1+events20:33:223
343759299238712,2371cyclictest3458551-21kworker/3:1+events20:43:183
343759299238612,2370cyclictest3479472-21kworker/3:0+events21:25:503
343759299238612,2370cyclictest3458551-21kworker/3:1+events20:17:593
343759299238412,2368cyclictest3486091-21kworker/3:1+events22:18:383
34375929923839,2371cyclictest3538076-21kworker/3:2+events00:29:053
343759299238311,2368cyclictest3486091-21kworker/3:1+events21:41:433
343759299237212,2356cyclictest3440751-21kworker/3:2+events19:57:323
343759299236812,2352cyclictest3461737-21kworker/3:0+events20:26:383
343759299236112,2346cyclictest3506456-21kworker/3:2+events23:05:133
343759299236112,2345cyclictest3484247-21kworker/3:2+events21:47:023
343759299236112,2345cyclictest3484247-21kworker/3:2+events21:47:023
343759299236012,2344cyclictest3486091-21kworker/3:1+events21:56:553
343759299234411,2329cyclictest3492395-21kworker/3:2+events22:35:463
343759299233111,2317cyclictest3464943-21kworker/3:2+events20:59:573
343759299233012,2314cyclictest3440751-21kworker/3:2+events20:12:103
343759299233011,2315cyclictest3484247-21kworker/3:2+events21:39:513
34375929923279,2315cyclictest3458551-21kworker/3:1+events20:36:523
343759299232612,2310cyclictest3486091-21kworker/3:1+events22:08:183
343759299230010,2287cyclictest3486091-21kworker/3:1+events22:00:233
343759299229210,2278cyclictest3453508-21kworker/3:0+events20:04:353
343759299228911,2273cyclictest3541545-21kworker/3:0+events00:39:273
343759299228812,2273cyclictest3509897-21kworker/3:0+events23:01:113
343759299225711,2242cyclictest3536472-21kworker/3:0+events00:18:183
343759299224711,2234cyclictest3533033-21kworker/3:1+events00:20:063
34375929922208,2208cyclictest3501650-21kworker/3:1+events22:48:013
343759299222011,2206cyclictest3526171-21kworker/3:2+events23:42:443
34375929922159,2203cyclictest3501650-21kworker/3:1+events22:43:333
343759299221511,2201cyclictest3440751-21kworker/3:2+events19:30:243
343759299221211,2198cyclictest3464943-21kworker/3:2+events20:52:273
343759299221011,2196cyclictest3440751-21kworker/3:2+events19:17:433
343759299220811,2194cyclictest3453508-21kworker/3:0+events20:06:043
343759299220810,2195cyclictest3524288-21kworker/3:0+events23:45:483
343759299220711,2193cyclictest3521106-21kworker/3:1+events23:34:293
34375929922048,2193cyclictest3529348-21kworker/3:2+events00:08:113
343759299220411,2191cyclictest3506456-21kworker/3:2+events22:55:223
34375929921949,2182cyclictest3529348-21kworker/3:2+events00:02:443
34375929921809,2167cyclictest3501650-21kworker/3:1+events22:33:243
343759299218011,2166cyclictest3514922-21kworker/3:0+events23:22:563
343759299218010,2166cyclictest3458551-21kworker/3:1+events20:24:463
343759299217911,2165cyclictest3527759-21kworker/3:1+events23:50:513
343759299217610,2163cyclictest3438761-21kworker/3:1+events19:14:473
34375929921709,2157cyclictest3486091-21kworker/3:1+events21:50:183
34372872153115,22sleep30-21swapper/319:08:273
34373262150108,25sleep00-21swapper/019:08:570
3436633214295,25sleep10-21swapper/119:05:171
34371432138105,28sleep20-21swapper/219:06:462
3437583997775,2cyclictest3508338-21kworker/u8:1+events_unbound23:40:010
3437583997471,2cyclictest3481311-21kworker/u8:1+flush-259:021:50:000
3437583997471,2cyclictest3481311-21kworker/u8:1+flush-259:021:50:000
3437583997169,2cyclictest3471427-21kworker/u8:2+flush-259:021:20:200
3437583996865,2cyclictest3508819-21kworker/u8:4+flush-259:023:15:000
3437583995150,1cyclictest3508819-21kworker/u8:4+flush-259:022:51:190
3437583994946,2cyclictest3532412-21kworker/u8:2+flush-259:000:40:010
3437583994845,2cyclictest3508338-21kworker/u8:1+flush-259:023:29:590
3437587994644,2cyclictest3438701-21kworker/u8:1+flush-259:019:49:551
3437583994539,2cyclictest3511778-21kworker/u8:0+events_unbound00:10:130
3437589994236,4cyclictest3533439-21latency_hist00:10:012
3437583994239,2cyclictest3498592-21kworker/u8:2+events_unbound22:20:130
3437583994139,2cyclictest3508338-21kworker/u8:1+flush-259:023:35:110
3437583994139,2cyclictest3438701-21kworker/u8:1+events_unbound19:50:190
3437583994038,2cyclictest3444025-21kworker/u8:0+events_unbound19:30:000
3437583993937,2cyclictest3468639-21kworker/u8:4+events_unbound20:50:180
3437583993937,2cyclictest3468639-21kworker/u8:4+events_unbound20:45:200
3437583993936,2cyclictest3511778-21kworker/u8:0+events_unbound23:45:130
3437589993834,3cyclictest3508819-21kworker/u8:4+flush-259:023:25:002
3437589993832,4cyclictest3538448-21latency_hist00:25:012
3437587993836,2cyclictest3508819-21kworker/u8:4+flush-259:023:10:111
3437587993833,4cyclictest3498592-21kworker/u8:2+flush-259:022:35:011
3437587993730,5cyclictest3484429-21apt-get21:34:591
3437587993730,5cyclictest3484429-21apt-get21:34:591
3437583993733,3cyclictest3508338-21kworker/u8:1+events_unbound23:05:000
3437589993633,2cyclictest3534255-21kworker/u8:1+flush-259:000:30:002
3437587993633,2cyclictest3525128-21kworker/u8:3+flush-259:023:50:001
3437583993635,1cyclictest211rcuc/000:09:560
3437583993634,2cyclictest3511778-21kworker/u8:0+flush-259:000:00:260
3437583993634,2cyclictest3459863-21kworker/u8:1+flush-259:020:30:210
3437587993529,4cyclictest3530073-21diskstats23:55:131
3437583993533,2cyclictest3444025-21kworker/u8:0+flush-259:021:10:210
3437583993533,2cyclictest3444025-21kworker/u8:0+events_unbound20:05:180
3437587993431,3cyclictest3508819-21kworker/u8:4+flush-259:023:21:581
3437587993429,3cyclictest3492566-21latency_hist22:00:011
3437587993428,4cyclictest3523830-21ntpq23:35:161
3437583993432,2cyclictest3438701-21kworker/u8:1+events_unbound20:10:150
3437583993331,2cyclictest3511778-21kworker/u8:0+events_unbound23:50:190
3437583993331,2cyclictest3481311-21kworker/u8:1+events_unbound21:50:120
3437587993229,2cyclictest3533438-21apt-get00:10:011
3437587993227,4cyclictest298-21systemd-journal20:30:281
3437583993229,2cyclictest3496930-21kworker/u8:0+events_unbound22:45:190
3437589993027,2cyclictest3484930-21kworker/u8:0+flush-259:021:40:002
3437587993028,2cyclictest3536593-21kworker/u8:3+flush-259:000:20:001
3437583993027,2cyclictest3471427-21kworker/u8:2+events_unbound21:05:350
3437589992926,2cyclictest3432718-21kworker/u8:0+flush-259:019:20:012
3437589992925,3cyclictest3456838-21latency_hist20:10:012
3437583992928,1cyclictest3471427-21kworker/u8:2+events_unbound22:00:180
3437583992925,2cyclictest678-21gmain19:35:280
3437589992824,3cyclictest3448987-21kworker/u8:2+flush-259:020:10:152
3437589992823,3cyclictest3481287-21latency_hist21:25:012
3437583992826,2cyclictest3435902-21kworker/u8:2+events_unbound19:14:540
3437583992724,2cyclictest3482903-21kworker/u8:3+flush-259:021:30:210
3437583992724,2cyclictest3482903-21kworker/u8:3+flush-259:021:30:200
3437583992724,2cyclictest3457838-21kworker/u8:3+events_unbound20:15:200
3437583992724,2cyclictest3445721-21kworker/u8:3+events_unbound19:45:130
3437589992622,3cyclictest3459863-21kworker/u8:1+flush-259:020:30:002
3437587992620,4cyclictest3508321-21apt-get22:50:011
3437583992624,2cyclictest3532412-21kworker/u8:2+flush-259:000:20:260
3437583992624,2cyclictest3487882-21kworker/u8:3+events_unbound21:55:200
3437583992624,2cyclictest3471427-21kworker/u8:2+events_unbound22:10:120
3437589992520,4cyclictest3498592-21kworker/u8:2+flush-259:022:25:002
3437587992523,2cyclictest3498592-21kworker/u8:2+flush-259:022:25:221
3437587992522,2cyclictest3532412-21kworker/u8:2+flush-259:000:28:541
3437583992523,2cyclictest3534255-21kworker/u8:1+events_unbound00:25:170
3437583992523,2cyclictest3496930-21kworker/u8:0+flush-259:022:40:160
3437583992522,2cyclictest3463453-21kworker/u8:2+flush-259:020:35:200
3437583992521,3cyclictest3508338-21kworker/u8:1+flush-259:023:15:100
3437589992419,3cyclictest3478359-21/usr/sbin/munin21:15:132
3437589992418,4cyclictest3476545-21latency_hist21:10:002
3437587992420,3cyclictest3534255-21kworker/u8:1+flush-259:000:35:171
3437583992422,2cyclictest3534255-21kworker/u8:1+events_unbound00:32:540
3437583992422,2cyclictest3496930-21kworker/u8:0+events_unbound22:25:100
3437583992422,2cyclictest3459863-21kworker/u8:1+events_unbound20:22:440
3437583992422,2cyclictest3445721-21kworker/u8:3+flush-259:020:00:150
3437583992422,2cyclictest3435902-21kworker/u8:2+events_unbound19:15:190
3437589992317,4cyclictest3520637-21ntpq23:25:162
3437587992321,2cyclictest3525128-21kworker/u8:3+events_unbound23:40:191
3437587992321,2cyclictest3508338-21kworker/u8:1+events_unbound23:05:181
3437587992321,2cyclictest3471427-21kworker/u8:2+events_unbound21:05:131
3437587992320,2cyclictest3444025-21kworker/u8:0+flush-259:021:00:161
3437583992321,2cyclictest3444025-21kworker/u8:0+events_unbound20:25:240
3437583992321,1cyclictest3438701-21kworker/u8:1+events_unbound19:40:140
3437587992218,3cyclictest3496930-21kworker/u8:0+flush-259:022:35:151
3437587992216,4cyclictest298-21systemd-journal21:25:271
3437583992221,1cyclictest3508819-21kworker/u8:4+events_unbound23:57:280
3437583992221,1cyclictest3471427-21kworker/u8:2+flush-259:021:40:100
3437583992220,2cyclictest3508819-21kworker/u8:4+events_unbound23:20:210
3437583992220,2cyclictest3508819-21kworker/u8:4+events_unbound22:55:130
3437583992220,2cyclictest3496930-21kworker/u8:0+flush-259:022:35:210
3437583992219,2cyclictest3481311-21kworker/u8:1+flush-259:022:15:110
3437583992219,2cyclictest3459863-21kworker/u8:1+flush-259:020:40:150
3437589992119,2cyclictest3482903-21kworker/u8:3+flush-259:021:33:322
3437589992119,2cyclictest3482903-21kworker/u8:3+flush-259:021:33:322
3437589992116,3cyclictest0-21swapper/220:00:002
3437587992120,1cyclictest21050irq/137-i91521:16:381
3437587992119,2cyclictest3511778-21kworker/u8:0+flush-259:000:15:011
3437587992119,2cyclictest3471427-21kworker/u8:2+events_unbound22:10:181
3437587992113,5cyclictest3541270-21python300:30:191
3437583992120,1cyclictest3511778-21kworker/u8:0+events_unbound23:30:130
3437583992120,1cyclictest3440681-21kworker/u8:4+events_unbound19:20:290
3437583992119,2cyclictest3444025-21kworker/u8:0+events_unbound21:01:450
3437589992014,4cyclictest298-21systemd-journal20:16:092
3437587992018,2cyclictest3498592-21kworker/u8:2+events_unbound22:20:191
3437587992017,2cyclictest3471427-21kworker/u8:2+flush-259:021:20:161
3437587992016,3cyclictest3456047-21ntp_kernel_pll_20:05:161
3437583992018,2cyclictest3536593-21kworker/u8:3+events_unbound00:19:410
3437583992018,2cyclictest3508338-21kworker/u8:1+events_unbound23:07:120
3437583992018,2cyclictest3442376-21kworker/u8:2+events_unbound19:34:130
3437589991916,2cyclictest846-21gmain23:39:592
3437589991915,3cyclictest3467047-21diskstats20:40:132
3437589991912,5cyclictest3498592-21kworker/u8:2+flush-259:022:40:012
3437583991917,2cyclictest3481311-21kworker/u8:1+events_unbound22:05:130
3437583991917,2cyclictest3444025-21kworker/u8:0+events_unbound19:55:030
3437583991915,3cyclictest3485498-21ntpq21:35:170
3437589991816,2cyclictest3444025-21kworker/u8:0+flush-259:020:01:322
3437589991814,3cyclictest3511778-21kworker/u8:0+flush-259:000:00:002
3437587991816,2cyclictest3508819-21kworker/u8:4+flush-259:022:55:221
3437587991815,2cyclictest3440681-21kworker/u8:4+events_unbound19:25:131
3437587991813,3cyclictest3454660-21python320:00:211
3437587991813,3cyclictest3452571-21diskstats19:55:111
3437583991817,1cyclictest3471427-21kworker/u8:2+flush-259:021:25:100
3437583991816,2cyclictest3496930-21kworker/u8:0+flush-259:022:30:270
3437589991717,0cyclictest1203-21snmpd23:06:582
3437589991714,2cyclictest3500726-21apt-get22:25:132
3437589991712,4cyclictest3477859-21taskset21:12:012
343758999171,14cyclictest678-21gmain20:35:272
3437587991716,1cyclictest3511778-21kworker/u8:0+events_unbound23:30:141
3437587991716,1cyclictest3468639-21kworker/u8:4+flush-259:020:45:221
3437587991715,2cyclictest3508338-21kworker/u8:1+flush-259:023:00:091
343758999163,1cyclictest0-21swapper/200:10:012
3437589991614,2cyclictest3508338-21kworker/u8:1+flush-259:023:15:152
3437589991612,3cyclictest3487016-21ntpq21:40:162
3437587991615,1cyclictest21050irq/137-i91500:23:331
3437587991613,2cyclictest3481311-21kworker/u8:1+events_unbound21:40:121
3437587991612,3cyclictest720-10wireplumber21:12:551
3437587991610,5cyclictest3472059-21/usr/sbin/munin20:55:131
3437589991512,2cyclictest3512967-21runrttasks23:00:482
3437589991511,3cyclictest3488121-21apt-get21:45:122
3437589991511,3cyclictest3488121-21apt-get21:45:122
3437589991511,3cyclictest3449204-21/usr/sbin/munin19:45:202
3437589991511,2cyclictest3443559-21pmu-power19:25:192
3437589991510,3cyclictest298-21systemd-journal20:50:002
343758999150,2cyclictest3474848-21sshd21:03:282
343758999150,2cyclictest3464689-21sshd20:31:392
343758999150,14cyclictest3509109-21/usr/sbin/munin22:50:132
343758999150,14cyclictest3471670-21taskset20:53:182
3437587991511,3cyclictest562-21dbus-daemon20:55:021
3437587991511,3cyclictest3466311-21chrt20:37:181
343758799150,2cyclictest1203-21snmpd22:00:421
343758799150,14cyclictest3498781-21seq22:19:121
343758799150,14cyclictest3488409-21/usr/sbin/munin21:45:141
343758799150,14cyclictest3488409-21/usr/sbin/munin21:45:141
343758799150,14cyclictest3463240-21seq20:28:041
3437583991513,2cyclictest3444025-21kworker/u8:0+events_unbound21:16:410
343758999149,3cyclictest3498338-21find22:15:202
3437589991411,2cyclictest3461749-21seq20:24:032
3437589991411,2cyclictest3444491-21/usr/sbin/munin19:30:112
3437589991410,3cyclictest3493975-21seq22:03:032
3437589991410,2cyclictest3492291-21runrttasks21:56:492
343758999140,3cyclictest3473140-21runrttasks20:57:132
343758999140,13cyclictest562-21dbus-daemon22:15:012
343758999140,13cyclictest3532594-21seq00:02:452
343758999140,13cyclictest3526228-21taskset23:42:242
343758999140,13cyclictest3506151-21taskset22:40:182
343758999140,13cyclictest3503532-21taskset22:34:262
343758999140,13cyclictest3447931-21/usr/sbin/munin19:40:132
343758999140,12cyclictest1203-21snmpd22:55:072
343758799149,5cyclictest3445721-21kworker/u8:3+flush-259:019:35:161
3437587991414,0cyclictest21050irq/137-i91523:26:071
3437587991414,0cyclictest21050irq/137-i91521:36:311
3437587991414,0cyclictest21050irq/137-i91519:22:181
3437587991413,1cyclictest21050irq/137-i91519:14:261
3437587991412,1cyclictest1203-21snmpd19:30:191
3437587991410,3cyclictest597-21gdbus00:00:011
3437587991410,3cyclictest3452076-21taskset19:54:511
343758799140,2cyclictest3467810-21seq20:41:051
343758799140,13cyclictest3458391-21taskset20:11:461
343758799140,13cyclictest3448569-21runrttasks19:40:211
343758999139,3cyclictest3450797-21apt-get19:50:122
343758999139,3cyclictest3438288-21ntpq19:10:162
3437589991311,1cyclictest746-21mta-sts-daemon22:45:202
3437589991311,1cyclictest3528588-21/usr/sbin/munin23:50:152
3437589991311,1cyclictest3494426-21apt-get22:05:122
3437589991310,2cyclictest3540994-21/usr/sbin/munin00:30:162
3437589991310,2cyclictest0-21swapper/219:35:132
343758999130,1cyclictest3543150-21seq00:36:312
343758999130,12cyclictest3490643-21chrt21:50:502
343758799138,2cyclictest3461847-21perf20:25:001
3437587991312,1cyclictest21050irq/137-i91520:18:441
3437587991311,2cyclictest3508338-21kworker/u8:1+events_unbound23:15:181
343758799130,1cyclictest3528299-21dpkg23:50:111
343758799130,12cyclictest3509774-21sshd22:50:271
343758799130,12cyclictest3489668-21/usr/sbin/munin21:50:201
343758799130,12cyclictest13155-21runrttasks19:17:011
343758999128,3cyclictest3516677-21chrt23:14:312
343758999126,4cyclictest3536065-21ntpq00:15:162
3437589991210,1cyclictest3527379-21/usr/sbin/munin23:45:192
3437589991210,1cyclictest1203-21snmpd23:32:172
343758799129,2cyclictest3506464-21seq22:41:241
343758799129,2cyclictest3495546-21runrttasks22:08:041
343758399128,3cyclictest3459863-21kworker/u8:1+events_unbound20:57:590
343758999119,1cyclictest3441324-21/usr/sbin/munin19:20:192
343758999118,2cyclictest3481531-21/usr/sbin/munin21:25:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional