You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-25 - 14:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Mon Nov 25, 2024 12:43:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
58130299543,43cyclictest12450irq/32-eth012:25:510
58130399519,17cyclictest702914-21ssh11:25:551
58130399519,17cyclictest702914-21ssh11:25:551
581303995124,4cyclictest956-21mta-sts-daemon09:30:581
581303995040,6cyclictest646643-21basename10:00:011
581303995034,10cyclictest422-21dbus-daemon11:05:511
581303995027,2cyclictest0-21swapper/109:45:011
581303994939,6cyclictest740605-21systemd-run12:15:021
581303994824,5cyclictest602284-21munin-run08:20:011
581303994824,5cyclictest602284-21munin-run08:20:011
581303994823,6cyclictest754258-21unixbench_singl12:31:071
581302994738,6cyclictest422-21dbus-daemon07:40:010
581303994636,6cyclictest474-21in:imuxsock08:30:021
581303994632,10cyclictest640832-21idleruntime-cro09:50:011
581303994620,3cyclictest0-21swapper/112:15:461
581303994539,3cyclictest730117-21ssh12:00:531
581303994530,11cyclictest657693-21mailstats10:16:041
581303994519,7cyclictest598970-21switchtime08:06:011
58130299452,38cyclictest659243-21kworker/u8:0+rpciod11:35:510
581302994515,25cyclictest565-21gmain08:10:520
581302994515,25cyclictest565-21gmain08:10:520
581304994436,3cyclictest0-21swapper/212:10:562
581303994416,10cyclictest422-21dbus-daemon10:33:551
58130399441,22cyclictest669930-21ntpq10:35:571
581304994337,3cyclictest0-21swapper/207:20:022
581304994334,6cyclictest744497-21munin-plugin-st12:20:012
581302994335,5cyclictest677821-21systemd-run10:50:010
581304994233,6cyclictest642487-21unixbench-2d09:51:042
581304994232,7cyclictest3959321-21cups-browsed11:31:082
581302994224,13cyclictest151ktimers/009:31:000
581304994133,5cyclictest638991-21latency_hist09:45:022
581304994133,5cyclictest623793-21latency_hist09:20:032
581304994133,5cyclictest617286-21diskmemload11:20:482
581304994133,5cyclictest453-21systemd-logind11:05:012
581304994132,6cyclictest679081-21ssh10:51:042
581304994132,6cyclictest679081-21ssh10:51:042
581304994131,6cyclictest0-21swapper/211:40:012
581303994131,7cyclictest588804-21systemd-run07:35:021
581302994134,4cyclictest683957-21systemctl11:00:010
581302994134,4cyclictest683957-21systemctl11:00:010
581302994129,8cyclictest12450irq/32-eth011:05:570
581304994033,3cyclictest0-21swapper/212:30:012
581304994032,5cyclictest588781-21cron07:35:012
581303994032,5cyclictest596277-21munin-run08:00:011
581303994031,5cyclictest697932-21sh11:20:011
581303994031,3cyclictest0-21swapper/109:15:581
581303994029,7cyclictest421-21avahi-daemon09:25:431
581303994017,10cyclictest760194-21systemctl12:40:011
58130299404,29cyclictest757473-21ssh12:35:510
581302994033,4cyclictest694895-21aten_r2power_fr11:15:400
581302994031,7cyclictest702989-21processes11:25:570
581302994031,7cyclictest702989-21processes11:25:570
581304993929,6cyclictest674993-21latency_hist10:45:022
581304993929,6cyclictest674993-21latency_hist10:45:022
58130299396,28cyclictest753749-21ssh12:31:010
581304993830,5cyclictest736673-21systemd-run12:10:022
581304993830,5cyclictest603790-21cron08:25:012
581304993830,5cyclictest0-21swapper/210:40:012
581304993821,13cyclictest351ktimers/208:45:522
581304993817,15cyclictest351ktimers/208:51:202
581303993829,6cyclictest694297-21cron11:15:011
581303993827,7cyclictest659963-21cpuspeed10:20:431
581302993831,5cyclictest730057-21ntpq12:00:510
581302993827,6cyclictest0-21swapper/012:21:060
581302993826,9cyclictest422-21dbus-daemon10:55:010
581302993826,9cyclictest422-21dbus-daemon10:55:010
581304993726,8cyclictest633779-21missed_timers09:35:462
581304993726,8cyclictest633779-21missed_timers09:35:462
58130299375,25cyclictest660484-21ssh10:20:550
58130299372,26cyclictest621708-21sshd09:15:580
581302993718,13cyclictest12450irq/32-eth011:30:550
581304993631,2cyclictest0-21swapper/208:40:012
581304993628,5cyclictest681549-21fschecks_count10:55:452
581304993628,5cyclictest681549-21fschecks_count10:55:452
581304993628,3cyclictest0-21swapper/212:00:512
581304993627,6cyclictest421-21avahi-daemon11:16:242
581304993627,5cyclictest0-21swapper/210:45:532
581303993628,5cyclictest653096-21grep10:10:011
581303993627,5cyclictest615669-21unixbench_multi09:01:031
581302993628,5cyclictest649939-21latency_hist10:05:020
581302993628,5cyclictest642469-21unixbench-2d09:51:030
581302993628,5cyclictest1-21systemd08:00:020
581302993627,7cyclictest715028-21/usr/sbin/munin11:41:030
58130299362,29cyclictest734149-21ssh12:05:580
581304993529,3cyclictest0-21swapper/208:35:012
581304993528,3cyclictest0-21swapper/211:55:542
581304993526,3cyclictest0-21swapper/210:15:582
581304993525,6cyclictest594931-21latency_hist07:55:022
58130499352,28cyclictest628064-21ssh09:25:572
581304993521,7cyclictest647778-21sh10:00:492
58130399354,23cyclictest647831-21ssh10:00:511
581303993527,5cyclictest713126-21cron11:40:011
581303993527,4cyclictest0-21swapper/109:10:011
581303993511,7cyclictest59950lldpd10:52:291
581303993511,7cyclictest59950lldpd10:52:291
581302993529,4cyclictest603799-21munin-plugin-st08:25:010
581302993525,7cyclictest619156-21unixbench_singl09:11:050
581302993525,6cyclictest713087-21kworker/u8:2-events_unbound11:45:500
581304993427,3cyclictest0-21swapper/211:05:562
581304993426,5cyclictest713712-21aten_r2power_cu11:40:382
581304993426,5cyclictest660610-21proc_pri10:20:592
581304993425,7cyclictest617310-21sh09:10:012
581303993427,4cyclictest583303-21fschecks_count07:15:431
581303993426,4cyclictest0-21swapper/108:40:541
581303993425,5cyclictest611305-21systemd-run08:50:021
581303993424,6cyclictest955-21mta-sts-daemon07:46:001
581303993422,9cyclictest591798-21/usr/sbin/munin07:45:021
581302993428,4cyclictest617286-21diskmemload11:05:020
581302993426,4cyclictest0-21swapper/009:55:530
581304993328,2cyclictest0-21swapper/209:20:442
581304993327,3cyclictest0-21swapper/208:00:012
581304993326,3cyclictest0-21swapper/210:30:442
581304993324,6cyclictest422-21dbus-daemon12:40:012
58130499332,25cyclictest745809-21ssh12:20:582
58130499331,5cyclictest171rcu_preempt10:05:522
581303993327,3cyclictest0-21swapper/109:40:011
581303993327,3cyclictest0-21swapper/109:40:011
581303993326,3cyclictest0-21swapper/111:50:021
581303993326,3cyclictest0-21swapper/110:46:571
581303993325,5cyclictest734439-21timerandwakeup12:06:041
581303993325,5cyclictest665780-21systemd-run10:30:021
581303993325,5cyclictest581403-21latency_hist07:10:031
581303993324,6cyclictest687178-21unin-run11:05:011
58130399332,22cyclictest706673-21ssh11:30:541
581303993321,7cyclictest1-21systemd11:25:021
58130299335,18cyclictest12450irq/32-eth011:10:530
581302993325,6cyclictest613583-21needreboot08:55:480
581302993323,7cyclictest609785-21cron08:45:010
581302993323,5cyclictest436-21gpm09:39:420
581302993323,5cyclictest436-21gpm09:39:420
581304993224,5cyclictest753170-21/usr/sbin/munin12:30:462
581304993224,5cyclictest604318-21fschecks_time08:25:472
581304993221,8cyclictest601648-21ntp_kernel_pll_08:15:502
581304993221,8cyclictest601648-21ntp_kernel_pll_08:15:502
581304993221,7cyclictest665745-21cron10:30:012
581303993226,3cyclictest0-21swapper/112:00:021
581303993225,3cyclictest0-21swapper/112:25:011
581303993224,5cyclictest585554-21timerandwakeup07:21:011
581303993224,4cyclictest0-21swapper/111:50:531
58130399321,25cyclictest657-21snmpd08:55:521
58130399321,24cyclictest749750-21ssh12:25:541
58130299323,23cyclictest663844-21ssh10:25:580
581302993226,4cyclictest597793-21idleruntime-cro08:05:010
581302993224,5cyclictest957-21mta-sts-daemon09:42:520
58130299322,24cyclictest669941-21ssh10:35:570
581302993213,17cyclictest0-21swapper/012:10:580
581304993126,2cyclictest0-21swapper/208:15:012
581304993126,2cyclictest0-21swapper/208:15:012
581304993125,3cyclictest0-21swapper/211:10:532
581304993124,3cyclictest0-21swapper/211:50:522
581304993122,5cyclictest957-21mta-sts-daemon07:35:122
581304993120,8cyclictest609192-21ntp_kernel_pll_08:40:522
581303993122,6cyclictest597360-21sshd08:00:571
581302993126,3cyclictest606250-21kworker/u8:2-rpciod08:40:010
581302993124,5cyclictest668914-21latency_hist10:35:030
581302993124,4cyclictest588788-21sh07:35:010
581302993122,7cyclictest12450irq/32-eth008:25:580
581302993121,7cyclictest624698-21ntp_kernel_err09:20:490
581302993119,9cyclictest585721-21unixbench_singl07:21:020
581302993119,9cyclictest0-21swapper/012:00:010
581302993119,5cyclictest741458-21fschecks_time12:15:450
58130499303,21cyclictest654246-21ssh10:10:552
581304993024,3cyclictest0-21swapper/210:00:012
581304993024,3cyclictest0-21swapper/207:25:552
581304993023,4cyclictest703044-21mailstats11:25:582
581304993023,4cyclictest703044-21mailstats11:25:582
581304993022,5cyclictest613306-21fschecks_time08:55:432
581304993022,5cyclictest590634-21aten_r2power_fr07:40:472
581304993021,6cyclictest718269-21ntp_kernel_err11:45:492
581304993020,8cyclictest597160-21ntp_kernel_pll_08:00:512
581303993024,4cyclictest581301-21cyclictest09:20:461
581303993024,3cyclictest0-21swapper/111:00:011
581303993024,3cyclictest0-21swapper/111:00:011
581302993023,5cyclictest421-21avahi-daemon10:05:520
581302993020,7cyclictest591110-21needreboot07:40:550
581302993010,13cyclictest649128-21kworker/u8:5+rpciod10:16:010
581304992922,4cyclictest620435-21cron09:15:012
581304992920,6cyclictest581309-21systemd-run07:10:022
58130399294,20cyclictest654444-21ps10:11:021
581303992924,4cyclictest581301-21cyclictest07:25:591
581303992924,2cyclictest0-21swapper/111:41:031
581303992924,2cyclictest0-21swapper/109:11:001
581302992922,4cyclictest3959321-21cups-browsed08:45:240
58130299291,21cyclictest582180-21sshd07:10:500
581304992822,4cyclictest581301-21cyclictest09:01:012
581303992822,3cyclictest0-21swapper/108:40:011
581303992820,5cyclictest672483-21fschecks_time10:40:451
581303992820,5cyclictest672483-21fschecks_time10:40:451
581303992820,5cyclictest589598-21needreboot07:35:531
58130299284,3cyclictest0-21swapper/008:51:200
581302992823,2cyclictest0-21swapper/009:10:010
581302992822,3cyclictest583303-21fschecks_count07:15:430
581302992822,3cyclictest0-21swapper/009:25:430
581302992821,4cyclictest654445-21grep10:11:020
581302992821,4cyclictest12550irq/33-eth007:55:010
581302992818,7cyclictest0-21swapper/008:20:010
581302992818,7cyclictest0-21swapper/008:20:010
581302992816,9cyclictest0-21swapper/008:10:010
58130299281,23cyclictest3959321-21cups-browsed07:25:500
581304992722,2cyclictest0-21swapper/209:30:492
581304992720,4cyclictest957-21mta-sts-daemon09:48:492
581304992720,4cyclictest585683-21unixbench_multi07:21:012
581303992720,4cyclictest603689-21unixbench_multi08:21:191
581303992719,5cyclictest642344-21timerandwakeup09:51:031
58130299272,20cyclictest672873-21ssh10:40:520
58130299272,20cyclictest672873-21ssh10:40:520
581302992721,3cyclictest698785-21fschecks_count11:20:450
581302992717,4cyclictest606672-21unixbench_multi08:31:060
58130299271,20cyclictest722358-21sh11:50:580
581304992621,2cyclictest0-21swapper/207:15:032
581303992618,5cyclictest0-21swapper/108:55:011
581303992616,7cyclictest606632-21unixbench-2d08:31:061
581302992619,5cyclictest592530-21missed_timers07:45:570
581303992517,5cyclictest600656-21unixbench_multi08:11:031
581303992517,5cyclictest600656-21unixbench_multi08:11:031
581304992417,5cyclictest599439-21latency_hist08:10:022
581302992418,3cyclictest959-21mta-sts-daemon09:00:560
581302992416,6cyclictest956-21mta-sts-daemon09:46:450
581304992314,6cyclictest266-21systemd-journal07:50:012
581303992315,5cyclictest594788-21idleruntime-cro07:55:011
581303992312,8cyclictest266-21systemd-journal07:10:571
58130599202,9cyclictest0-21swapper/311:30:583
58130599193,8cyclictest0-21swapper/310:31:033
58130599193,7cyclictest0-21swapper/310:05:563
58130599185,8cyclictest0-21swapper/309:10:573
58130599184,7cyclictest0-21swapper/307:25:523
58130599182,9cyclictest0-21swapper/312:30:563
58130599182,8cyclictest0-21swapper/312:25:503
58130599182,8cyclictest0-21swapper/311:20:503
58130599182,8cyclictest0-21swapper/308:25:583
58130599181,11cyclictest0-21swapper/312:35:543
581302991811,5cyclictest581316-21munin-run07:10:020
58130599174,8cyclictest0-21swapper/310:50:583
58130599174,8cyclictest0-21swapper/310:50:583
58130599173,8cyclictest0-21swapper/312:21:013
58130599173,8cyclictest0-21swapper/312:15:523
58130599173,8cyclictest0-21swapper/311:15:543
58130599173,7cyclictest0-21swapper/311:10:533
58130599172,8cyclictest0-21swapper/310:15:543
58130599172,8cyclictest0-21swapper/309:20:543
58130599172,6cyclictest0-21swapper/309:15:523
58130599171,9cyclictest0-21swapper/312:05:543
58130599171,8cyclictest0-21swapper/311:40:493
58130599164,8cyclictest0-21swapper/311:05:513
58130599163,9cyclictest0-21swapper/311:51:573
58130599163,8cyclictest0-21swapper/312:00:583
58130599163,8cyclictest0-21swapper/310:45:503
58130599163,8cyclictest0-21swapper/310:00:503
58130599163,7cyclictest0-21swapper/311:25:503
58130599163,7cyclictest0-21swapper/311:25:503
58130599163,7cyclictest0-21swapper/308:45:553
58130599163,6cyclictest0-21swapper/307:15:493
58130599162,9cyclictest0-21swapper/310:20:513
58130599162,8cyclictest0-21swapper/312:10:533
58130599162,7cyclictest0-21swapper/308:35:523
58130599162,6cyclictest0-21swapper/307:50:513
58130599161,8cyclictest0-21swapper/309:35:543
58130599161,8cyclictest0-21swapper/309:35:543
58130599161,8cyclictest0-21swapper/307:46:013
58130599155,6cyclictest0-21swapper/310:25:533
58130599153,6cyclictest0-21swapper/311:45:493
58130599153,6cyclictest0-21swapper/311:35:403
58130599153,6cyclictest0-21swapper/308:30:553
58130599152,9cyclictest0-21swapper/309:55:543
58130599152,8cyclictest0-21swapper/311:55:563
58130599152,7cyclictest0-21swapper/310:55:523
58130599152,7cyclictest0-21swapper/310:55:523
58130599152,7cyclictest0-21swapper/309:25:423
58130599152,7cyclictest0-21swapper/307:55:563
58130599151,8cyclictest0-21swapper/310:11:003
58130599151,6cyclictest0-21swapper/308:10:503
58130599151,6cyclictest0-21swapper/308:10:503
58130599144,7cyclictest0-21swapper/307:10:513
58130599144,6cyclictest0-21swapper/310:35:553
58130599144,5cyclictest0-21swapper/308:05:503
58130599143,5cyclictest0-21swapper/307:35:553
58130599142,6cyclictest0-21swapper/309:05:503
58130599141,7cyclictest0-21swapper/307:30:533
58130599141,6cyclictest0-21swapper/308:40:503
58130599132,6cyclictest0-21swapper/309:30:433
58130599132,5cyclictest0-21swapper/308:51:163
58130599131,8cyclictest0-21swapper/310:40:423
58130599131,8cyclictest0-21swapper/310:40:423
58130599131,7cyclictest0-21swapper/311:05:023
58130599131,7cyclictest0-21swapper/309:40:443
58130599131,6cyclictest0-21swapper/307:20:523
58130599130,7cyclictest0-21swapper/309:00:543
58130599123,5cyclictest0-21swapper/308:15:543
58130599123,5cyclictest0-21swapper/308:15:543
58130599122,6cyclictest0-21swapper/309:50:383
58130599122,6cyclictest0-21swapper/307:40:573
58130599121,7cyclictest0-21swapper/309:50:013
58130599113,4cyclictest0-21swapper/308:00:503
58130599112,5cyclictest0-21swapper/308:21:023
58130599111,6cyclictest0-21swapper/308:55:483
5813059991,4cyclictest0-21swapper/307:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional