You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-25 - 09:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Nov 25, 2024 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
732435997813,43cyclictest16950irq/224-800000021:10:030
732435997570,2cyclictest0-21swapper/021:00:020
73243599755,3cyclictest131rcu_preempt19:15:030
732435997470,2cyclictest0-21swapper/020:25:030
73243599722,16cyclictest235-21systemd-journal21:25:010
73243599712,4cyclictest738874-21munin-run21:15:010
732435997066,2cyclictest0-21swapper/022:00:020
732440996815,15cyclictest741777-21date22:10:011
73243599686,50cyclictest736003-21modprobe20:20:000
732440996611,21cyclictest744400-21latency_hist23:00:011
732435996616,4cyclictest740739-21latency_hist21:50:020
732435996530,29cyclictest734950-21cron20:00:000
732435996530,29cyclictest734950-21cron20:00:000
73243599652,15cyclictest745895-21munin-run23:30:000
73244099649,18cyclictest740405-21sed21:40:021
73243599642,14cyclictest742289-21munin-run22:20:010
73243599642,14cyclictest737563-21cron20:50:000
73243599642,10cyclictest739128-21modprobe21:20:000
732435996359,2cyclictest0-21swapper/000:00:050
73243599633,18cyclictest739680-21unin-run21:30:000
732440996211,14cyclictest735592-21wc20:10:021
73243599624,10cyclictest743575-21unin-run22:45:020
732435996225,32cyclictest271-21cron00:35:000
732435996219,8cyclictest735732-21sshd20:14:260
732435996219,8cyclictest735732-21sshd20:14:260
732440996110,17cyclictest748024-21munin-run00:10:001
73243599619,12cyclictest748539-21modprobe00:20:010
732435996122,34cyclictest1-21systemd19:40:010
73244099608,14cyclictest746724-21cut23:45:011
73243599605,13cyclictest747185-21grep23:55:010
73243599604,15cyclictest745161-21latency_hist23:15:020
73243599604,13cyclictest735488-21modprobe20:10:000
73243599602,14cyclictest235-21systemd-journal23:45:020
732435996019,35cyclictest748203-21wc00:10:020
73243599601,55cyclictest0-21swapper/023:34:250
73243599601,55cyclictest0-21swapper/019:30:020
732435996012,19cyclictest272-21dbus-daemon22:50:000
73244599593,6cyclictest131rcu_preempt20:00:022
73244599593,6cyclictest131rcu_preempt20:00:022
73243599595,19cyclictest740054-21cat21:35:020
73243599594,12cyclictest734202-21modprobe19:45:010
73243599594,12cyclictest734202-21modprobe19:45:000
73243599593,11cyclictest740156-21cron21:40:000
73243599592,15cyclictest748965-21latency_hist00:25:010
73243599592,14cyclictest744195-21latency_hist22:55:020
732440995810,15cyclictest738818-21latency_hist21:10:021
73243599583,9cyclictest732439-21latency_hist19:10:010
73243599583,15cyclictest235-21systemd-journal19:15:010
73243599583,15cyclictest235-21systemd-journal19:15:010
732435995826,28cyclictest272-21dbus-daemon20:45:010
732435995826,28cyclictest272-21dbus-daemon20:45:010
73243599582,15cyclictest734698-21cron19:55:000
73243599582,12cyclictest749043-21sshd00:25:260
732435995813,37cyclictest12-21ksoftirqd/000:00:000
732445995748,6cyclictest30-21ksoftirqd/222:00:012
732440995739,14cyclictest1-21systemd21:15:011
73243599576,14cyclictest748304-21wc00:15:010
732435995740,13cyclictest1-21systemd22:05:010
73243599573,12cyclictest749575-21cron00:40:000
732435995729,23cyclictest743026-21sh22:35:000
732435995717,4cyclictest0-21swapper/020:01:530
73243599571,2cyclictest0-21swapper/020:35:460
732449995647,7cyclictest37-21ksoftirqd/321:05:013
73243599564,11cyclictest742517-21modprobe22:25:000
73243599561,10cyclictest738067-21cron21:00:000
73243599561,10cyclictest738067-21cron21:00:000
73244999551,9cyclictest747467-40mandb00:00:003
73243599554,9cyclictest745349-21cron23:20:000
73243599554,10cyclictest736282-21modprobe20:25:000
73243599553,19cyclictest746398-21cron23:40:010
73243599553,12cyclictest736793-21munin-plugin-st20:35:000
73243599552,10cyclictest733669-21modprobe19:35:000
732435995512,8cyclictest744858-21systemctl23:10:010
73243599551,22cyclictest0-21swapper/022:38:280
73243599543,15cyclictest741797-21cat22:10:010
73243599542,13cyclictest733152-21sshd19:23:260
73244999535,7cyclictest37-21ksoftirqd/321:50:023
73244999534,7cyclictest37-21ksoftirqd/321:56:263
73244999533,12cyclictest37-21ksoftirqd/300:35:013
73244099539,16cyclictest733989-21sed19:40:011
73243599534,9cyclictest742780-21cut22:30:000
73243599533,11cyclictest0-21swapper/023:00:000
73243599524,11cyclictest737832-21cut20:55:000
73243599522,17cyclictest744611-21unin-run23:05:000
73243599522,17cyclictest742164-21latency_hist22:15:020
73243599522,17cyclictest740959-21true21:55:000
73243599522,15cyclictest734423-21sshd19:45:260
73244999513,10cyclictest37-21ksoftirqd/320:15:023
73244999513,10cyclictest37-21ksoftirqd/320:15:013
73244999511,6cyclictest0-21swapper/300:10:013
732445995139,8cyclictest30-21ksoftirqd/223:20:002
73244999506,7cyclictest37-21ksoftirqd/321:20:003
73244999503,8cyclictest37-21ksoftirqd/322:45:033
732449995035,11cyclictest235-21systemd-journal20:05:023
732449995019,3cyclictest150-21pr/ttyS200:30:033
732445995028,16cyclictest235-21systemd-journal22:25:002
732440995032,15cyclictest749555-21sshd00:36:261
73243599502,9cyclictest746856-21latency_hist23:45:020
73243599502,13cyclictest740538-21latency_hist21:45:010
73244999497,8cyclictest37-21ksoftirqd/300:15:023
732449994933,12cyclictest272-21dbus-daemon23:40:023
732449994912,6cyclictest131rcu_preempt19:25:013
73244599499,6cyclictest131rcu_preempt23:10:012
73244599495,23cyclictest748032-21kworker/u8:000:15:022
73244599494,6cyclictest131rcu_preempt00:10:012
73244999486,9cyclictest37-21ksoftirqd/320:00:003
73244999486,9cyclictest37-21ksoftirqd/320:00:003
73244999485,7cyclictest37-21ksoftirqd/320:35:013
73244999482,9cyclictest37-21ksoftirqd/322:20:013
732445994838,7cyclictest30-21ksoftirqd/200:00:022
732440994831,13cyclictest745317-21sshd23:15:261
732440994831,11cyclictest738092-21systemctl21:00:001
732440994831,11cyclictest738092-21systemctl21:00:001
732440994830,14cyclictest748497-21sshd00:15:261
73244999473,9cyclictest37-21ksoftirqd/323:45:013
732449994730,12cyclictest272-21dbus-daemon20:40:003
73244599473,7cyclictest30-21ksoftirqd/219:20:262
732445994734,9cyclictest235-21systemd-journal21:21:262
732445994732,11cyclictest235-21systemd-journal22:55:012
73244599471,5cyclictest131rcu_preempt00:20:012
732445994711,6cyclictest131rcu_preempt23:50:002
732445994710,6cyclictest131rcu_preempt21:10:002
732440994730,14cyclictest743014-21sshd22:34:261
732440994728,14cyclictest734222-21munin-run19:45:021
732440994728,14cyclictest734222-21munin-run19:45:011
732449994636,7cyclictest37-21ksoftirqd/321:55:013
732449994634,9cyclictest37-21ksoftirqd/319:50:013
732449994624,18cyclictest37-21ksoftirqd/322:55:023
732449994611,5cyclictest131rcu_preempt20:55:003
73244599466,8cyclictest30-21ksoftirqd/200:39:592
732445994636,7cyclictest30-21ksoftirqd/220:15:012
732445994636,7cyclictest30-21ksoftirqd/220:15:012
732445994635,7cyclictest30-21ksoftirqd/221:30:022
732445994633,9cyclictest272-21dbus-daemon20:10:012
732445994632,11cyclictest30-21ksoftirqd/222:50:012
732445994631,11cyclictest272-21dbus-daemon22:05:002
732445994627,14cyclictest1-21systemd19:35:002
73244599462,5cyclictest131rcu_preempt23:15:012
73244599461,6cyclictest131rcu_preempt21:45:012
732440994634,6cyclictest1-21systemd00:00:001
732440994633,9cyclictest734683-21sshd19:51:261
732440994631,10cyclictest272-21dbus-daemon23:10:011
732440994630,12cyclictest742778-21munin-run22:30:011
732440994630,12cyclictest741449-21sshd22:00:261
732440994619,22cyclictest735005-21/usr/sbin/munin20:00:011
732440994619,22cyclictest735005-21/usr/sbin/munin20:00:011
73244999456,9cyclictest37-21ksoftirqd/323:50:003
73244999454,10cyclictest37-21ksoftirqd/321:45:013
732449994531,9cyclictest738079-21cron21:00:003
732449994531,9cyclictest738079-21cron21:00:003
73244999453,10cyclictest37-21ksoftirqd/321:30:013
73244999452,8cyclictest37-21ksoftirqd/300:20:023
732449994516,6cyclictest131rcu_preempt20:30:003
73244599456,8cyclictest30-21ksoftirqd/220:45:012
73244599456,8cyclictest30-21ksoftirqd/220:45:002
732445994534,8cyclictest30-21ksoftirqd/221:20:022
732445994532,10cyclictest30-21ksoftirqd/200:35:022
732445994528,13cyclictest272-21dbus-daemon23:00:002
732440994532,10cyclictest748280-21munin-run00:15:011
732440994532,10cyclictest742497-21sshd22:21:261
732440994530,12cyclictest749313-21sshd00:33:251
732440994530,11cyclictest746932-21munin-run23:50:011
732440994530,11cyclictest737325-21munin-run20:45:011
732440994530,11cyclictest737325-21munin-run20:45:001
732440994529,11cyclictest742001-21munin-run22:15:011
732440994529,10cyclictest735749-21munin-run20:15:011
732440994529,10cyclictest735749-21munin-run20:15:011
732440994521,19cyclictest733185-21systemd-run19:25:011
73244999446,8cyclictest37-21ksoftirqd/322:35:003
73244999442,8cyclictest37-21ksoftirqd/323:18:263
732449994428,13cyclictest1-21systemd21:40:013
73244999441,9cyclictest37-21ksoftirqd/319:35:003
73244599446,7cyclictest30-21ksoftirqd/220:20:012
73244599446,6cyclictest131rcu_preempt19:15:022
73244599446,6cyclictest131rcu_preempt19:15:022
732445994435,6cyclictest30-21ksoftirqd/220:30:012
732445994435,6cyclictest30-21ksoftirqd/200:30:002
732445994427,13cyclictest30-21ksoftirqd/221:00:012
732445994427,13cyclictest30-21ksoftirqd/221:00:012
73244599442,11cyclictest30-21ksoftirqd/219:10:012
732440994431,9cyclictest235-21systemd-journal21:30:261
732440994430,9cyclictest744066-21munin-run22:55:011
732440994430,11cyclictest747163-21sshd23:53:331
732440994430,10cyclictest748842-21munin-run00:25:011
732440994430,10cyclictest739395-21sshd21:23:261
732440994429,12cyclictest732848-21sshd19:15:271
732440994429,11cyclictest740953-21munin-run21:55:001
732440994428,13cyclictest737018-21sshd20:37:261
732440994428,12cyclictest739115-21sshd21:18:271
732440994426,12cyclictest734445-21cron19:50:001
732440994424,16cyclictest272-21dbus-daemon20:20:001
732440994420,20cyclictest743670-21sed22:45:021
732440994420,19cyclictest747706-21sshd00:03:261
73244999435,8cyclictest37-21ksoftirqd/322:15:003
73244999434,7cyclictest37-21ksoftirqd/323:30:003
732449994335,6cyclictest361rcuc/319:20:013
732449994333,6cyclictest736307-21latency_hist20:25:013
73244999432,10cyclictest37-21ksoftirqd/320:20:013
73244999431,17cyclictest0-21swapper/320:49:273
73244599435,5cyclictest131rcu_preempt23:40:002
732445994328,11cyclictest30-21ksoftirqd/223:45:012
73244599431,14cyclictest0-21swapper/222:16:012
732440994331,9cyclictest745877-21sshd23:29:261
732440994330,9cyclictest23-21ksoftirqd/119:30:001
732440994330,8cyclictest736289-21munin-run20:25:011
732440994329,10cyclictest749092-21munin-run00:30:021
732440994329,10cyclictest737576-21munin-run20:50:001
732440994327,12cyclictest746440-21munin-run23:40:011
732440994323,15cyclictest738325-21sshd21:00:261
73244999429,4cyclictest131rcu_preempt23:25:023
73244999428,5cyclictest131rcu_preempt23:05:013
73244999426,9cyclictest37-21ksoftirqd/321:10:033
73244999424,9cyclictest37-21ksoftirqd/319:15:003
73244999424,9cyclictest37-21ksoftirqd/319:15:003
732449994231,8cyclictest37-21ksoftirqd/322:00:273
732449994229,10cyclictest1-21systemd19:55:003
732445994234,5cyclictest30-21ksoftirqd/219:45:022
732445994234,5cyclictest30-21ksoftirqd/219:45:022
732445994228,10cyclictest235-21systemd-journal20:46:252
732445994226,12cyclictest235-21systemd-journal21:40:022
732445994217,8cyclictest131rcu_preempt22:10:012
732445994215,8cyclictest30-21ksoftirqd/220:25:002
732440994230,9cyclictest745101-21munin-run23:15:011
732440994230,9cyclictest736554-21munin-run20:30:011
732440994230,9cyclictest732599-21sshd19:10:321
732440994230,9cyclictest732599-21sshd19:10:321
732440994230,9cyclictest363-21snmpd21:47:131
732440994229,9cyclictest235-21systemd-journal21:44:271
732440994228,10cyclictest746161-21munin-run23:35:021
732440994226,11cyclictest743814-21munin-run22:50:001
732440994226,11cyclictest363-21snmpd21:27:251
732440994223,14cyclictest363-21snmpd23:20:311
732435994229,9cyclictest745630-21cron23:25:010
73244999414,7cyclictest37-21ksoftirqd/300:36:313
73244999413,5cyclictest735233-21modprobe20:04:593
732449994114,5cyclictest131rcu_preempt00:05:023
732445994134,5cyclictest30-21ksoftirqd/219:55:022
732445994133,6cyclictest30-21ksoftirqd/219:20:012
732445994113,5cyclictest131rcu_preempt21:52:272
732445994110,6cyclictest131rcu_preempt23:35:002
732440994128,9cyclictest363-21snmpd20:52:031
732440994111,26cyclictest743288-21modprobe22:40:011
732449994028,8cyclictest1-21systemd23:35:003
732449994025,11cyclictest1-21systemd22:10:013
732445994030,7cyclictest30-21ksoftirqd/220:37:262
732445994029,8cyclictest30-21ksoftirqd/222:40:002
732445994028,9cyclictest30-21ksoftirqd/219:35:262
732445994026,10cyclictest306-21in:imuxsock23:55:012
732440994030,7cyclictest735245-21munin-run20:05:001
732440994028,9cyclictest363-21snmpd21:59:351
732440994028,9cyclictest363-21snmpd19:31:461
732440994026,11cyclictest742258-21systemctl22:20:001
732449993932,5cyclictest37-21ksoftirqd/319:40:023
732449993930,5cyclictest306-21in:imuxsock23:00:013
73244999392,9cyclictest37-21ksoftirqd/321:25:003
73244999391,8cyclictest0-21swapper/319:30:003
732449993911,5cyclictest131rcu_preempt23:15:003
73244599398,4cyclictest131rcu_preempt22:26:262
732445993927,9cyclictest30-21ksoftirqd/222:15:012
73244599391,9cyclictest30-21ksoftirqd/222:35:012
73244599391,9cyclictest30-21ksoftirqd/221:30:002
732445993911,5cyclictest131rcu_preempt23:00:262
732440993926,10cyclictest363-21snmpd23:02:301
73244999385,9cyclictest37-21ksoftirqd/300:23:533
732449993821,12cyclictest235-21systemd-journal22:26:033
73244999381,8cyclictest0-21swapper/321:15:003
732449993813,5cyclictest131rcu_preempt22:40:003
732449993810,5cyclictest131rcu_preempt20:45:013
732449993810,5cyclictest131rcu_preempt20:45:013
73244599383,7cyclictest30-21ksoftirqd/200:01:262
73244599383,6cyclictest131rcu_preempt23:25:022
732445993823,10cyclictest235-21systemd-journal20:35:002
73244599381,6cyclictest0-21swapper/221:05:012
732440993825,10cyclictest363-21snmpd20:32:001
73244999378,4cyclictest131rcu_preempt22:50:003
73244999371,9cyclictest37-21ksoftirqd/322:21:023
73244599376,4cyclictest131rcu_preempt23:27:362
73244599372,6cyclictest30-21ksoftirqd/222:43:262
732445993725,8cyclictest30-21ksoftirqd/221:50:002
73244599371,5cyclictest0-21swapper/219:25:342
73244999361,6cyclictest0-21swapper/323:08:413
73244999361,12cyclictest0-21swapper/321:33:373
732449993610,5cyclictest131rcu_preempt19:41:273
732449993610,5cyclictest131rcu_preempt19:41:263
732445993626,7cyclictest737829-21modprobe20:55:002
73244999356,9cyclictest37-21ksoftirqd/323:52:363
732449993524,7cyclictest37-21ksoftirqd/319:10:023
73244599352,10cyclictest30-21ksoftirqd/200:23:542
732445993516,15cyclictest30-21ksoftirqd/220:02:092
732440993526,6cyclictest732571-21latency_hist19:10:021
732445993310,5cyclictest131rcu_preempt21:15:032
732445993123,5cyclictest291rcuc/219:48:272
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional