You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 08:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Nov 22, 2024 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
90062245166,25sleep10-21swapper/119:05:391
92722241164,25sleep00-21swapper/019:09:050
93372239165,22sleep30-21swapper/319:09:563
89902204167,25sleep20-21swapper/219:05:282
3049921650,4sleep2945199cyclictest21:10:012
1122121540,4sleep1944499cyclictest21:38:461
120582650,5sleep1944499cyclictest00:10:241
179362610,3sleep10-21swapper/100:25:131
194262600,3sleep019431-21gltestperf23:10:130
239522580,3sleep1221ktimersoftd/123:20:171
138092530,3sleep339-21rcuc/319:15:253
281542430,3sleep2945199cyclictest19:50:122
944499418,24cyclictest2039-21sshd23:48:171
944499399,11cyclictest30108-21diskstats23:35:131
9444993914,22cyclictest12531-21/usr/sbin/munin22:55:111
944499389,25cyclictest581-21systemd-journal23:05:011
944499388,16cyclictest27333-21sh23:30:001
9444993812,22cyclictest20755-21perf23:15:021
9444993810,24cyclictest27332-21cut22:15:151
943899388,14cyclictest21997-21munin-run00:35:000
944499377,27cyclictest16789-21kernelversion21:50:211
9444993717,17cyclictest12625-21/usr/sbin/munin21:40:201
9444993613,20cyclictest25497-21awk19:45:011
9444993613,11cyclictest4346-21/usr/sbin/munin20:10:261
9444993613,11cyclictest21415-21ntp_states22:00:221
9444993611,7cyclictest14490-21crond19:20:001
943899369,13cyclictest27679-21munin-run19:50:010
9444993514,17cyclictest7612-21ntpq20:15:231
944499348,14cyclictest23772-21rt-features22:05:251
944499348,11cyclictest0-21swapper/100:30:201
9444993415,15cyclictest22234-21smart_sda23:15:241
9444993313,17cyclictest16707-21awk19:25:011
944499329,20cyclictest31326-21gltestperf21:10:171
944499329,19cyclictest4113-21sadc20:10:011
944499329,13cyclictest6387-21runrttasks23:57:451
944499327,14cyclictest29385-21expr22:20:131
944499327,12cyclictest15719-21latency_hist21:50:021
9444993214,4cyclictest32587-21/usr/sbin/munin22:25:231
9444993213,16cyclictest1160-21grep21:15:161
9444993210,18cyclictest19831-21tr20:45:111
944499319,9cyclictest25906-21grep22:10:241
944499319,12cyclictest277372cstates23:30:161
944499316,6cyclictest0-21swapper/121:20:121
9444993121,6cyclictest10291-21smart_sda00:05:251
9444993111,7cyclictest19208-21ntp_states21:55:191
9444993111,17cyclictest760-21fgrep23:40:231
9444993111,17cyclictest760-21fgrep23:40:231
9444993110,18cyclictest9140-21awk21:35:011
9444993110,18cyclictest9140-21awk21:35:011
9444993110,17cyclictest581-21systemd-journal22:43:101
944499308,12cyclictest18143-21unixbench_singl23:05:261
9444993011,7cyclictest25712-21/usr/sbin/munin19:45:201
9444993010,17cyclictest22920-21cut20:50:201
301072300,7sleep081ktimersoftd/022:20:190
107172300,1sleep20-21swapper/219:10:222
9444992916,9cyclictest13490-21diskstats20:30:141
9444992912,14cyclictest21097-21awk19:35:011
9444992911,6cyclictest28525-21/usr/sbin/munin21:05:261
9444992911,6cyclictest21315-21/usr/sbin/munin19:35:121
943899291,13cyclictest13546-21idleruntime-cro21:45:000
945199282,9cyclictest0-21swapper/221:15:162
944499287,12cyclictest32268-21cat20:00:021
944499285,11cyclictest0-21swapper/120:55:131
9444992820,5cyclictest17882-21diskstats20:40:141
9444992814,5cyclictest4445-21ntp_states22:35:211
9444992814,5cyclictest4445-21ntp_states22:35:211
943899289,16cyclictest14500-21munin-run19:20:010
284652280,4sleep2945199cyclictest23:30:232
944499278,4cyclictest0-21swapper/119:25:251
944499278,16cyclictest10746-21if_enp2s022:50:191
944499277,6cyclictest9294-21fschecks_time20:20:151
944499276,13cyclictest0-21swapper/100:17:311
9444992718,5cyclictest8118-21diskstats22:45:131
9444992711,13cyclictest7486-21sh00:00:141
9444992710,14cyclictest12200-21smart_sda20:25:231
9444992710,14cyclictest12200-21smart_sda20:25:231
944499263,12cyclictest86750irq/128-enp1s0-20:35:191
9444992618,5cyclictest32649-21diskstats20:00:161
9444992617,6cyclictest2466-21/usr/sbin/munin23:50:091
9444992617,5cyclictest15981-21diskstats00:20:151
9451992511,5cyclictest86550irq/126-enp1s0-23:40:132
9451992511,5cyclictest86550irq/126-enp1s0-23:40:132
944499257,10cyclictest86850irq/129-enp1s0-00:35:121
944499255,12cyclictest86750irq/128-enp1s0-19:50:201
944499254,9cyclictest0-21swapper/121:00:161
9444992517,5cyclictest2449-21smart_sda22:30:281
9444992515,7cyclictest848-21sssd_nss19:10:141
9444992515,7cyclictest6014-21ntp_offset21:25:221
943899256,16cyclictest25148-21munin-run23:25:010
945199241,8cyclictest0-21swapper/219:48:022
945599231,11cyclictest0-21swapper/322:25:143
945199236,8cyclictest86550irq/126-enp1s0-00:35:012
945199235,9cyclictest0-21swapper/200:15:202
945199231,14cyclictest0-21swapper/200:25:252
945199231,10cyclictest0-21swapper/221:01:042
945599221,4cyclictest6559-21kworker/u8:100:13:043
945199229,4cyclictest311ktimersoftd/222:10:022
945199227,6cyclictest12050irq/123-ahci[0022:50:022
945199227,3cyclictest0-21swapper/219:39:122
945199221,7cyclictest17453-21sort20:40:022
943899225,5cyclictest0-21swapper/023:54:490
943899224,15cyclictest2199-21awk23:50:010
943899221,8cyclictest0-21swapper/023:58:210
943899221,11cyclictest16185-21tune2fs00:20:150
43912221,2sleep214634-21snmpd21:20:462
945599213,10cyclictest24116-21/usr/sbin/munin20:55:163
945599211,6cyclictest0-21swapper/322:59:253
9455992115,3cyclictest28260-21hwlatdetect23:30:213
945199219,4cyclictest86850irq/129-enp1s0-23:15:252
945199219,3cyclictest0-21swapper/220:05:302
945199218,3cyclictest88050irq/133-enp2s0-22:42:542
945199216,4cyclictest0-21swapper/221:59:402
945199213,3cyclictest0-21swapper/200:23:482
945199212,9cyclictest30-21rcuc/222:15:202
945199211,8cyclictest0-21swapper/221:27:362
945199211,7cyclictest16692-21latency_hist19:25:012
945199211,10cyclictest0-21swapper/220:53:442
9451992110,2cyclictest86850irq/129-enp1s0-23:25:182
943899217,11cyclictest22750-21kernelversion20:50:190
943899213,15cyclictest26109-21idleruntime-cro21:00:020
191212211,3sleep219189-21cut23:10:122
114722211,16sleep311615-21hddtemp_smartct20:25:143
114722211,16sleep311615-21hddtemp_smartct20:25:143
945599203,8cyclictest8755-21cut22:45:193
945599202,8cyclictest0-21swapper/322:30:233
945599201,9cyclictest19215-21grep19:30:103
945599201,16cyclictest7428-21grep00:00:153
945599201,16cyclictest17648-21ntp_states23:05:233
945599201,11cyclictest9020-21/usr/sbin/munin00:05:193
945599201,10cyclictest30493-21sadc21:10:003
945199209,2cyclictest0-21swapper/221:40:142
945199208,5cyclictest86550irq/126-enp1s0-20:55:102
945199208,3cyclictest32-21ksoftirqd/223:05:272
945199208,2cyclictest0-21swapper/222:00:162
945199206,6cyclictest12050irq/123-ahci[0020:50:022
945199206,5cyclictest3285-21cat22:35:012
945199204,8cyclictest86550irq/126-enp1s0-21:15:012
945199204,3cyclictest0-21swapper/220:22:342
945199203,9cyclictest9702-21diskstats21:35:122
945199203,3cyclictest32-21ksoftirqd/222:56:582
945199202,2cyclictest30-21rcuc/223:00:202
945199201,9cyclictest311ktimersoftd/200:35:272
945199201,6cyclictest30-21rcuc/219:35:012
945199201,16cyclictest24735-21rt-features19:40:262
945199201,12cyclictest0-21swapper/223:46:462
943899209,2cyclictest6925-21fschecks_count20:15:160
943899208,3cyclictest4633-21sensors_temp22:35:230
943899208,3cyclictest4633-21sensors_temp22:35:230
943899206,4cyclictest0-21swapper/022:27:440
943899204,5cyclictest0-21swapper/021:30:500
943899204,5cyclictest0-21swapper/021:30:500
943899202,9cyclictest23779-21gunzip22:05:240
943899202,10cyclictest27377-21cut22:15:140
943899202,10cyclictest13433-21/usr/sbin/munin00:15:280
943899201,7cyclictest2780-21cpu21:20:110
943899201,6cyclictest0-21swapper/022:12:560
9438992016,2cyclictest6016-21ntpq21:25:210
943899201,16cyclictest7545-21munin-run22:45:010
943899201,11cyclictest0-21swapper/000:35:330
943899201,10cyclictest9337-21date21:35:010
943899201,10cyclictest14634-21snmpd19:35:020
943899201,10cyclictest11834-21kworker/0:123:05:130
943899201,10cyclictest0-21swapper/021:45:530
279832200,2sleep30-21swapper/321:01:083
945599195,4cyclictest26708-21timerandwakeup23:25:243
945599193,3cyclictest3457-21grep22:35:123
945599193,3cyclictest3457-21grep22:35:123
945599192,9cyclictest0-21swapper/320:16:493
945599191,8cyclictest25989-21find22:10:253
945599191,7cyclictest0-21swapper/323:03:393
9455991914,3cyclictest29044-21grep19:50:213
9455991912,4cyclictest30950-21grep19:55:163
9455991912,4cyclictest15104-21idleruntime-cro20:35:003
945599191,15cyclictest11557-21/usr/sbin/munin21:40:113
945599191,11cyclictest0-21swapper/321:53:163
945599191,10cyclictest825-21lldpd21:47:313
945599191,10cyclictest26950-21/usr/sbin/munin22:15:263
945599191,10cyclictest18774-21/usr/sbin/munin23:10:163
945599191,10cyclictest18145-21/usr/sbin/munin21:55:193
945199198,3cyclictest30-21rcuc/222:35:212
945199198,3cyclictest30-21rcuc/222:35:212
945199192,9cyclictest30-21rcuc/222:10:252
945199192,10cyclictest32-21ksoftirqd/221:48:022
945199191,9cyclictest5058-21basename20:10:182
945199191,8cyclictest0-21swapper/222:20:122
945199191,10cyclictest0-21swapper/200:00:332
943899199,3cyclictest0-21swapper/021:10:160
943899199,2cyclictest6806-21/usr/sbin/munin00:00:220
943899198,3cyclictest17464-21grep21:50:270
943899197,3cyclictest0-21swapper/023:33:570
943899195,4cyclictest16235-21unixbench_multi19:20:260
943899194,9cyclictest0-21swapper/020:34:520
943899191,9cyclictest0-21swapper/000:10:180
9438991912,4cyclictest3573-21/usr/sbin/munin20:05:250
9438991912,4cyclictest31238-21grep23:35:260
943899191,10cyclictest30296-21sshd21:07:390
943899191,10cyclictest15513-21pmu-power23:00:210
943899191,10cyclictest11005-21awk22:50:230
943899190,17cyclictest0-21swapper/023:15:170
185742191,2sleep20-21swapper/219:26:052
143012191,2sleep20-21swapper/219:17:382
945599184,5cyclictest30494-21diskmemload21:10:183
945599183,12cyclictest20030-21/usr/sbin/munin00:30:233
945599181,9cyclictest9980-21/usr/sbin/munin22:50:283
945599181,7cyclictest7702-21cat22:45:023
945599181,7cyclictest0-21swapper/300:39:403
945599181,7cyclictest0-21swapper/300:21:453
945599181,4cyclictest0-21swapper/323:50:053
9455991813,3cyclictest22310-21threads23:15:253
945599181,11cyclictest86550irq/126-enp1s0-23:48:383
945599181,11cyclictest0-21swapper/320:50:203
945599181,10cyclictest842-21lldpd21:15:063
9455991810,5cyclictest0-21swapper/321:25:133
945599180,9cyclictest0-21swapper/321:20:223
945199185,2cyclictest86550irq/126-enp1s0-00:13:562
945199183,9cyclictest0-21swapper/223:20:062
945199182,6cyclictest17025-21ntp_states21:50:242
945199182,3cyclictest0-21swapper/222:27:222
945199181,8cyclictest30494-21diskmemload23:55:202
945199181,7cyclictest581-21systemd-journal20:20:012
9451991811,3cyclictest4190-21kworker/u8:200:07:402
945199181,10cyclictest0-21swapper/223:38:232
945199180,8cyclictest32-21ksoftirqd/222:50:162
945199180,4cyclictest30-21rcuc/223:51:572
943899189,2cyclictest25951-21chrt23:25:150
943899188,2cyclictest12354-21cstates22:55:100
943899187,3cyclictest8324-21perl22:45:150
943899187,3cyclictest19728-21/usr/sbin/munin20:45:130
943899186,4cyclictest0-21swapper/021:57:290
943899185,5cyclictest0-21swapper/000:10:020
943899184,9cyclictest0-21swapper/020:20:180
943899184,8cyclictest0-21swapper/020:28:520
943899184,8cyclictest0-21swapper/020:28:520
943899181,9cyclictest784-21NetworkManager22:00:310
943899181,8cyclictest0-21swapper/021:00:180
943899181,14cyclictest30494-21diskmemload23:40:130
943899181,14cyclictest30494-21diskmemload23:40:130
943899181,11cyclictest0-21swapper/022:30:190
316602181,2sleep30-21swapper/323:39:163
297122181,3sleep00-21swapper/019:53:040
251612180,2sleep00-21swapper/019:41:000
244062180,3sleep30-21swapper/322:08:403
152822180,2sleep30-21swapper/300:18:383
15272181,2sleep00-21swapper/020:00:310
945599172,12cyclictest21315-21/usr/sbin/munin19:35:233
945599171,9cyclictest0-21swapper/322:22:393
945599171,8cyclictest14634-21snmpd20:00:033
945599171,7cyclictest32291-21cat23:40:133
945599171,7cyclictest32291-21cat23:40:133
9455991715,1cyclictest0-21swapper/300:25:153
9455991713,3cyclictest0-21swapper/322:03:423
9455991712,3cyclictest9636-21expr21:35:103
9455991712,3cyclictest18159-21gltestperf20:40:143
9455991712,3cyclictest15859-21sed19:20:233
9455991712,3cyclictest10548-21load19:10:193
945599171,15cyclictest0-21swapper/323:57:353
945599171,13cyclictest2145-21/usr/sbin/munin20:05:133
945199171,6cyclictest18625-21/usr/sbin/munin20:40:222
945199171,3cyclictest0-21swapper/220:33:062
9451991711,3cyclictest7955-21cat21:30:222
9451991711,3cyclictest7955-21cat21:30:222
9451991711,3cyclictest11151-21expr20:25:112
9451991711,3cyclictest11151-21expr20:25:112
945199171,10cyclictest0-21swapper/219:55:192
943899175,4cyclictest18183-21smart_sda19:25:240
943899175,3cyclictest0-21swapper/019:59:470
943899171,7cyclictest5899-21grep20:10:270
943899171,7cyclictest463-21/usr/sbin/munin21:15:190
943899171,7cyclictest17533-21/usr/sbin/munin20:40:200
943899171,6cyclictest0-21swapper/019:10:100
9438991711,4cyclictest18470-21head00:25:160
9438991711,3cyclictest15669-21df20:35:120
74202170,4sleep341-21ksoftirqd/321:30:173
74202170,4sleep341-21ksoftirqd/321:30:173
247762171,2sleep30-21swapper/323:20:293
945599161,3cyclictest814-21sshd19:43:373
9455991611,3cyclictest21110-21cat20:45:233
9455991611,3cyclictest0-21swapper/320:20:193
945599161,13cyclictest0-21swapper/319:45:143
943899162,7cyclictest0-21swapper/019:31:470
186422160,2sleep30-21swapper/319:26:573
945599159,3cyclictest14634-21snmpd20:10:053
9455991510,3cyclictest15429-21sort20:35:103
945199151,5cyclictest32301-21/usr/sbin/munin20:00:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional