You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 14:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 27, 2024 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8277210476,9sleep10-21swapper/119:05:011
10295210175,9sleep30-21swapper/319:08:303
1020228564,6sleep20-21swapper/219:07:162
1036628270,8sleep00-21swapper/019:09:220
47562680,0sleep30-21swapper/323:44:013
317642440,0sleep10-21swapper/122:43:091
1049899411,20cyclictest2052-21ssh22:45:010
1049899390,20cyclictest0-21swapper/022:31:140
10498993815,3cyclictest0-21swapper/022:41:000
1050399310,0cyclictest0-21swapper/123:24:151
1049899294,20cyclictest1952-21irqstats22:25:210
1050899250,20cyclictest0-21swapper/321:45:103
10508992421,2cyclictest39-21ksoftirqd/323:15:073
1049899244,18cyclictest25740-21cron22:39:590
1050899230,1cyclictest0-21swapper/300:09:293
1049899236,7cyclictest141rcu_preempt19:15:000
1049899232,19cyclictest26714-21grep19:40:270
1049899231,20cyclictest7744-21snmpd19:32:000
10508992219,2cyclictest39-21ksoftirqd/322:20:003
10498992219,1cyclictest0-21swapper/019:45:300
10498992219,1cyclictest0-21swapper/019:36:220
10498992219,1cyclictest0-21swapper/019:36:210
1050599211,19cyclictest27450-21apt19:45:122
10508992010,7cyclictest472-21avahi-daemon00:00:043
1050899200,1cyclictest0-21swapper/300:20:033
1050599203,7cyclictest224-21systemd-journal23:28:322
10505992018,1cyclictest0-21swapper/220:25:112
1049899200,1cyclictest0-21swapper/021:35:120
1049899200,1cyclictest0-21swapper/021:35:120
10508991913,2cyclictest39-21ksoftirqd/321:17:163
1050899190,18cyclictest0-21swapper/323:08:403
10505991916,1cyclictest5779-21latency_hist21:50:002
10505991914,5cyclictest37850irq/131-snd_hda00:07:172
1050399190,1cyclictest0-21swapper/123:59:421
1050399190,19cyclictest0-21swapper/122:03:461
1049899199,8cyclictest20979-21apache223:37:280
1049899190,19cyclictest0-21swapper/021:45:200
1050899185,3cyclictest20472-21cut00:35:123
10508991815,2cyclictest381rcuc/323:00:123
1050899180,17cyclictest0-21swapper/323:53:583
1050399180,1cyclictest0-21swapper/100:31:071
10498991813,1cyclictest141rcu_preempt20:30:000
1049899180,15cyclictest0-21swapper/023:01:310
1050899171,14cyclictest0-21swapper/300:31:593
1050599179,4cyclictest224-21systemd-journal23:50:002
1050599177,4cyclictest4372-21munin-run21:29:592
1050599170,1cyclictest0-21swapper/223:58:242
1050599170,1cyclictest0-21swapper/222:18:562
10503991716,1cyclictest0-21swapper/121:32:511
1050399170,3cyclictest141rcu_preempt00:05:561
1050399170,17cyclictest0-21swapper/122:09:011
1050399170,0cyclictest0-21swapper/100:03:471
1050899169,2cyclictest39-21ksoftirqd/321:20:143
1050899163,3cyclictest141rcu_preempt22:00:053
10508991613,3cyclictest15200-21kworker/u8:220:45:083
10508991613,2cyclictest39-21ksoftirqd/323:57:213
10508991613,2cyclictest39-21ksoftirqd/322:34:593
1050899160,15cyclictest0-21swapper/322:54:283
10505991615,1cyclictest29856-21kworker/2:022:49:382
1050599161,14cyclictest3667-21apache_accesses23:05:112
1050599160,15cyclictest3027-21diskmemload21:44:272
1050599160,14cyclictest0-21swapper/221:30:202
1050399160,15cyclictest0-21swapper/122:54:271
1050399160,15cyclictest0-21swapper/100:12:191
1050399160,14cyclictest0-21swapper/123:25:501
1050899158,5cyclictest0-21swapper/319:20:133
10508991513,1cyclictest39-21ksoftirqd/322:10:073
10508991512,2cyclictest39-21ksoftirqd/321:50:223
1050899150,0cyclictest141rcu_preempt21:35:063
1050899150,0cyclictest141rcu_preempt21:35:063
1050599152,13cyclictest943-21kworker/2:122:31:122
10505991510,2cyclictest31931-21wc23:40:192
1050599150,15cyclictest0-21swapper/200:25:242
10503991510,3cyclictest15579-21cat22:15:001
1050399150,2cyclictest0-21swapper/120:40:151
1049899150,1cyclictest141rcu_preempt23:05:150
1049899150,15cyclictest0-21swapper/000:20:020
1050899147,1cyclictest8049-21runrttasks23:45:233
1050899146,5cyclictest39-21ksoftirqd/322:40:453
1050899141,3cyclictest20115-21strings20:35:213
10508991413,1cyclictest0-21swapper/322:07:173
10508991411,2cyclictest6207-21wc23:25:193
10508991411,2cyclictest39-21ksoftirqd/321:05:003
10508991410,3cyclictest3872-21head20:00:313
1050899140,0cyclictest141rcu_preempt21:11:133
1050599140,13cyclictest23098-21sh00:35:592
1050599140,13cyclictest0-21swapper/221:10:062
1050399144,8cyclictest0-21swapper/122:35:171
1050399140,2cyclictest0-21swapper/100:26:021
1050399140,1cyclictest0-21swapper/123:41:191
1050399140,14cyclictest0-21swapper/100:37:481
1050399140,0cyclictest0-21swapper/123:48:541
1049899140,0cyclictest0-21swapper/021:19:440
1050899139,3cyclictest39-21ksoftirqd/323:20:223
1050899139,2cyclictest27371-21wc19:45:013
1050899139,1cyclictest39-21ksoftirqd/323:11:483
1050899137,6cyclictest39-21ksoftirqd/300:17:343
1050899137,3cyclictest381rcuc/323:30:213
10508991312,1cyclictest39-21ksoftirqd/300:11:193
10508991311,1cyclictest39-21ksoftirqd/319:15:213
10508991311,1cyclictest39-21ksoftirqd/300:25:053
10508991310,2cyclictest9018-21munin-plugin-st20:15:013
10508991310,2cyclictest5209-21cut20:05:163
10508991310,2cyclictest26939-21grep20:50:173
10508991310,2cyclictest18416-21ssh22:35:203
1050899130,4cyclictest141rcu_preempt21:25:433
1050899130,1cyclictest141rcu_preempt22:46:463
1050599139,4cyclictest37850irq/131-snd_hda23:24:032
1050599139,3cyclictest8275-21grep21:50:312
1050599138,5cyclictest254152sleep222:58:412
1050599138,4cyclictest206302sleep223:34:222
1050599138,2cyclictest30907-21wget00:20:132
1050599131,1cyclictest3027-21diskmemload21:37:392
1050599131,1cyclictest3027-21diskmemload21:37:382
10505991311,1cyclictest0-21swapper/222:41:082
10505991310,2cyclictest9523-21tail22:10:252
10505991310,2cyclictest31641-21cut21:00:202
10505991310,2cyclictest22939-21grep19:35:152
10505991310,2cyclictest22939-21grep19:35:152
10505991310,2cyclictest15778-21grep19:20:152
10505991310,2cyclictest1132-21grep21:05:132
1050599130,12cyclictest724-21wc22:05:282
1050399139,3cyclictest16024-21sed23:50:171
1050399139,2cyclictest4559-21wc21:30:001
1050399131,10cyclictest28481-21fschecks_time23:00:181
10503991310,2cyclictest24355-21head00:15:231
10503991310,2cyclictest24073-21cron22:20:001
1050399130,12cyclictest31195-21tr21:00:171
1049899139,3cyclictest30274-21sed20:55:290
1049899139,3cyclictest11210-21head20:15:290
1049899133,5cyclictest28334-21perf20:55:000
10498991311,1cyclictest13-21ksoftirqd/022:11:440
10498991311,1cyclictest13-21ksoftirqd/020:40:000
10498991310,2cyclictest4802-21grep20:05:150
10498991310,2cyclictest24389-21wc00:15:230
1049899130,3cyclictest0-21swapper/023:47:250
1049899130,3cyclictest0-21swapper/000:30:100
1050899129,2cyclictest29545-21cron19:50:003
1050899129,2cyclictest24937-21tr22:20:123
1050899129,2cyclictest14547-21grep20:25:123
1050899129,2cyclictest1174-21grep21:05:143
1050899128,3cyclictest9594-21grep20:15:133
1050899128,3cyclictest28837-21tr20:55:153
10508991211,1cyclictest39-21ksoftirqd/323:36:303
1050899121,10cyclictest830-21/usr/sbin/munin22:25:183
10508991210,2cyclictest39-21ksoftirqd/321:30:143
1050599129,2cyclictest999-21wc00:00:252
1050599129,2cyclictest856-21grep19:55:242
1050599129,2cyclictest26037-21cat19:40:212
1050599129,2cyclictest2422-21grep20:00:152
1050599129,2cyclictest22574-21cut20:40:242
1050599129,2cyclictest12392-21tail19:10:282
1050599128,3cyclictest28392-21cron21:25:002
1050599128,3cyclictest18337-21cat22:35:202
1050599128,3cyclictest11597-21cron20:20:002
1050599126,5cyclictest9009-21cron20:15:012
1050599123,2cyclictest30253-21apt-config00:20:012
1050599122,8cyclictest19703-21fschecks_time20:35:182
10505991211,1cyclictest0-21swapper/222:04:092
1050599120,11cyclictest6199-21sed20:05:272
1050599120,11cyclictest3027-21diskmemload23:35:142
1050599120,11cyclictest15128-21taskset00:10:092
1050599120,11cyclictest14560-21basename21:55:092
1050599120,10cyclictest0-21swapper/223:03:442
1050399129,2cyclictest6224-21ssh21:50:131
1050399129,2cyclictest5464-21grep21:10:281
1050399129,2cyclictest4148-21wc22:45:301
1050399129,2cyclictest28430-21cat21:25:001
1050399129,2cyclictest22085-21grep21:40:131
1050399129,2cyclictest15207-21df21:55:131
1050399129,2cyclictest11831-21cat19:10:211
1050399129,2cyclictest10589-21grep20:15:221
1050399128,3cyclictest26727-21grep19:40:281
1050399128,2cyclictest15469-21rm23:11:261
1050399127,3cyclictest21190-21munin-run20:39:591
1050399124,4cyclictest224-21systemd-journal23:30:001
1050399120,11cyclictest17601-21cron19:25:001
1049899129,2cyclictest4516-21cat21:30:000
1049899129,2cyclictest3515-21chrt21:10:110
1049899129,2cyclictest28396-21sh21:25:000
1049899129,2cyclictest24607-21grep22:00:250
1049899129,2cyclictest20826-21tr23:15:140
1049899129,2cyclictest20487-21uname22:55:230
1049899129,2cyclictest19518-21grep19:25:290
1049899129,2cyclictest17349-21head20:30:190
1049899129,2cyclictest16306-21sensors21:55:280
1049899129,2cyclictest13573-21tr23:10:230
1049899129,2cyclictest11547-21systemd00:07:230
1049899129,2cyclictest10983-21tr22:50:120
1049899128,3cyclictest2008-21grep21:05:200
1049899128,3cyclictest1017-21wc19:55:260
1049899128,2cyclictest7035-21grep21:50:180
1049899125,6cyclictest5201-21sh21:30:190
1049899123,2cyclictest224-21systemd-journal19:25:000
1049899121,6cyclictest26860-21perf00:40:010
1049899120,7cyclictest22938-21idleruntime-cro23:55:000
1049899120,11cyclictest26791-21grep22:20:300
1050899118,2cyclictest30524-21grep19:50:173
1050899118,2cyclictest21395-21ssh22:55:343
1050899118,2cyclictest17290-21sort20:30:193
1050899117,3cyclictest18101-21tr19:25:153
1050899117,3cyclictest15287-21cut21:55:163
1050899113,1cyclictest141rcu_preempt21:40:283
1050599118,2cyclictest8981-21date22:30:002
1050599118,2cyclictest37850irq/131-snd_hda22:50:122
1050599118,2cyclictest30205-21grep20:55:292
1050599118,2cyclictest26946-21tail20:50:172
1050599118,2cyclictest24916-21ssh22:20:132
1050599118,2cyclictest22695-21ssh23:15:292
1050599118,2cyclictest21781-21ssh23:54:072
1050599118,2cyclictest17837-21ssh21:18:252
1050599118,2cyclictest15897-21chrt23:11:462
1050599118,2cyclictest15216-21cron19:20:002
1050599118,2cyclictest13437-21aten2_r7power_c00:30:122
1050399118,2cyclictest5054-21sh23:05:221
1050399118,2cyclictest30917-21grep19:50:201
1050399118,2cyclictest2658-21rm21:47:411
1050399118,2cyclictest2601-21cat20:00:151
1050399118,2cyclictest23728-21ssh23:16:171
1050399118,2cyclictest23329-21sh22:57:041
1050399118,2cyclictest23112-21expr19:35:161
1050399118,2cyclictest23112-21expr19:35:161
1050399118,2cyclictest22174-21seq19:32:021
1050399118,2cyclictest17030-21ssh21:36:481
1050399118,2cyclictest17030-21ssh21:36:471
1050399118,2cyclictest1567-21grep21:05:151
1050399118,2cyclictest14805-21sed19:15:281
1050399118,2cyclictest11964-21rm21:15:111
1050399117,2cyclictest9004-21cron20:15:011
1050399114,3cyclictest224-21systemd-journal22:25:001
1050399110,11cyclictest0-21swapper/122:30:341
1050399110,11cyclictest0-21swapper/122:26:191
1049899119,1cyclictest30052-21grep19:50:130
1049899119,1cyclictest22141-21grep20:40:200
1049899118,2cyclictest8446-21ssh00:25:530
1049899118,2cyclictest8164-21ssh23:26:150
1049899118,2cyclictest4228-21taskset20:04:400
1049899118,2cyclictest3462-21ssh00:02:100
1049899118,2cyclictest29992-21grep23:20:180
1049899118,2cyclictest26643-21ssh23:56:210
1049899118,2cyclictest1939-21taskset23:41:500
1049899118,2cyclictest16122-21cut22:15:110
1049899118,2cyclictest15516-21cut00:10:150
1049899118,2cyclictest14815-21sed19:15:270
1049899118,2cyclictest13621-21chrt23:30:160
1049899117,3cyclictest32659-21sed21:00:310
1049899110,1cyclictest0-21swapper/022:05:140
1049899110,1cyclictest0-21swapper/020:24:010
1049899110,10cyclictest22893-21ssh21:40:210
1050899109,1cyclictest20970-21apache219:10:033
1050899108,1cyclictest15262sleep319:55:403
1050899107,2cyclictest39-21ksoftirqd/320:40:203
1050899107,2cyclictest21530-21sed19:30:243
1050899107,2cyclictest13682-21taskset20:21:013
1050899104,3cyclictest23165-21sh19:35:173
1050899104,3cyclictest23165-21sh19:35:163
1050599108,1cyclictest30853-21cat19:50:202
1050599108,1cyclictest18842-21expr19:25:212
1050599108,1cyclictest11869-21basename20:20:102
1050599107,2cyclictest21136-21basename19:30:182
1050599105,4cyclictest16858-21sort20:30:152
1050599100,2cyclictest0-21swapper/220:45:112
1050399108,1cyclictest32437-21cut19:55:151
1050399108,1cyclictest31106-21expr00:20:131
1050399108,1cyclictest29408-21taskset19:48:171
1050399108,1cyclictest21907-21ssh23:35:091
1050399108,1cyclictest14718-21expr20:25:141
1050399107,2cyclictest28593-21cut20:55:131
1050399107,2cyclictest26241-21wget20:50:131
1050399107,2cyclictest24386-21sed20:45:131
1050399107,2cyclictest12643-21grep20:20:161
1050399106,3cyclictest19567-21sed19:25:301
10498991010,0cyclictest0-21swapper/020:10:120
105039996,2cyclictest16824-21cpu20:30:151
105039993,5cyclictest7744-21snmpd20:07:341
104989997,1cyclictest24418-21chrt20:45:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional