You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-11 - 23:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 11, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27379210375,9sleep30-21swapper/307:05:333
2757729279,9sleep10-21swapper/107:08:081
2558828669,13sleep20-21swapper/207:05:082
166452840,0sleep30-21swapper/310:56:323
2757628370,9sleep00-21swapper/007:08:060
2781799270,1cyclictest0-21swapper/309:50:063
2781799240,23cyclictest0-21swapper/310:20:033
27811992410,13cyclictest20421-21runrttasks10:32:161
190642230,0chrt0-21swapper/109:35:201
2781599210,21cyclictest0-21swapper/212:17:342
2781599210,20cyclictest0-21swapper/211:08:542
2781199214,4cyclictest141rcu_preempt11:33:511
2781599200,19cyclictest0-21swapper/211:00:182
27817991917,1cyclictest21133-21diskmemload10:16:553
2781599190,5cyclictest0-21swapper/212:35:162
2781599190,19cyclictest0-21swapper/210:17:502
2781599190,18cyclictest0-21swapper/209:10:192
2781799180,2cyclictest0-21swapper/309:36:353
27811991817,1cyclictest25-21ksoftirqd/111:40:381
2780799186,6cyclictest5794-21munin-run10:20:000
27815991717,0cyclictest0-21swapper/209:38:452
2781599171,16cyclictest3286-21kworker/2:012:22:382
27815991710,6cyclictest19778-21cut09:05:162
2780799174,6cyclictest652-21perf09:20:010
2781799169,3cyclictest141rcu_preempt11:55:283
2781799162,2cyclictest0-21swapper/311:48:123
27817991611,4cyclictest21633-21entropy11:30:203
2781599165,8cyclictest11636-21apt09:30:012
27815991616,0cyclictest0-21swapper/211:26:092
27815991615,1cyclictest3721-21kworker/2:211:21:492
27815991615,1cyclictest346-21kworker/2:109:59:092
27815991615,1cyclictest346-21kworker/2:109:49:492
27811991615,1cyclictest25-21ksoftirqd/108:35:401
2781199160,15cyclictest0-21swapper/112:19:371
2780799165,6cyclictest14596-21munin-run07:50:010
27817991511,1cyclictest27801-21cyclictest12:18:283
2781799150,1cyclictest0-21swapper/309:18:493
2781599155,4cyclictest21133-21diskmemload10:55:012
27815991514,1cyclictest25546-21kworker/2:210:12:082
27815991513,1cyclictest0-21swapper/209:00:112
2781199150,2cyclictest0-21swapper/110:22:161
27817991412,2cyclictest0-21swapper/311:36:523
27817991410,3cyclictest22034-21grep11:00:303
2781799140,3cyclictest0-21swapper/310:29:253
2781799140,0cyclictest0-21swapper/309:27:483
2781599140,14cyclictest0-21swapper/211:31:132
2781599140,14cyclictest0-21swapper/207:35:202
27811991410,3cyclictest9057-21ssh10:50:181
27811991410,3cyclictest2133-21grep10:15:301
2781199140,1cyclictest0-21swapper/110:02:041
2780799145,5cyclictest224-21systemd-journal08:50:000
2780799140,8cyclictest0-21swapper/007:55:130
2780799140,1cyclictest0-21swapper/011:18:100
21482140,2sleep00-21swapper/009:20:220
2781799135,6cyclictest1291-21runrttasks09:40:273
27817991310,2cyclictest9594-21head10:50:243
27817991310,2cyclictest8055-21df_abs07:35:163
27817991310,2cyclictest30911-21grep07:15:163
27817991310,2cyclictest20282-21wc09:05:213
27817991310,2cyclictest1615-21grep07:20:193
27817991310,2cyclictest15564-21grep07:50:183
27817991310,2cyclictest1510-21grep08:25:273
27817991310,2cyclictest148302sleep308:55:153
27817991310,2cyclictest13355-21chrt09:57:313
27817991310,2cyclictest13339-21grep08:50:303
2781799130,3cyclictest141rcu_preempt11:20:553
2781599139,3cyclictest6922-21grep11:40:262
2781599139,3cyclictest14169-21head07:45:252
2781599139,2cyclictest7418-21grep08:40:172
2781599132,4cyclictest623-21munin-run09:20:002
27815991310,2cyclictest29008-21cut11:55:182
27815991310,2cyclictest17682-21grep10:00:262
27815991310,2cyclictest12932-21wc08:50:252
27815991310,2cyclictest10002-21cut07:40:152
2781599130,1cyclictest0-21swapper/207:55:242
2781599130,0cyclictest0-21swapper/211:38:052
2781199132,7cyclictest19127-21aten2_r7power_c12:10:221
2781199132,7cyclictest12591-21sh12:05:241
2781199131,8cyclictest0-21swapper/110:26:061
27811991310,2cyclictest4482-21wc12:20:201
27811991310,2cyclictest30691-21cron11:10:011
27811991310,2cyclictest30149-21turbostat.cron07:15:001
27811991310,2cyclictest18544-21grep12:30:101
2781199130,12cyclictest15211-21grep07:50:181
27807991311,1cyclictest14921-21ssh12:07:130
27807991310,2cyclictest4426-21tail07:25:270
2780799130,12cyclictest28724-21wc12:15:180
2780799130,12cyclictest19980-21cat09:05:190
2781799129,2cyclictest6477-21sed11:15:273
2781799129,2cyclictest30467-21cut08:20:173
2781799129,2cyclictest20426-21cron11:30:013
2781799129,2cyclictest20070-21tr11:50:113
2781799128,2cyclictest3780-21grep08:30:293
2781799128,2cyclictest17121-21grep10:30:153
2781799124,4cyclictest224-21systemd-journal12:40:013
2781799124,4cyclictest224-21systemd-journal12:15:003
2781799122,9cyclictest5683-21ssh11:40:163
2781799122,5cyclictest224-21systemd-journal10:14:593
27817991211,1cyclictest0-21swapper/307:58:123
27817991210,2cyclictest39-21ksoftirqd/312:01:323
2781599129,2cyclictest8125-21head10:20:262
2781599129,2cyclictest4892-21tr11:15:142
2781599129,2cyclictest4782-21tr08:35:172
2781599129,2cyclictest32500-21tr08:25:162
2781599129,2cyclictest25301-21tr09:40:262
2781599129,2cyclictest19393-21tr12:10:222
2781599129,2cyclictest19057-21grep12:30:122
2781599129,2cyclictest16712-21grep07:50:302
27815991212,0cyclictest0-21swapper/210:37:292
2781599120,11cyclictest30460-21grep10:40:172
2781599120,0cyclictest0-21swapper/212:06:132
2781199129,2cyclictest9208-21sed07:35:261
2781199129,2cyclictest31940-21cron08:25:001
2781199129,2cyclictest30678-21grep07:15:161
2781199129,2cyclictest23368-21grep08:05:201
2781199129,2cyclictest2168-21cat09:20:211
2781199129,2cyclictest21137-21turbostat.cron09:10:011
2781199129,2cyclictest14546-21df10:55:161
2781199129,2cyclictest14092-21sed07:45:251
2781199129,2cyclictest0-21swapper/111:56:511
2781199121,9cyclictest21133-21diskmemload11:18:271
2781199120,3cyclictest0-21swapper/111:39:511
2780799129,2cyclictest6635-21sh08:39:590
2780799129,2cyclictest6352-21basename10:20:130
2780799129,2cyclictest6295-21grep11:40:190
2780799129,2cyclictest28471-21tr08:15:220
2780799129,2cyclictest24470-21cat10:35:240
2780799129,2cyclictest22087-21cat11:30:220
2780799129,2cyclictest21452-21grep08:00:260
2780799129,2cyclictest17362-21grep07:55:010
2780799129,2cyclictest13574-21sh10:55:000
2780799129,2cyclictest13442-21sed09:30:260
2780799129,2cyclictest11879-21sh12:05:000
2780799129,2cyclictest10359-21head09:55:170
2780799128,2cyclictest14935-21ssh10:55:190
2780799128,1cyclictest22864-21cut10:34:590
2780799124,6cyclictest13-21ksoftirqd/011:23:310
27807991211,1cyclictest0-21swapper/007:15:150
27807991210,1cyclictest25088-21grep09:40:210
27807991210,1cyclictest19839-21sort12:10:250
2781799118,2cyclictest7246-21munin-plugin-st07:35:013
2781799118,2cyclictest4717-21ssh10:46:213
2781799118,2cyclictest3675-21ssh09:21:163
2781799118,2cyclictest3504-21ssh12:20:133
2781799118,2cyclictest31683-21ssh09:45:473
2781799118,2cyclictest30851-21sed10:40:203
2781799118,2cyclictest26409-21ssh09:14:203
2781799118,2cyclictest23924-21df10:35:193
2781799118,2cyclictest22202-21ssh10:05:173
2781799118,2cyclictest18398-21aten2_r7power_v12:30:103
2781799118,2cyclictest16942-21ssh09:33:433
2781799118,2cyclictest15829-21grep10:00:133
2781799118,2cyclictest14098-21grep07:45:253
2781799118,2cyclictest12751-21ssh12:25:573
2781799118,2cyclictest12145-21ssh12:05:033
2781799118,2cyclictest10397-21grep08:45:223
2781799118,2cyclictest10170-21tr07:40:153
2781799115,5cyclictest23029-21apache211:14:453
2781799115,2cyclictest278-21systemd-udevd07:28:213
2781799113,4cyclictest224-21systemd-journal09:05:013
2781799111,10cyclictest0-21swapper/311:07:103
2781599119,1cyclictest0-21swapper/209:30:212
2781599118,2cyclictest9970-21cut08:45:202
2781599118,2cyclictest6951-21chrt09:52:242
2781599118,2cyclictest5787-21cut07:30:202
2781599118,2cyclictest4479-21grep07:25:262
2781599118,2cyclictest3348-21chrt09:20:552
2781599118,2cyclictest32542-21apt07:20:012
2781599118,2cyclictest32412-21grep11:10:252
2781599118,2cyclictest24944-21aten2_r7power_p08:10:182
2781599118,2cyclictest20457-21sort08:00:152
2781599118,2cyclictest20083-21cut11:50:112
2781599118,2cyclictest19962-21ssh10:31:442
2781599118,2cyclictest13528-21sh12:26:352
2781599118,2cyclictest13500-21rm11:45:222
2781599117,4cyclictest388-21awk07:20:112
2781599116,4cyclictest32030-21latency_hist08:25:002
2781599115,2cyclictest14024-21apt-get10:55:142
2781599113,8cyclictest24270-21kworker/2:008:31:462
2781199118,2cyclictest9901-21grep08:45:201
2781199118,2cyclictest8601-21grep08:40:271
2781199118,2cyclictest8159-21ssh09:26:121
2781199118,2cyclictest682-21chrt09:47:391
2781199118,2cyclictest4925-21head12:00:191
2781199118,2cyclictest4889-21ssh10:46:351
2781199118,2cyclictest4733-21taskset07:28:261
2781199118,2cyclictest3387-21grep08:30:251
2781199118,2cyclictest31185-21ssh11:10:171
2781199118,2cyclictest28788-21ssh10:11:101
2781199118,2cyclictest28248-21sed09:15:171
2781199118,2cyclictest25274-21ssh10:35:331
2781199118,2cyclictest24213-21ssh11:52:171
2781199118,2cyclictest23354-21ssh10:06:321
2781199118,2cyclictest23255-21ssh09:10:301
2781199118,2cyclictest21212-21cut08:00:211
2781199118,2cyclictest1850-21cat07:20:221
2781199118,2cyclictest17777-21ntpq09:00:181
2781199118,2cyclictest17748-21ssh12:29:571
2781199118,2cyclictest17616-21chrt09:34:331
2781199118,2cyclictest16217-21ssh11:25:151
2781199118,2cyclictest14611-21expr08:55:141
2781199118,2cyclictest13628-21ssh09:57:501
2781199118,2cyclictest13387-21sed08:50:291
2781199118,2cyclictest13228-21sh11:45:101
2781199118,2cyclictest12017-21ssh11:20:241
2781199118,2cyclictest10684-21head07:40:181
2781199116,1cyclictest19510-21apt-get11:00:021
2781199115,5cyclictest18700-21ps07:55:221
2781199113,4cyclictest29660-21apt-get10:40:131
2781199110,1cyclictest0-21swapper/112:36:421
2781199110,10cyclictest5002-21timerandwakeup09:50:221
2781199110,10cyclictest30747-21ssh09:44:591
2780799119,1cyclictest25124-21grep08:10:180
2780799119,1cyclictest16584-21sh10:30:000
2780799118,2cyclictest7269-21sh09:25:130
2780799118,2cyclictest31956-21grep11:10:210
2780799118,2cyclictest30532-21ssh11:35:220
2780799118,2cyclictest30182-21tail11:55:280
2780799118,2cyclictest27917-21wc12:35:220
2780799118,2cyclictest27350-21rm11:06:060
2780799118,2cyclictest26304-21ssh11:53:550
2780799118,2cyclictest2443-21tr08:30:180
2780799118,2cyclictest21547-21rm09:10:120
2780799118,2cyclictest21304-21ssh12:31:130
2780799118,2cyclictest19160-21grep09:35:210
2780799118,2cyclictest15162-21ssh11:46:420
2780799118,2cyclictest12013-21munin-run07:45:000
2780799113,5cyclictest0-21swapper/011:00:020
2780799110,1cyclictest141rcu_preempt10:45:110
2780799110,10cyclictest8955-21ssh12:23:090
2781799108,1cyclictest28791-21grep07:10:203
2781799108,1cyclictest24853-21apache_volume08:10:173
2781799107,2cyclictest6890-21chrt08:40:093
2781799107,2cyclictest22945-21tail08:05:163
2781799100,9cyclictest0-21swapper/308:35:163
2781599108,1cyclictest5653-21sh10:48:002
2781599108,1cyclictest22914-21grep08:05:172
2781599107,2cyclictest7444-21ssh12:01:302
2781599107,2cyclictest14933-21sort08:55:162
2781599107,2cyclictest14648-21ssh10:27:542
2781599106,3cyclictest27504-21tr08:15:142
2781599104,6cyclictest62050irq/128-enp2s0-07:10:292
2781599100,4cyclictest224-21systemd-journal10:06:152
2781199108,1cyclictest27323-21cut08:15:141
2781199108,1cyclictest248152sleep108:10:091
2781199107,2cyclictest5950-21wc07:30:211
2781199107,2cyclictest449-21cut08:25:171
2780799108,2cyclictest32345-21ssh10:41:200
2780799108,1cyclictest7783-21if_enp0s31f608:40:190
2780799108,1cyclictest7628-21cut07:35:150
2780799108,1cyclictest5421-21tr07:30:170
2780799108,1cyclictest3984-21grep09:50:130
2780799108,1cyclictest32469-21tr08:25:150
2780799108,1cyclictest31358-21sshd09:45:250
2780799108,1cyclictest27509-21ssh10:10:180
2780799108,1cyclictest23672sleep007:23:280
2780799108,1cyclictest16494-21ssh11:25:320
2780799108,1cyclictest16208-21cat10:00:140
2780799108,1cyclictest14100-21cat08:55:010
2780799108,1cyclictest11768-21ssh12:25:110
2780799107,2cyclictest29934-21aten2_r7power_p08:20:150
2780799107,2cyclictest22027-21ssh10:05:050
2780799107,2cyclictest17029-21cat09:00:120
278179997,1cyclictest19849-21cut08:00:113
278179996,2cyclictest27261-21wget08:15:133
278079998,1cyclictest12240-21sed08:50:180
278079997,1cyclictest28294-21expr07:10:180
278079997,1cyclictest22406-21grep08:05:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional