You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Apr 03, 2025 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
834999517,35cyclictest11155-21sendmail-msp21:20:151
834999472,38cyclictest24168-21sendmail-msp20:00:121
836099452,26cyclictest25866irq/58-eth022:30:103
834999452,37cyclictest3364-21sendmail-msp22:00:181
834999431,34cyclictest17976-21sendmail-msp19:20:131
834399422,13cyclictest13390-21apt-get19:15:150
835699404,32cyclictest12481-21munin-run20:55:012
835699391,10cyclictest8455-21ssh23:35:142
836099382,22cyclictest25866irq/58-eth023:55:153
835699382,10cyclictest15842-21fschecks_time22:10:122
834399381,11cyclictest20036-21ssh22:35:010
8360993719,10cyclictest25866irq/58-eth000:30:093
8349993717,10cyclictest7313-21sendmail-msp21:40:161
834399372,4cyclictest23326-21apt-get21:30:120
835699361,8cyclictest30929-21kthreadcore22:40:272
835699361,13cyclictest12817-21tr21:20:242
834399361,3cyclictest601-21rngd20:15:130
834399361,32cyclictest14733-21cut00:00:270
834399360,12cyclictest19563-21apt-get19:55:100
835699351,8cyclictest20532-21sendmail-msp23:21:332
834399353,28cyclictest24503-21sendmail-msp23:00:180
834399351,9cyclictest20532-21sendmail-msp23:20:200
834399351,5cyclictest0-21swapper/020:25:140
834399351,4cyclictest27863-21apt-get19:30:150
835699343,5cyclictest8384-21ssh21:40:232
835699341,7cyclictest3967-21ssh22:00:232
8356993411,19cyclictest1227-21ssh21:56:532
835699340,9cyclictest20523-21fschecks_count22:35:162
834999345,21cyclictest601-21rngd23:50:091
834399347,23cyclictest11155-21sendmail-msp21:20:350
834399346,7cyclictest3364-21sendmail-msp22:00:350
834399341,11cyclictest6686-21sendmail-msp00:20:320
835699335,5cyclictest4812-21sendmail-msp19:40:482
835699333,10cyclictest22146irq/40-dwc2_hso20:15:132
835699331,9cyclictest23994-21/usr/sbin/munin00:10:242
835699331,7cyclictest16566-21sendmail_mailtr00:00:382
835699331,10cyclictest8140-21rm21:15:332
8356993310,20cyclictest24503-21sendmail-msp23:00:312
834399334,8cyclictest7313-21sendmail-msp21:40:250
834399331,7cyclictest1029-21ssh23:50:400
834399331,4cyclictest9970-21apt-get19:45:130
835699326,2cyclictest0-21swapper/221:10:202
835699321,8cyclictest900-21snmpd21:00:122
835699321,8cyclictest19527-21/usr/sbin/munin21:50:232
835699321,8cyclictest17976-21sendmail-msp19:20:212
835699321,7cyclictest32663-21fschecks_time22:20:222
835699321,7cyclictest23591-21ssh21:30:162
835699321,6cyclictest24468-21munin-plugin-st00:35:012
835699321,10cyclictest31630-21cut20:05:232
835699320,8cyclictest17005-21if_eth023:40:212
8349993210,16cyclictest12264-21sendmail-msp00:01:311
834399327,22cyclictest11006-21sendmail-msp20:20:540
834399325,7cyclictest16308-21sendmail-msp23:41:360
834399321,8cyclictest3381-21ssh23:09:130
834399321,4cyclictest0-21swapper/020:35:130
834399320,9cyclictest14154-21ssh22:52:070
836099311,23cyclictest0-21swapper/323:10:093
835699316,21cyclictest0-21swapper/200:35:362
835699315,22cyclictest8208-21awk22:50:102
835699313,11cyclictest30420-21sendmail-msp20:40:232
835699311,9cyclictest31843-21/usr/sbin/munin00:15:262
835699311,8cyclictest8476-21sshd22:25:312
835699311,8cyclictest4263-21ssh23:55:032
835699311,7cyclictest7921-21ntp_states23:10:222
835699311,6cyclictest5042-21ntp_kernel_pll_20:10:272
835699311,6cyclictest32766-21ssh23:30:132
835699311,6cyclictest32766-21ssh23:30:132
835699311,6cyclictest3071-21ssh21:35:302
835699311,6cyclictest16698-21unixbench_singl23:15:282
835699311,12cyclictest7148-21fschecks_count00:20:182
834999312,19cyclictest6686-21sendmail-msp00:20:151
834399318,19cyclictest24200-21df_inode22:15:170
834399315,6cyclictest1817-21sendmail_mailqu20:40:340
834399311,9cyclictest4812-21sendmail-msp19:40:210
836099307,15cyclictest25866irq/58-eth021:30:143
836099301,21cyclictest25866irq/58-eth021:40:163
835699307,19cyclictest0-21swapper/200:07:582
835699307,19cyclictest0-21swapper/200:07:582
835699305,6cyclictest11006-21sendmail-msp20:20:082
835699304,5cyclictest31722-21sh23:29:092
835699302,6cyclictest24078-21apt-get22:15:142
835699301,9cyclictest13795-21cut19:15:212
835699301,9cyclictest12254-21rm22:30:132
835699301,7cyclictest3953-21munin-run23:55:032
835699301,7cyclictest25420-21ssh23:45:542
835699301,7cyclictest20849-21ssh21:27:012
835699301,7cyclictest14420-21ssh00:25:212
835699301,6cyclictest16647-21kthreadcore19:50:242
835699301,25cyclictest24613-21sed20:00:182
834399309,19cyclictest601-21rngd20:05:190
834399309,17cyclictest32079-21sendmail-msp22:20:190
834399303,7cyclictest28369-21sendmail-msp22:40:160
834399301,9cyclictest16781-21grep22:55:150
834399301,8cyclictest3364-21sendmail-msp22:00:020
834399301,8cyclictest31806-21df21:35:150
834399301,7cyclictest7297-21tr23:10:180
834399301,7cyclictest17092-21latency_hist00:30:020
834399301,13cyclictest17312-21sendmail-msp21:00:250
834399300,9cyclictest25427-21ssh22:35:460
834399300,6cyclictest8411-21/usr/sbin/munin23:35:250
835699291,7cyclictest11845-21ps19:45:222
835699291,6cyclictest27059-21diskmemload22:47:002
835699291,13cyclictest0-21swapper/219:55:112
835699291,10cyclictest32653-21/usr/sbin/munin19:35:162
835699290,7cyclictest22914-21ssh22:57:422
834999291,22cyclictest17312-21sendmail-msp21:00:121
834399299,17cyclictest10112-21sshd21:18:310
834399292,24cyclictest24168-21sendmail-msp20:00:290
834399291,7cyclictest6324-21sh23:32:250
834399291,7cyclictest6324-21sh23:32:250
834399291,7cyclictest3375-21cat20:45:190
834399291,11cyclictest2739-21ssh21:14:390
834399291,10cyclictest15643-21apt-get22:10:080
835699283,5cyclictest21324-21irqcore20:30:222
835699281,9cyclictest28809-21cut20:35:232
835699281,6cyclictest24722-21cut19:25:212
835699281,5cyclictest27059-21diskmemload21:45:242
835699281,24cyclictest11880-21/usr/sbin/munin19:10:242
835699280,6cyclictest16535-21sed20:55:302
834399281,8cyclictest7979-21cat23:55:250
834399281,8cyclictest4570-21kthreadcore20:10:250
834399281,7cyclictest8126-21hwlatdetect20:50:210
834399281,7cyclictest24237-21fschecks_time00:10:130
834399281,7cyclictest21224-21tr19:20:270
834399281,6cyclictest8614-21ssh22:25:420
834399281,6cyclictest8405-21cat19:10:020
834399281,6cyclictest13359-21sed19:15:020
834399281,5cyclictest601-21rngd00:15:280
834399281,4cyclictest601-21rngd21:50:240
834399281,10cyclictest18961-21ssh23:17:520
836099273,17cyclictest0-21swapper/300:09:313
836099273,17cyclictest0-21swapper/300:09:313
836099273,16cyclictest0-21swapper/323:49:323
835699277,3cyclictest740-21sh23:05:172
835699271,9cyclictest16241-21/usr/sbin/munin20:25:162
835699271,8cyclictest15359-21ssh22:10:012
835699271,10cyclictest7568-21cat20:45:332
834999273,18cyclictest601-21rngd23:10:091
834999273,15cyclictest21-21ksoftirqd/123:00:171
834999272,19cyclictest16308-21sendmail-msp23:40:161
834399273,7cyclictest28480-21ntpq00:35:280
834399271,9cyclictest29413-21ssh23:25:440
834399271,8cyclictest6021-21ssh22:47:310
834399271,8cyclictest21031-21ssh21:27:170
834399271,8cyclictest18849-21ssh21:49:280
834399271,7cyclictest26405-21ssh23:47:230
834399271,7cyclictest24467-21apt00:35:010
834399271,7cyclictest14984-21fw_packets19:50:200
834399271,7cyclictest12899-21ssh22:06:340
834399271,7cyclictest12761-21ls20:55:140
834399271,6cyclictest27643-21basename19:30:010
834399271,11cyclictest32764-21diskstats19:35:160
836099262,19cyclictest0-21swapper/323:04:313
836099262,17cyclictest0-21swapper/322:09:313
836099260,15cyclictest25866irq/58-eth022:50:083
835699261,6cyclictest29710-21tr19:30:232
834399261,9cyclictest23506-21grep20:30:250
834399261,6cyclictest23729-21latency_hist00:10:010
834399261,6cyclictest23729-21latency_hist00:10:010
834399261,5cyclictest22813-21ps21:05:190
836099259,10cyclictest0-21swapper/322:44:313
836099259,10cyclictest0-21swapper/300:17:313
835699251,5cyclictest22977-21ps21:05:192
8349992415,6cyclictest11006-21sendmail-msp20:20:201
834999241,18cyclictest20532-21sendmail-msp23:20:291
836099236,12cyclictest0-21swapper/300:00:163
836099233,13cyclictest0-21swapper/323:43:313
836099233,13cyclictest0-21swapper/321:47:313
834999232,15cyclictest601-21rngd23:49:311
8349992314,6cyclictest12228-21cron00:00:011
836099221,10cyclictest0-21swapper/321:53:313
834999226,5cyclictest21-21ksoftirqd/123:07:531
834999226,10cyclictest601-21rngd19:50:171
834999223,4cyclictest601-21rngd23:18:301
834999223,4cyclictest21-21ksoftirqd/122:23:481
834999222,3cyclictest21-21ksoftirqd/122:49:241
836099219,6cyclictest0-21swapper/323:05:593
8360992115,4cyclictest0-21swapper/321:20:233
8360992114,5cyclictest0-21swapper/323:20:203
834999213,4cyclictest601-21rngd23:33:421
834999213,4cyclictest601-21rngd23:33:411
834999212,4cyclictest21-21ksoftirqd/120:40:231
834999212,13cyclictest601-21rngd21:47:311
8349992112,6cyclictest9673-21basename19:45:011
834999211,14cyclictest28369-21sendmail-msp22:40:141
834999211,14cyclictest27956-21ls21:55:131
8349992110,7cyclictest1621-21apt-get20:10:161
836099203,11cyclictest0-21swapper/323:15:313
836099202,11cyclictest0-21swapper/321:19:313
8360992011,6cyclictest25866irq/58-eth022:10:093
8360992010,6cyclictest0-21swapper/322:23:183
834999206,9cyclictest601-21rngd21:10:171
834999206,4cyclictest0-21swapper/121:17:461
834999203,4cyclictest601-21rngd22:54:131
834999203,3cyclictest21-21ksoftirqd/122:30:281
834999202,3cyclictest21-21ksoftirqd/121:55:011
8349992012,6cyclictest24465-21cron00:35:021
836099199,7cyclictest0-21swapper/323:26:233
836099199,7cyclictest0-21swapper/321:25:563
836099193,11cyclictest0-21swapper/323:38:313
836099193,11cyclictest0-21swapper/322:46:313
836099193,10cyclictest0-21swapper/319:30:273
836099193,10cyclictest0-21swapper/319:12:283
836099192,11cyclictest25866irq/58-eth019:45:143
836099192,11cyclictest0-21swapper/319:35:273
8360991911,2cyclictest0-21swapper/319:15:153
8360991910,6cyclictest0-21swapper/323:31:093
8360991910,6cyclictest0-21swapper/323:31:093
8360991910,6cyclictest0-21swapper/322:38:583
8360991910,6cyclictest0-21swapper/321:57:263
8360991910,6cyclictest0-21swapper/321:13:103
8360991910,4cyclictest0-21swapper/322:59:183
834999197,8cyclictest601-21rngd21:30:121
834999194,4cyclictest21-21ksoftirqd/100:28:581
834999194,4cyclictest131rcu_preempt00:06:511
834999194,4cyclictest131rcu_preempt00:06:511
834999194,3cyclictest21-21ksoftirqd/121:28:231
834999194,2cyclictest131rcu_preempt22:15:391
834999193,4cyclictest601-21rngd22:26:501
834999193,4cyclictest21-21ksoftirqd/123:36:211
834999193,4cyclictest21-21ksoftirqd/100:35:381
834999193,3cyclictest601-21rngd00:17:391
834999193,3cyclictest21-21ksoftirqd/122:35:471
834999193,12cyclictest601-21rngd20:30:171
834999192,4cyclictest24152-21latency_hist20:00:011
834999190,4cyclictest601-21rngd22:05:091
834999190,4cyclictest31523-21munin-run00:15:011
836099185,10cyclictest0-21swapper/322:15:143
836099183,10cyclictest0-21swapper/320:41:273
836099182,9cyclictest0-21swapper/320:45:163
836099182,9cyclictest0-21swapper/320:05:153
836099182,10cyclictest0-21swapper/321:35:143
836099182,10cyclictest0-21swapper/321:01:273
836099182,10cyclictest0-21swapper/320:55:143
836099182,10cyclictest0-21swapper/320:30:053
836099182,10cyclictest0-21swapper/319:21:283
836099182,10cyclictest0-21swapper/300:10:103
8360991813,3cyclictest321rcuc/323:53:373
8360991811,4cyclictest0-21swapper/322:01:033
8360991810,5cyclictest25866irq/58-eth000:35:163
834999183,6cyclictest131rcu_preempt19:15:181
834999182,4cyclictest21-21ksoftirqd/123:25:061
834999182,4cyclictest21-21ksoftirqd/122:56:121
834999182,10cyclictest0-21swapper/120:05:151
834999181,4cyclictest21-21ksoftirqd/121:35:291
834999180,3cyclictest601-21rngd22:10:231
834999180,15cyclictest16188-21ls20:25:151
836099176,8cyclictest0-21swapper/319:55:143
836099172,9cyclictest0-21swapper/320:20:153
8360991712,3cyclictest321rcuc/322:25:343
8360991711,4cyclictest0-21swapper/300:20:233
834999173,2cyclictest131rcu_preempt19:25:291
834999171,11cyclictest845-21munin-node19:30:161
8349991710,3cyclictest601-21rngd19:10:141
834999170,4cyclictest601-21rngd20:55:191
834999170,3cyclictest601-21rngd21:05:271
836099167,4cyclictest0-21swapper/300:27:573
836099161,9cyclictest0-21swapper/320:30:013
836099161,12cyclictest0-21swapper/320:35:303
836099161,10cyclictest0-21swapper/319:50:183
836099160,10cyclictest0-21swapper/319:25:143
835699160,13cyclictest8462-21cat19:10:022
834999168,6cyclictest9381-21cut20:15:231
834999164,6cyclictest131rcu_preempt19:35:171
834999164,2cyclictest131rcu_preempt20:50:191
834999161,8cyclictest30360-21users20:35:321
834999161,10cyclictest601-21rngd20:45:301
836099152,8cyclictest0-21swapper/320:00:133
8360991511,3cyclictest321rcuc/320:10:273
834999151,3cyclictest21-21ksoftirqd/119:45:161
836099149,3cyclictest0-21swapper/320:50:213
836099141,7cyclictest0-21swapper/319:40:183
8360991410,2cyclictest321rcuc/321:05:193
836099138,3cyclictest0-21swapper/320:16:353
836099111,7cyclictest0-21swapper/319:10:023
834999101,6cyclictest601-21rngd19:10:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional