You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-16 - 17:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 16, 2025 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1268799643,40cyclictest18327-21ssh09:55:142
1268799562,42cyclictest31230-21ssh10:55:112
1267299527,38cyclictest12906-21sendmail-msp10:40:110
1268099512,40cyclictest19149-21sendmail-msp10:20:141
1267299502,38cyclictest22558-21sendmail-msp11:40:200
1268799482,28cyclictest8251-21ssh11:55:142
1267299472,35cyclictest10631-21ssh09:50:090
1269599454,24cyclictest25866irq/58-eth009:50:083
12687994327,10cyclictest1941-21sendmail-msp08:40:142
12687994220,13cyclictest30208-21sendmail-msp11:20:132
1268799411,30cyclictest21141-21sendmail-msp09:00:142
1268099411,32cyclictest15184-21sendmail-msp08:20:131
1267299413,33cyclictest22164-21sendmail-msp07:20:160
1268799382,27cyclictest21607-21kthreadcore07:50:262
1268799371,27cyclictest7941-21ssh11:30:092
1267299374,30cyclictest752-21ssh11:50:090
1267299373,23cyclictest32091-21sendmail-msp09:40:140
12672993718,17cyclictest25377-21sendmail-msp10:01:250
1268099357,25cyclictest15505-21sendmail-msp12:00:141
1268099357,25cyclictest15505-21sendmail-msp12:00:141
1268099355,26cyclictest11526-21ssh12:20:151
12672993516,12cyclictest19149-21sendmail-msp10:20:160
1269599343,14cyclictest0-21swapper/307:50:263
1269599341,21cyclictest25866irq/58-eth009:30:103
1268799340,22cyclictest261rcuc/212:15:122
1268099343,28cyclictest5569-21sh10:35:161
1268099343,28cyclictest5569-21sh10:35:151
12687993316,9cyclictest5905-21sendmail-msp11:00:152
12687993312,14cyclictest27-21ksoftirqd/208:15:182
12687993312,14cyclictest27-21ksoftirqd/208:15:182
1268099334,26cyclictest28298-21sendmail-msp08:00:161
1268099332,27cyclictest25377-21sendmail-msp10:00:141
1267299332,23cyclictest7419-21ssh11:29:250
1268799325,22cyclictest27-21ksoftirqd/210:49:252
1268799323,18cyclictest22146irq/40-dwc2_hso07:30:172
1268799322,7cyclictest15817-21ssh12:00:142
1268799322,7cyclictest15817-21ssh12:00:142
1268099322,22cyclictest21141-21sendmail-msp09:01:251
12680993215,12cyclictest0-21swapper/108:05:151
1267299323,24cyclictest9103-21sendmail-msp07:40:130
1267299322,22cyclictest21608-21kthreadcore07:50:260
1268799312,23cyclictest32091-21sendmail-msp09:40:122
1268799312,22cyclictest0-21swapper/207:10:252
1267299313,18cyclictest12-21ksoftirqd/010:35:160
1267299313,18cyclictest12-21ksoftirqd/010:35:160
1269599302,15cyclictest25866irq/58-eth011:50:103
1268799307,18cyclictest27-21ksoftirqd/210:29:242
1268799302,21cyclictest9103-21sendmail-msp07:40:252
1268799302,18cyclictest22146irq/40-dwc2_hso11:10:092
1268799302,11cyclictest17554-21ntp_192.168.11407:45:252
1267299304,24cyclictest30307-21ssh10:30:100
1269599294,12cyclictest25866irq/58-eth011:55:143
1269599291,14cyclictest25866irq/58-eth009:15:153
1268799292,20cyclictest0-21swapper/207:35:052
1268799292,17cyclictest22146irq/40-dwc2_hso10:15:182
1267299292,19cyclictest8511-21kthreadcore08:10:240
1268799285,7cyclictest0-21swapper/212:30:082
1268799282,17cyclictest15184-21sendmail-msp08:20:142
1268099284,15cyclictest22558-21sendmail-msp11:40:211
1268099282,4cyclictest21-21ksoftirqd/109:30:261
1267299282,19cyclictest1314-21kthreadcore12:35:240
1269599273,15cyclictest0-21swapper/308:10:253
1268799272,20cyclictest601-21rngd08:01:252
1268799272,19cyclictest24926-21ssh09:35:122
1268799271,11cyclictest6152-21ssh12:38:252
1268099274,7cyclictest21-21ksoftirqd/111:00:011
1267299275,16cyclictest12-21ksoftirqd/010:50:100
1267299272,19cyclictest2638-21grep07:30:240
12672992715,4cyclictest12-21ksoftirqd/011:00:150
12672992710,13cyclictest7867-21sendmail-msp09:20:160
1268799268,9cyclictest12906-21sendmail-msp10:40:312
1268799261,14cyclictest22146irq/40-dwc2_hso08:45:122
1268099262,4cyclictest601-21rngd10:30:091
1268099262,19cyclictest2316-21kthreadcore12:35:241
1267299261,21cyclictest17738-21ssh09:30:100
12695992510,10cyclictest0-21swapper/310:20:163
1268799258,4cyclictest131rcu_preempt09:30:262
1268099252,19cyclictest73-21kswapd008:50:181
1268099252,16cyclictest26551-21ssh09:10:111
1269599243,14cyclictest0-21swapper/312:20:153
1269599242,14cyclictest0-21swapper/311:25:243
1268799249,7cyclictest27-21ksoftirqd/208:25:142
1268799249,3cyclictest131rcu_preempt10:30:102
1268799244,17cyclictest131rcu_preempt08:05:182
1268799242,16cyclictest0-21swapper/207:15:262
1268799241,10cyclictest1092666ssh09:20:262
1268099245,10cyclictest21-21ksoftirqd/110:15:021
1268099243,4cyclictest21-21ksoftirqd/108:00:011
1268099242,19cyclictest22164-21sendmail-msp07:21:251
1268099242,16cyclictest73-21kswapd008:30:181
1267299245,3cyclictest12-21ksoftirqd/009:35:110
1267299242,8cyclictest601-21rngd08:02:250
1267299242,7cyclictest131rcu_preempt08:41:240
1267299241,16cyclictest15831-21ssh12:00:150
1267299241,16cyclictest15831-21ssh12:00:140
1268799232,6cyclictest0-21swapper/210:23:252
1268799232,6cyclictest0-21swapper/208:14:252
1268799232,16cyclictest0-21swapper/209:47:262
1268799232,10cyclictest0-21swapper/207:55:132
1268799230,12cyclictest131rcu_preempt09:19:192
1268099233,10cyclictest21-21ksoftirqd/112:19:371
1268099232,17cyclictest32626-21grep09:40:181
1268099231,4cyclictest601-21rngd11:55:151
12680992314,6cyclictest12906-21sendmail-msp10:40:161
1268099230,5cyclictest21-21ksoftirqd/111:55:011
1267299238,4cyclictest131rcu_preempt12:20:350
1267299232,18cyclictest10029-21ssh09:49:250
1269599223,13cyclictest0-21swapper/312:35:243
12695992211,7cyclictest0-21swapper/311:00:173
1268799225,9cyclictest30116-21cron11:45:012
1268799225,14cyclictest25377-21sendmail-msp10:00:342
1268799222,6cyclictest0-21swapper/210:14:252
1268799222,16cyclictest0-21swapper/207:28:242
1268799222,15cyclictest0-21swapper/211:35:042
1268799221,17cyclictest976066ssh10:35:342
1268799221,17cyclictest976066ssh10:35:332
1268099228,3cyclictest21-21ksoftirqd/111:28:061
1268099226,8cyclictest21-21ksoftirqd/109:15:061
1268099225,7cyclictest21-21ksoftirqd/109:52:421
1268099221,4cyclictest601-21rngd11:50:021
1268099221,4cyclictest21-21ksoftirqd/110:11:271
1268099221,16cyclictest5905-21sendmail-msp11:00:181
1268099221,13cyclictest30208-21sendmail-msp11:20:201
1267299222,15cyclictest30208-21sendmail-msp11:21:240
1267299221,13cyclictest4246-21ssh10:10:100
1267299221,11cyclictest632-21Async17:44:200
1269599213,11cyclictest0-21swapper/308:20:253
12695992112,6cyclictest0-21swapper/311:30:093
1268799213,5cyclictest0-21swapper/211:51:092
1268799213,12cyclictest845-21munin-node08:50:172
1268799212,14cyclictest23195-21apt-get11:15:132
1268799211,7cyclictest167-21systemd-network09:13:382
1268799211,2cyclictest131rcu_preempt10:08:552
1268799211,16cyclictest21122-21munin-run09:00:012
1268799211,11cyclictest15126-21memory12:20:292
1268799210,8cyclictest23212-21kthreadcore07:20:232
1268099219,3cyclictest0-21swapper/111:11:571
1268099216,8cyclictest21-21ksoftirqd/111:17:171
1268099216,3cyclictest21-21ksoftirqd/111:39:241
1268099215,6cyclictest21-21ksoftirqd/110:05:561
1268099215,2cyclictest131rcu_preempt10:53:171
1268099214,7cyclictest21-21ksoftirqd/112:27:541
1268099212,6cyclictest21-21ksoftirqd/107:10:131
1268099212,5cyclictest21-21ksoftirqd/108:55:151
1268099212,4cyclictest21-21ksoftirqd/112:30:271
1268099212,4cyclictest21-21ksoftirqd/108:35:281
1268099212,4cyclictest21-21ksoftirqd/108:15:251
1268099212,4cyclictest21-21ksoftirqd/108:15:251
1268099212,4cyclictest21-21ksoftirqd/107:25:271
1268099212,13cyclictest601-21rngd08:10:251
1268099211,16cyclictest73-21kswapd009:35:121
12680992110,8cyclictest1941-21sendmail-msp08:40:321
12680992110,3cyclictest0-21swapper/112:09:531
1267299212,14cyclictest11693-21memory11:55:240
1267299212,11cyclictest0-21swapper/011:35:110
1269599209,6cyclictest0-21swapper/312:17:493
1269599203,11cyclictest0-21swapper/308:16:253
1269599203,11cyclictest0-21swapper/308:16:253
1269599203,11cyclictest0-21swapper/308:08:253
1269599203,11cyclictest0-21swapper/307:10:253
1269599202,10cyclictest0-21swapper/310:35:163
1269599202,10cyclictest0-21swapper/310:35:163
1268799202,12cyclictest23811-21sh10:50:092
1268799201,6cyclictest131rcu_preempt11:47:362
12687992014,4cyclictest0-21swapper/208:30:172
12687992010,7cyclictest27-21ksoftirqd/212:26:412
12687992010,3cyclictest10624-21apt-get09:50:082
1268099206,8cyclictest21-21ksoftirqd/112:11:141
1268099203,4cyclictest21-21ksoftirqd/111:30:261
1268099202,4cyclictest52246usb-storage09:20:281
1268099202,11cyclictest601-21rngd07:30:251
12680992011,4cyclictest21-21ksoftirqd/110:28:411
1267299209,6cyclictest601-21rngd08:20:170
1269599199,7cyclictest0-21swapper/310:46:213
1269599194,12cyclictest0-21swapper/309:55:133
1269599193,11cyclictest0-21swapper/307:41:253
1269599193,10cyclictest0-21swapper/307:21:253
1269599192,11cyclictest0-21swapper/309:45:243
1269599192,11cyclictest0-21swapper/308:00:043
12695991911,6cyclictest0-21swapper/310:00:283
1269599191,14cyclictest0-21swapper/307:15:153
1269599191,12cyclictest0-21swapper/307:25:163
12695991910,7cyclictest25866irq/58-eth010:10:103
12695991910,6cyclictest0-21swapper/312:31:123
12695991910,6cyclictest0-21swapper/312:13:583
12695991910,6cyclictest0-21swapper/311:13:323
12695991910,6cyclictest0-21swapper/310:43:313
1268799197,9cyclictest29796-21kthreadcore12:10:182
1268799196,5cyclictest27-21ksoftirqd/211:25:162
1268799191,9cyclictest29915-21uniq08:35:182
12687991910,7cyclictest0-21swapper/209:28:032
1268099197,4cyclictest21-21ksoftirqd/111:05:221
1268099193,4cyclictest21-21ksoftirqd/107:35:311
1268099193,11cyclictest601-21rngd09:55:141
1268099192,4cyclictest21-21ksoftirqd/107:50:181
1268099192,3cyclictest131rcu_preempt09:48:251
1268099191,4cyclictest21-21ksoftirqd/108:25:361
1268099191,3cyclictest601-21rngd10:46:171
12680991910,5cyclictest601-21rngd07:40:241
1268099190,4cyclictest21-21ksoftirqd/109:29:101
1267299195,4cyclictest131rcu_preempt09:20:010
1267299191,9cyclictest11780-21/usr/sbin/munin08:50:320
12672991911,6cyclictest19111-21cron10:20:010
1269599189,6cyclictest0-21swapper/311:20:023
1269599189,6cyclictest0-21swapper/309:36:133
1269599183,8cyclictest0-21swapper/309:10:093
1269599183,11cyclictest0-21swapper/311:21:233
1269599183,11cyclictest0-21swapper/310:50:093
1269599183,11cyclictest0-21swapper/310:19:263
1269599183,10cyclictest0-21swapper/311:41:243
1269599183,10cyclictest0-21swapper/309:40:253
1269599183,10cyclictest0-21swapper/309:23:253
1269599182,11cyclictest0-21swapper/311:46:253
1269599182,10cyclictest0-21swapper/312:00:163
1269599182,10cyclictest0-21swapper/312:00:163
1269599182,10cyclictest0-21swapper/307:35:043
1269599182,10cyclictest0-21swapper/307:30:183
12695991812,4cyclictest25866irq/58-eth010:30:103
12695991812,4cyclictest0-21swapper/311:35:033
1269599181,13cyclictest0-21swapper/308:30:273
1269599181,12cyclictest0-21swapper/307:45:243
12695991810,6cyclictest0-21swapper/310:27:343
12695991810,4cyclictest0-21swapper/308:40:163
12695991810,3cyclictest0-21swapper/310:58:333
1268799187,8cyclictest26241-21diskmemload11:08:182
1268799181,8cyclictest26276-21cat09:10:022
1268099189,6cyclictest17555-21sshd07:45:251
1268099183,4cyclictest21-21ksoftirqd/107:15:161
1267299185,10cyclictest27178-21grep11:15:310
12672991811,4cyclictest22141-21swap09:55:300
1267299181,14cyclictest21141-21sendmail-msp09:00:200
1267299180,9cyclictest601-21rngd11:10:430
1269599179,6cyclictest0-21swapper/311:05:563
1269599179,6cyclictest0-21swapper/310:05:463
1269599179,6cyclictest0-21swapper/309:25:113
1269599177,8cyclictest0-21swapper/312:25:253
1269599173,10cyclictest0-21swapper/308:50:183
1269599172,10cyclictest0-21swapper/308:55:143
1269599171,12cyclictest0-21swapper/308:35:253
12695991710,4cyclictest0-21swapper/312:06:313
1268799178,7cyclictest26092-21sh12:08:412
1268799171,9cyclictest12734-21sort07:10:022
1268099171,4cyclictest21-21ksoftirqd/108:45:131
1268099171,4cyclictest21-21ksoftirqd/107:10:021
1267299179,6cyclictest29563-21sshd08:33:250
1267299178,7cyclictest32548-21kthreadcore10:55:180
1267299174,3cyclictest131rcu_preempt12:10:160
1267299173,3cyclictest131rcu_preempt08:25:320
1267299172,7cyclictest5583-21kworker/0:108:10:010
1267299171,9cyclictest18478-21kthreadcore08:55:210
12672991711,4cyclictest12-21ksoftirqd/012:20:010
1269599162,10cyclictest0-21swapper/308:25:153
1269599161,8cyclictest0-21swapper/309:05:023
1269599161,10cyclictest0-21swapper/309:10:013
1267299168,6cyclictest27417-21df_inode07:25:150
1267299165,3cyclictest131rcu_preempt11:10:010
1267299162,9cyclictest15184-21sendmail-msp08:20:020
1267299162,9cyclictest15184-21sendmail-msp08:20:020
1267299161,7cyclictest25347-21cut07:55:200
1267299161,13cyclictest346-21dpkg11:50:020
1267299161,12cyclictest23619-21diskstats12:30:110
1267299161,12cyclictest12914-21diskstats07:10:150
1267299161,12cyclictest11812-21cat11:30:270
1267299161,11cyclictest28466-21ssh10:27:250
1267299160,4cyclictest601-21rngd10:50:010
1268099155,8cyclictest21-21ksoftirqd/109:05:281
1267299159,4cyclictest23262-21ssh12:30:020
1267299154,6cyclictest601-21rngd09:10:090
1267299152,9cyclictest0-21swapper/007:15:150
1267299151,11cyclictest7016-21apt-get08:45:130
1267299151,11cyclictest4219-21apt-get07:35:030
1267299151,11cyclictest26391-21latency_hist09:10:020
1267299151,11cyclictest17446-21munin-plugin-st09:30:010
1267299150,9cyclictest1364-21ssh10:05:370
1267299150,6cyclictest30308-21ps08:35:210
1269599148,4cyclictest0-21swapper/308:47:303
1267299142,9cyclictest12712-21latency_hist07:10:010
1267299141,4cyclictest601-21rngd12:05:100
1269599131,8cyclictest0-21swapper/307:55:143
1269599101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional