You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 04:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 02, 2025 00:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
891999601,36cyclictest8061-21ssh22:55:122
891799505,31cyclictest21472-21sendmail-msp20:40:150
891999494,30cyclictest27-21ksoftirqd/223:50:092
891899492,36cyclictest8718-21sendmail-msp23:40:111
891799481,38cyclictest26505-21ssh21:15:120
891799461,35cyclictest12155-21sendmail-msp23:20:160
8917994425,12cyclictest1986-21sendmail-msp21:20:160
8919994216,18cyclictest27-21ksoftirqd/221:50:092
891899423,33cyclictest24372-21ssh00:15:141
8918994225,12cyclictest21-21ksoftirqd/100:00:161
891899422,36cyclictest30936-21sendmail-msp21:40:141
892099415,20cyclictest25866irq/58-eth023:30:083
8919994131,5cyclictest22146irq/40-dwc2_hso23:40:122
891899413,35cyclictest5294-21sendmail-msp19:40:151
8918994018,10cyclictest21-21ksoftirqd/122:35:121
891899392,34cyclictest16014-21sendmail-msp23:00:121
891899392,31cyclictest6821-21sendmail-msp20:20:141
8918993816,15cyclictest21-21ksoftirqd/122:55:131
8918993815,17cyclictest21472-21sendmail-msp20:40:251
891799382,29cyclictest24525-21sendmail-msp20:00:150
891999372,26cyclictest0-21swapper/219:10:242
8919993711,10cyclictest11021-21ssh23:18:252
8919993711,10cyclictest11021-21ssh23:18:242
891999362,18cyclictest24387-21ssh00:15:172
891999361,27cyclictest22146irq/40-dwc2_hso22:10:082
8919993612,17cyclictest0-21swapper/219:30:242
891999353,25cyclictest3367-21ssh22:30:102
891999352,18cyclictest2890-21ssh23:35:252
891899351,26cyclictest27187-21sendmail-msp22:00:121
891999342,25cyclictest23252-21kthreadcore20:40:252
891999342,25cyclictest18247-21ssh21:10:172
8918993415,13cyclictest601-21rngd22:40:151
892099330,20cyclictest25866irq/58-eth021:35:143
891899332,23cyclictest16269-21ssh00:31:241
891999326,16cyclictest22936-21ssh21:35:132
891999325,16cyclictest22146irq/40-dwc2_hso20:35:122
891999322,24cyclictest0-21swapper/219:38:242
8919993216,9cyclictest27-21ksoftirqd/220:15:152
891899312,22cyclictest12464-21sed19:10:251
891899311,23cyclictest8205-21sendmail-msp21:00:131
891999306,18cyclictest8916-21cyclictest19:24:232
891999304,16cyclictest19219-21ssh21:55:152
891999303,17cyclictest22146irq/40-dwc2_hso20:50:182
8919993020,6cyclictest32135-21ssh22:50:112
891899302,20cyclictest29262-21ssh22:21:241
892099293,17cyclictest25866irq/58-eth021:10:153
892099291,19cyclictest0-21swapper/322:15:123
891999291,21cyclictest1986-21sendmail-msp21:20:152
8919992910,8cyclictest27-21ksoftirqd/221:20:012
891899295,21cyclictest201rcuc/123:10:091
891799295,2cyclictest131rcu_preempt22:30:260
891799290,23cyclictest23506-21sendmail-msp22:20:130
891999284,15cyclictest22146irq/40-dwc2_hso19:50:202
891999283,13cyclictest22146irq/40-dwc2_hso00:35:142
891999282,20cyclictest31846-21sendmail-msp00:20:142
8919992820,6cyclictest27-21ksoftirqd/220:10:122
8919992820,6cyclictest27-21ksoftirqd/220:10:112
891899283,15cyclictest21-21ksoftirqd/100:20:151
892099273,17cyclictest25866irq/58-eth023:50:093
891999273,14cyclictest22146irq/40-dwc2_hso23:55:142
891999273,10cyclictest4387-21kworker/u8:322:15:132
891999272,14cyclictest27187-21sendmail-msp22:00:132
891999271,10cyclictest20579-21grep23:00:382
891899272,7cyclictest52246usb-storage00:40:011
8917992714,9cyclictest601-21rngd21:40:150
8917992714,6cyclictest601-21rngd19:30:160
891999268,3cyclictest167-21systemd-network23:20:342
891999264,13cyclictest22146irq/40-dwc2_hso20:30:092
891999263,17cyclictest0-21swapper/222:45:242
891999262,9cyclictest0-21swapper/200:30:162
8919992612,7cyclictest1901-21ssh22:28:252
891899262,18cyclictest0-21swapper/119:45:231
891799262,16cyclictest601-21rngd23:54:240
892099253,13cyclictest25866irq/58-eth019:50:183
892099253,13cyclictest0-21swapper/300:18:243
891999252,8cyclictest27579-21ssh22:44:242
891999251,9cyclictest8819-21sed21:00:212
8919992512,10cyclictest17914-21ssh22:37:252
891899253,19cyclictest18495-21sendmail-msp19:20:141
891799252,16cyclictest601-21rngd00:31:240
891799251,17cyclictest0-21swapper/023:30:090
891999241,8cyclictest26465-21kthreadcore19:25:232
891899241,18cyclictest13904-21find19:15:121
891899241,17cyclictest14625-21ssh21:30:151
891799249,12cyclictest16014-21sendmail-msp23:01:170
8917992412,9cyclictest23474-21cron22:20:020
892099233,13cyclictest0-21swapper/323:18:253
892099233,13cyclictest0-21swapper/323:18:243
891999236,12cyclictest27-21ksoftirqd/223:30:092
891999233,12cyclictest22146irq/40-dwc2_hso00:00:152
891999232,6cyclictest0-21swapper/220:55:132
891999232,6cyclictest0-21swapper/220:45:242
891999231,6cyclictest24104-21ls00:15:012
8919992311,9cyclictest0-21swapper/219:41:342
891999231,12cyclictest1280-21kthreadcore21:40:242
891999230,8cyclictest18498-21ls21:30:322
891899231,17cyclictest73-21kswapd020:30:101
891799238,7cyclictest12-21ksoftirqd/022:10:510
891799238,7cyclictest12-21ksoftirqd/000:10:250
891799235,4cyclictest12-21ksoftirqd/021:59:420
891799233,12cyclictest131rcu_preempt00:28:030
892099223,13cyclictest0-21swapper/319:10:253
892099222,13cyclictest25866irq/58-eth022:00:143
891999224,8cyclictest27-21ksoftirqd/219:55:142
891999222,16cyclictest27-21ksoftirqd/223:29:242
891999222,15cyclictest0-21swapper/223:46:242
891999222,15cyclictest0-21swapper/222:09:252
891999222,15cyclictest0-21swapper/221:29:232
891999222,14cyclictest0-21swapper/200:25:242
891999222,10cyclictest24501-21irqrtprio22:20:212
891999221,8cyclictest9689-21kthreadcore20:20:252
891899222,6cyclictest52246usb-storage20:01:431
891899222,14cyclictest30486-21ssh21:39:251
891899221,18cyclictest15717-21ls22:15:161
891899221,18cyclictest11061-21ls22:10:271
891899221,17cyclictest3331-21find22:30:091
891799229,4cyclictest12-21ksoftirqd/000:01:550
891799228,4cyclictest12-21ksoftirqd/023:07:450
891799225,3cyclictest131rcu_preempt21:35:310
891799223,7cyclictest131rcu_preempt23:47:030
891799222,16cyclictest4162-21ssh23:15:150
891799222,16cyclictest4162-21ssh23:15:140
8917992210,9cyclictest27187-21sendmail-msp22:00:240
892099212,10cyclictest0-21swapper/322:55:053
891999219,10cyclictest9538-21ssh21:48:332
891999211,16cyclictest24496-21apt-get20:00:052
891999211,16cyclictest15997-21cron00:10:012
891999211,10cyclictest12313-21kthreadcore19:45:212
891899218,6cyclictest8916-21cyclictest20:45:131
891899217,9cyclictest13253-21irqstats23:20:251
891899212,15cyclictest2246-21apt-get20:15:131
891899212,13cyclictest601-21rngd23:18:241
891899212,13cyclictest601-21rngd23:18:241
8918992111,7cyclictest1986-21sendmail-msp21:20:361
891899210,18cyclictest22083-21ls21:10:321
891799219,5cyclictest12-21ksoftirqd/023:40:060
891799218,7cyclictest12-21ksoftirqd/022:27:530
891799216,6cyclictest131rcu_preempt21:47:430
891799214,3cyclictest131rcu_preempt00:35:050
891799213,5cyclictest12-21ksoftirqd/019:20:260
891799212,4cyclictest131rcu_preempt00:20:180
8917992116,3cyclictest12-21ksoftirqd/022:50:260
892099203,12cyclictest0-21swapper/321:15:143
8920992011,6cyclictest0-21swapper/323:49:253
8920992011,3cyclictest0-21swapper/322:40:323
891999203,14cyclictest13921-21irqstats21:05:192
891999201,9cyclictest28848-21sed23:10:162
891999200,15cyclictest14100-21fschecks_time19:15:172
891899202,12cyclictest19980-21ntp_states00:10:251
891799207,7cyclictest12-21ksoftirqd/021:26:070
891799204,2cyclictest131rcu_preempt21:52:480
891799204,2cyclictest131rcu_preempt00:16:330
891799202,9cyclictest12-21ksoftirqd/022:09:490
891799202,3cyclictest131rcu_preempt22:40:210
891799202,2cyclictest131rcu_preempt23:12:380
891799202,12cyclictest7506-21ssh23:38:240
8917992011,6cyclictest0-21swapper/022:55:130
891799201,11cyclictest12-21ksoftirqd/022:46:020
891799200,3cyclictest601-21rngd23:57:150
891799200,2cyclictest131rcu_preempt21:34:070
891799200,2cyclictest131rcu_preempt00:08:020
892099197,8cyclictest25866irq/58-eth023:55:143
892099193,11cyclictest0-21swapper/319:30:253
892099193,11cyclictest0-21swapper/319:24:243
892099193,10cyclictest0-21swapper/323:20:183
892099193,10cyclictest0-21swapper/320:40:243
892099192,11cyclictest0-21swapper/319:36:253
892099192,11cyclictest0-21swapper/319:25:233
892099192,11cyclictest0-21swapper/300:29:253
8920991914,3cyclictest25966irq/59-eth000:30:143
8920991911,6cyclictest0-21swapper/323:27:423
8920991911,6cyclictest0-21swapper/323:10:153
8920991911,3cyclictest0-21swapper/321:41:113
8920991910,6cyclictest0-21swapper/322:55:003
8920991910,6cyclictest0-21swapper/321:27:343
891999191,10cyclictest29669-21cron20:10:002
891899199,7cyclictest24534-21gpgv20:00:021
891899196,9cyclictest23000-21sendmail_mailqu21:55:311
891899192,5cyclictest52246usb-storage21:25:061
8918991913,4cyclictest30009-21irqstats23:55:191
891799192,3cyclictest131rcu_preempt19:15:150
891799191,6cyclictest52246usb-storage21:00:210
891799191,5cyclictest52246usb-storage20:20:330
891799191,5cyclictest12-21ksoftirqd/019:10:180
891799190,2cyclictest131rcu_preempt22:35:230
892099189,6cyclictest0-21swapper/323:35:413
892099189,6cyclictest0-21swapper/300:20:323
892099189,4cyclictest0-21swapper/321:33:133
892099188,6cyclictest25866irq/58-eth022:10:103
892099183,10cyclictest0-21swapper/323:44:243
892099182,10cyclictest0-21swapper/322:09:243
892099182,10cyclictest0-21swapper/320:55:153
8920991811,5cyclictest25966irq/59-eth021:50:093
892099181,12cyclictest0-21swapper/323:00:363
892099181,12cyclictest0-21swapper/320:45:303
8920991810,6cyclictest0-21swapper/323:08:423
8920991810,6cyclictest0-21swapper/300:39:453
8920991810,6cyclictest0-21swapper/300:06:493
8920991810,3cyclictest0-21swapper/321:20:163
8920991810,3cyclictest0-21swapper/300:11:113
891999181,5cyclictest8925-21latency_hist19:10:022
891999181,3cyclictest0-21swapper/223:05:152
891899182,9cyclictest601-21rngd23:35:151
891899181,14cyclictest26494-21apt-get21:15:131
891899181,12cyclictest11920-21munin-run20:30:001
891799188,7cyclictest8365-21sed21:00:020
891799188,7cyclictest6799-21cron20:20:010
891799182,3cyclictest131rcu_preempt21:05:250
891799182,2cyclictest131rcu_preempt19:25:140
891799181,3cyclictest131rcu_preempt21:14:350
891799181,3cyclictest131rcu_preempt19:41:370
8917991812,4cyclictest0-21swapper/023:26:500
891799180,4cyclictest601-21rngd20:50:340
892099179,6cyclictest0-21swapper/322:31:263
892099179,6cyclictest0-21swapper/322:21:273
892099179,6cyclictest0-21swapper/321:49:003
892099179,3cyclictest0-21swapper/322:48:223
892099173,10cyclictest0-21swapper/322:29:243
892099172,11cyclictest0-21swapper/321:05:153
892099172,10cyclictest0-21swapper/320:20:153
892099171,9cyclictest0-21swapper/320:10:013
892099171,12cyclictest0-21swapper/319:45:283
892099171,10cyclictest0-21swapper/319:16:243
891899179,5cyclictest0-21swapper/120:55:291
891899178,7cyclictest26981-21ntpq19:25:251
891899172,11cyclictest601-21rngd23:29:231
891899172,11cyclictest601-21rngd19:54:241
891899172,11cyclictest24486-21ssh23:30:081
8918991712,3cyclictest0-21swapper/122:09:141
891899171,14cyclictest10928-21cpuspeed21:50:091
891899171,11cyclictest32269-21fschecks_count22:50:131
891899171,10cyclictest601-21rngd20:35:281
8918991710,5cyclictest16839-21run-parts21:05:291
891799179,5cyclictest601-21rngd19:40:020
891799175,3cyclictest131rcu_preempt19:45:230
891799173,4cyclictest12-21ksoftirqd/020:45:260
891799172,11cyclictest131rcu_preempt20:10:290
891799172,11cyclictest131rcu_preempt20:10:280
8917991712,4cyclictest12-21ksoftirqd/020:00:010
8917991711,4cyclictest12470-21irqcore20:30:140
8917991710,5cyclictest19698-21sshd19:53:240
892099168,6cyclictest0-21swapper/322:39:073
892099168,5cyclictest25866irq/58-eth020:10:113
892099168,5cyclictest25866irq/58-eth020:10:113
892099167,6cyclictest0-21swapper/321:55:093
892099162,9cyclictest0-21swapper/320:35:143
892099162,9cyclictest0-21swapper/320:30:093
892099162,9cyclictest0-21swapper/319:55:143
892099162,7cyclictest0-21swapper/320:00:143
892099162,10cyclictest0-21swapper/320:15:163
892099162,10cyclictest0-21swapper/300:00:163
891899169,5cyclictest5176-1kworker/u9:200:07:541
891899169,5cyclictest0-21swapper/120:10:161
891899169,5cyclictest0-21swapper/120:10:151
891899168,5cyclictest10559-21ssh00:30:011
891899166,7cyclictest4104-21sshd19:35:241
891899164,10cyclictest167-21systemd-network23:52:391
891899162,10cyclictest601-21rngd23:46:231
891899162,10cyclictest601-21rngd22:29:241
891899161,8cyclictest31382-21df_inode20:50:221
8918991611,3cyclictest0-21swapper/122:47:421
891899161,12cyclictest413-21vmstat19:30:341
891799167,6cyclictest29675-21munin-plugin-st20:10:000
892099151,9cyclictest0-21swapper/319:43:233
891899157,5cyclictest0-21swapper/123:10:011
891899150,13cyclictest8593-21ssh21:47:141
891799157,6cyclictest18027-21cut20:35:200
891799151,3cyclictest0-21swapper/020:26:400
892099149,3cyclictest0-21swapper/320:26:013
892099141,6cyclictest0-21swapper/320:50:293
891999141,6cyclictest11902-21sshd20:29:242
891899141,10cyclictest29711-21latency_hist20:10:021
892099138,3cyclictest0-21swapper/321:00:193
891899136,5cyclictest8955-21sed19:10:021
891799121,8cyclictest8926-21latency_hist19:10:010
892099101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional