You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-25 - 14:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Mon Nov 25, 2024 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13379995526,22cyclictest15666-21sendmail-msp21:40:120
1340599501,29cyclictest14927-21ssh22:40:142
1340599471,6cyclictest0-21swapper/221:20:132
1339299472,39cyclictest16215-21sendmail-msp23:40:121
1337999472,35cyclictest2894-21sendmail-msp20:40:130
13392994522,20cyclictest22915-21sendmail-msp19:20:121
1339299444,32cyclictest9895-21sendmail-msp19:40:151
1337999441,31cyclictest15790-21ssh22:00:120
1337999423,31cyclictest28426-21ssh00:10:150
1337999421,3cyclictest131rcu_preempt21:20:120
1337999421,28cyclictest11930-21sendmail-msp00:00:120
13416994124,11cyclictest25866irq/58-eth000:25:113
13416994124,11cyclictest25866irq/58-eth000:25:103
1339299414,23cyclictest32380-21ssh21:10:141
1337999410,27cyclictest31843-21ssh23:10:170
1339299406,26cyclictest15615-21sendmail-msp23:20:131
13392993915,5cyclictest131rcu_preempt23:25:131
1337999391,9cyclictest0-21swapper/022:35:120
1341699371,20cyclictest25866irq/58-eth023:25:143
1341699362,22cyclictest25866irq/58-eth021:10:133
1339299351,31cyclictest7644-21ssh23:15:111
1337999343,26cyclictest24110-21ssh21:25:120
13379993415,10cyclictest632-21Async17:44:200
1340599339,19cyclictest26124-21ssh00:28:532
1340599339,19cyclictest26124-21ssh00:28:532
1340599332,21cyclictest131rcu_preempt23:10:172
1339299332,24cyclictest31973-21ssh21:50:181
13392993316,13cyclictest14966-21sendmail-msp22:20:141
13416993216,11cyclictest25866irq/58-eth021:50:173
1340599324,18cyclictest22146irq/40-dwc2_hso22:55:122
13392993224,4cyclictest131rcu_preempt22:05:141
1337999322,21cyclictest12-21ksoftirqd/020:00:140
1340599311,24cyclictest14856-21sendmail-msp23:00:132
1340599311,22cyclictest29164-21sendmail-msp20:00:122
13405993112,14cyclictest16045-21sendmail-msp20:20:132
1339299313,25cyclictest14856-21sendmail-msp23:00:131
1340599303,17cyclictest22146irq/40-dwc2_hso19:25:122
1339299309,17cyclictest21-21ksoftirqd/120:30:131
1339299309,17cyclictest21-21ksoftirqd/120:30:121
1339299303,22cyclictest24019-21ssh21:45:131
13392993017,10cyclictest22248-21sendmail-msp21:00:131
1339299301,25cyclictest23564-21sh23:05:131
13379993016,10cyclictest23470-21ssh22:25:160
1340599293,18cyclictest22146irq/40-dwc2_hso00:35:142
1340599293,17cyclictest22146irq/40-dwc2_hso20:15:142
1340599293,15cyclictest22146irq/40-dwc2_hso19:30:122
1340599292,20cyclictest22146irq/40-dwc2_hso20:05:132
13405992911,6cyclictest0-21swapper/222:00:142
13392992913,8cyclictest21-21ksoftirqd/122:15:141
13392992821,5cyclictest0-21swapper/121:30:151
13392992819,6cyclictest21-21ksoftirqd/100:25:121
13392992819,6cyclictest21-21ksoftirqd/100:25:111
13392992813,11cyclictest14569-21sendmail-msp22:40:141
1339299281,21cyclictest20537-21ssh00:05:101
1337999282,23cyclictest11530-21ssh00:20:130
1337999281,17cyclictest25367-21ssh00:05:330
13416992711,10cyclictest25866irq/58-eth000:19:373
1340599273,15cyclictest22146irq/40-dwc2_hso20:45:122
1340599272,18cyclictest522-21usb-storage23:05:032
1340599272,16cyclictest22146irq/40-dwc2_hso23:15:102
13405992715,9cyclictest27-21ksoftirqd/200:30:162
1340599271,16cyclictest12243-21find00:00:112
1339299272,19cyclictest28455-21sh23:50:121
1339299272,18cyclictest24125-21ssh21:25:131
1339299271,20cyclictest31797-21sh22:10:181
1337999273,21cyclictest24270-21ssh23:25:120
13379992720,5cyclictest12-21ksoftirqd/021:10:120
13379992711,3cyclictest131rcu_preempt21:45:220
1341699263,15cyclictest0-21swapper/300:33:373
1340599268,12cyclictest27-21ksoftirqd/223:20:332
1340599266,14cyclictest22146irq/40-dwc2_hso20:10:112
1340599263,15cyclictest22146irq/40-dwc2_hso22:50:152
1340599261,16cyclictest11581-21ls21:15:232
1339299263,17cyclictest15963-21ssh21:40:111
1339299262,18cyclictest3763-21ssh00:15:131
1339299261,20cyclictest32608-21ssh23:30:171
1337999262,7cyclictest131rcu_preempt21:15:130
1337999262,19cyclictest15972-21ssh23:20:140
13379992611,11cyclictest12-21ksoftirqd/022:55:130
13416992513,9cyclictest0-21swapper/323:15:013
1340599255,12cyclictest22146irq/40-dwc2_hso22:20:122
1340599252,17cyclictest22146irq/40-dwc2_hso21:05:132
1340599252,15cyclictest22146irq/40-dwc2_hso19:55:142
1339299255,15cyclictest201rcuc/100:30:151
1339299252,18cyclictest11930-21sendmail-msp00:00:281
1337999251,19cyclictest16045-21sendmail-msp20:20:280
13416992418,4cyclictest25866irq/58-eth023:50:123
1341699241,15cyclictest0-21swapper/300:20:133
1340599249,12cyclictest522-21usb-storage21:00:112
1340599242,6cyclictest0-21swapper/222:45:132
1340599242,17cyclictest22146irq/40-dwc2_hso00:10:132
13405992417,3cyclictest8382-21ssh23:35:132
1340599241,19cyclictest7932-21ssh21:35:112
1340599241,18cyclictest31468-21idleruntime22:30:202
1340599241,18cyclictest16215-21sendmail-msp23:40:122
13405992411,10cyclictest0-21swapper/222:40:002
1339299242,19cyclictest21-21ksoftirqd/119:55:131
1339299242,17cyclictest14857-21awk23:00:011
1339299241,20cyclictest23161-21ssh22:45:141
1339299241,18cyclictest23045-21ls19:20:021
13392992410,10cyclictest15744-21sendmail-msp21:20:301
1337999243,14cyclictest7931-21sh21:35:120
13379992411,9cyclictest7201-21ssh00:15:230
1341699235,13cyclictest25866irq/58-eth022:50:153
1341699233,15cyclictest25866irq/58-eth023:05:123
1341699233,10cyclictest0-21swapper/322:35:113
1341699231,13cyclictest25866irq/58-eth022:40:133
1340599239,4cyclictest3585-21sh21:10:272
1340599232,7cyclictest0-21swapper/221:25:112
1340599231,19cyclictest13638-21kthreadcore00:20:232
13405992311,9cyclictest0-21swapper/219:20:172
1339299234,15cyclictest7578-21ssh21:55:131
1339299232,17cyclictest31216-21ssh22:30:161
1339299232,16cyclictest31906-21diskmemload00:24:381
1339299231,17cyclictest406-21ls19:30:151
13379992315,5cyclictest3706-21ssh23:30:280
1341699223,14cyclictest0-21swapper/323:00:143
13416992210,7cyclictest25866irq/58-eth023:35:133
1340599221,11cyclictest19101-21tr21:40:252
13405992210,9cyclictest7320-21sh22:15:122
13405992210,9cyclictest0-21swapper/219:40:292
13405992210,8cyclictest22146irq/40-dwc2_hso20:40:282
13379992214,5cyclictest0-21swapper/021:00:110
1337999221,16cyclictest14569-21sendmail-msp22:40:310
1341699217,9cyclictest25866irq/58-eth021:25:133
1341699212,15cyclictest25866irq/58-eth022:10:163
1341699212,13cyclictest25866irq/58-eth019:15:133
1341699212,13cyclictest0-21swapper/321:45:133
1341699211,13cyclictest0-21swapper/322:20:123
1340599213,12cyclictest22146irq/40-dwc2_hso20:25:122
1340599211,17cyclictest31933-21find21:50:162
1340599211,16cyclictest9122-21grep19:35:292
1339299212,16cyclictest19445-21ntp_kernel_pll_22:00:291
1339299211,6cyclictest52246usb-storage00:40:021
13392992113,6cyclictest18060-21sshd19:11:361
1339299211,10cyclictest711-21cron20:30:011
1339299211,10cyclictest10768-21unixbench_multi22:35:291
1339299210,18cyclictest15208-21ls20:15:301
1337999219,3cyclictest131rcu_preempt23:00:280
13379992112,6cyclictest22915-21sendmail-msp19:20:090
1337999211,18cyclictest32422-21cut21:30:190
1337999211,15cyclictest4154-21find23:55:120
1337999211,15cyclictest17896-21ls20:55:160
1337999211,15cyclictest14966-21sendmail-msp22:20:150
1337999210,18cyclictest27588-21ls22:05:290
1341699209,9cyclictest0-21swapper/323:57:033
1341699202,9cyclictest25866irq/58-eth023:20:123
13416992010,6cyclictest25866irq/58-eth023:15:123
1340599203,5cyclictest131rcu_preempt21:45:192
1340599203,10cyclictest32514-21kthreadcore22:10:242
1340599202,10cyclictest30872-21tune2fs20:35:172
1340599201,9cyclictest4445-21sed23:55:172
1340599201,9cyclictest19524-21grep20:55:202
1340599201,16cyclictest32439-21tune2fs21:30:192
1340599201,13cyclictest28454-21ssh23:50:112
1340599201,11cyclictest23893-21sort22:05:162
1340599201,11cyclictest23534-21kthreadcore00:05:222
1340599201,10cyclictest7716-21fschecks_count21:55:162
1340599201,10cyclictest4473-21kthreadcore00:15:202
1340599201,10cyclictest28108-21mailstats23:25:292
1339299204,10cyclictest0-21swapper/120:35:151
1339299201,5cyclictest52246usb-storage21:00:021
1339299201,5cyclictest52246usb-storage20:45:021
13392992013,4cyclictest0-21swapper/121:35:321
1339299201,14cyclictest8085-21ssh21:15:131
1339299201,10cyclictest8799-21kthreadcore20:10:201
13392992010,7cyclictest16045-21sendmail-msp20:20:201
1337999208,9cyclictest6643-21grep00:35:310
13379992012,4cyclictest29157-21kthreadcore00:30:250
1337999201,15cyclictest31972-21sh21:50:170
1337999200,15cyclictest8361-21find23:35:110
1341699193,13cyclictest25866irq/58-eth022:05:143
1341699192,11cyclictest0-21swapper/323:30:183
13416991911,6cyclictest0-21swapper/322:48:193
1341699191,14cyclictest25866irq/58-eth021:30:153
1341699191,12cyclictest0-21swapper/321:55:153
1341699191,12cyclictest0-21swapper/320:25:223
1340599196,5cyclictest659-21irqcore23:30:222
1340599192,9cyclictest19905-21kthreadcore19:15:212
1340599191,9cyclictest27501-21kthreadcore20:30:212
1340599191,9cyclictest27501-21kthreadcore20:30:212
1340599191,9cyclictest16482-21kthreadcore19:45:212
1340599191,8cyclictest23494-21rm22:25:152
1340599191,15cyclictest12945-21fschecks_count20:50:162
1340599191,10cyclictest16899-21awk19:10:242
1339299199,7cyclictest23435-21fschecks_count22:25:161
1339299196,10cyclictest19984-21hwlatdetect19:50:201
1339299193,12cyclictest6907-21ssh23:55:211
1339299191,2cyclictest131rcu_preempt23:10:161
13392991912,4cyclictest0-21swapper/123:47:091
1339299191,15cyclictest8370-21apt-get23:35:121
1339299191,15cyclictest28416-21apt-get00:10:141
13392991911,5cyclictest0-21swapper/120:05:181
1337999199,7cyclictest7638-21apt-get23:15:110
1337999199,7cyclictest16215-21sendmail-msp23:40:140
1337999192,14cyclictest30825-21apt-get22:50:050
1337999192,13cyclictest16265-21ntp_192.168.11420:50:250
1337999191,12cyclictest845-21munin-node20:05:140
1337999191,10cyclictest14812-21tr19:10:210
13379991910,6cyclictest31413-21ssh22:30:200
1341699189,6cyclictest25866irq/58-eth000:35:153
1341699189,6cyclictest0-21swapper/322:33:043
1341699189,6cyclictest0-21swapper/321:17:593
1341699189,6cyclictest0-21swapper/300:05:073
1341699189,6cyclictest0-21swapper/300:02:253
1341699188,7cyclictest0-21swapper/322:15:203
1341699188,7cyclictest0-21swapper/321:35:123
1341699187,7cyclictest25866irq/58-eth022:25:133
1341699183,9cyclictest25866irq/58-eth023:40:123
1341699183,10cyclictest0-21swapper/322:00:053
1341699182,10cyclictest0-21swapper/321:00:123
13416991812,4cyclictest0-21swapper/321:45:003
13416991811,4cyclictest0-21swapper/321:20:143
1341699181,10cyclictest0-21swapper/320:55:013
13416991810,5cyclictest25866irq/58-eth000:10:153
1340599181,8cyclictest22614-21kthreadcore19:50:242
1340599181,14cyclictest28119-21munin-plugin-st23:50:022
1339299188,7cyclictest0-21swapper/119:25:281
1339299182,10cyclictest0-21swapper/122:50:141
1339299181,14cyclictest29926-21sed20:00:201
1339299181,13cyclictest711-21cron19:50:011
1339299181,13cyclictest12966-21cut20:50:161
1339299181,13cyclictest12754-21latency_hist20:50:011
13392991810,6cyclictest31888-21sshd21:05:551
1337999187,8cyclictest2853-21cron20:40:010
1337999185,9cyclictest8479-21kthreadcore21:55:200
1337999183,12cyclictest15601-21kernelversion19:45:200
1337999183,10cyclictest0-21swapper/000:27:380
1337999183,10cyclictest0-21swapper/000:27:370
1337999182,12cyclictest9895-21sendmail-msp19:40:170
13379991811,5cyclictest31891-21ssh22:10:190
13379991811,5cyclictest11182-21cron20:15:020
1337999181,14cyclictest25220-21kthreadcore23:05:190
13379991810,6cyclictest1571-21sshd23:51:380
1341699179,6cyclictest0-21swapper/323:45:303
1341699172,9cyclictest0-21swapper/319:45:133
1341699172,10cyclictest0-21swapper/320:30:143
1341699172,10cyclictest0-21swapper/320:30:133
1341699171,13cyclictest0-21swapper/319:30:153
13416991710,4cyclictest0-21swapper/322:55:023
1339299172,10cyclictest0-21swapper/119:35:141
1337999171,8cyclictest8732-21ssh22:15:200
1337999171,8cyclictest25387-21cut22:45:200
1337999171,13cyclictest21785-21kthreadcore20:25:190
1337999171,11cyclictest4100-21wc19:30:290
13379991710,5cyclictest25510-21ssh23:46:140
13379991710,3cyclictest23065-21cat19:50:260
1337999170,10cyclictest8912-21ls19:35:280
1341699163,9cyclictest0-21swapper/319:40:163
1341699161,8cyclictest0-21swapper/321:10:013
1341699161,8cyclictest0-21swapper/320:15:023
13416991611,3cyclictest33-21ksoftirqd/319:25:263
1341699161,12cyclictest0-21swapper/320:55:143
1341699161,12cyclictest0-21swapper/319:55:143
1339299162,11cyclictest13462-21latency_hist19:10:011
1337999164,9cyclictest11408-21sort20:15:130
1337999161,9cyclictest0-21swapper/019:15:160
13379991611,3cyclictest24862-21interrupts19:55:180
1337999161,11cyclictest28103-21diskstats19:25:140
13379991610,4cyclictest0-21swapper/021:05:230
1341699152,10cyclictest0-21swapper/319:50:173
1337999151,11cyclictest25906-21apt-get20:30:140
1337999151,11cyclictest25906-21apt-get20:30:130
1341699142,7cyclictest0-21swapper/319:20:133
1341699141,10cyclictest0-21swapper/320:15:193
1341699138,3cyclictest0-21swapper/320:44:363
1341699131,9cyclictest0-21swapper/320:20:153
1341699131,8cyclictest0-21swapper/320:50:013
1341699131,8cyclictest0-21swapper/320:00:243
1341699131,5cyclictest0-21swapper/320:05:213
1341699131,5cyclictest0-21swapper/319:35:203
1340599131,9cyclictest13441-21latency_hist19:10:012
1341699127,3cyclictest0-21swapper/320:36:023
1341699124,5cyclictest0-21swapper/319:10:143
1337999121,2cyclictest131rcu_preempt19:10:000
1341699101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional