You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-23 - 20:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Nov 23, 2024 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1632699771,50cyclictest5369-21ssh12:15:162
1632699771,50cyclictest5369-21ssh12:15:152
16326996216,34cyclictest22146irq/40-dwc2_hso08:30:142
1632699593,37cyclictest32340-21sh10:55:112
1631799562,45cyclictest6510-21sendmail-msp11:40:121
1632699553,41cyclictest32393-21sendmail-msp08:00:142
1632699553,41cyclictest32393-21sendmail-msp08:00:132
1631799542,43cyclictest18072-21sendmail-msp09:20:121
16326995317,27cyclictest0-21swapper/211:05:122
16326995218,13cyclictest3464-21ssh09:10:182
1632699512,26cyclictest16336-21ssh10:45:112
16326995116,23cyclictest27-21ksoftirqd/210:00:122
1631799481,35cyclictest9202-21ssh10:40:131
1631799472,34cyclictest13286-21sendmail-msp10:00:111
1631799442,4cyclictest131rcu_preempt09:30:191
16326994316,19cyclictest9472-21ssh11:00:222
1632699430,37cyclictest23418-21ssh09:45:142
1632699430,37cyclictest23418-21ssh09:45:132
1631799422,36cyclictest30858-21ssh09:50:171
16326994129,8cyclictest11543-21apt-get08:45:112
16326994115,10cyclictest181131/usr/sbin/munin07:45:142
16326994113,18cyclictest0-21swapper/209:42:052
16326994024,8cyclictest261rcuc/212:30:162
1633499392,23cyclictest25866irq/58-eth012:30:163
16326993918,17cyclictest21024-21/usr/sbin/munin10:05:282
16326993917,14cyclictest0-21swapper/212:05:132
16326993917,14cyclictest0-21swapper/212:05:132
16326993913,13cyclictest28019-21ssh11:32:432
16326993829,6cyclictest22146irq/40-dwc2_hso08:25:112
16326993819,14cyclictest0-21swapper/211:55:132
16326993816,12cyclictest0-21swapper/209:55:142
16334993722,9cyclictest25866irq/58-eth011:35:143
1633499371,19cyclictest25866irq/58-eth011:10:153
1633499371,19cyclictest25866irq/58-eth011:10:143
16326993721,12cyclictest14586-21kthreadcore10:20:272
16326993718,9cyclictest0-21swapper/207:30:132
16326993717,16cyclictest8322-21/usr/sbin/munin09:35:212
16326993715,19cyclictest28555-21grep09:25:202
16326993711,11cyclictest0-21swapper/209:19:082
1632699369,11cyclictest0-21swapper/212:24:062
1632699362,28cyclictest15474-21ssh11:23:382
16326993621,12cyclictest4511-21ssh09:52:562
16326993619,9cyclictest1159-21grep09:05:202
16326993619,14cyclictest705-21acpi09:30:192
16326993619,14cyclictest6636-21/usr/sbin/munin08:40:112
16326993616,16cyclictest32637-21cat10:10:332
16326993615,15cyclictest0-21swapper/208:15:132
16326993613,11cyclictest30431-21ssh12:28:072
16317993611,22cyclictest19442-21sendmail-msp08:20:121
16334993520,8cyclictest0-21swapper/311:45:143
1633499351,20cyclictest25866irq/58-eth009:05:133
16326993520,12cyclictest28731-21acpi12:10:142
16326993520,12cyclictest0-21swapper/207:15:132
16326993518,14cyclictest12802-21grep08:10:232
16326993513,12cyclictest27785-21tr07:20:222
1630999352,23cyclictest12928-21sendmail-msp07:40:130
1630999352,23cyclictest12928-21sendmail-msp07:40:120
1630999351,25cyclictest8161-21sendmail-msp11:20:130
1633499345,16cyclictest0-21swapper/312:05:153
1633499345,16cyclictest0-21swapper/312:05:143
16326993419,8cyclictest11853-21processes07:35:272
16326993419,12cyclictest25016-21unixbench_multi08:55:282
16326993419,12cyclictest23194-21/usr/sbin/munin08:20:292
16326993419,12cyclictest20124-21switchtime07:10:272
16326993419,12cyclictest19752-21/usr/sbin/munin11:45:272
16326993418,8cyclictest3619-21ssh11:36:332
16326993418,12cyclictest20140-21timerwakeupswit08:50:292
16326993417,8cyclictest11436-21cat11:40:332
16326993417,13cyclictest26348-21grep10:50:242
16326993417,12cyclictest3549-21/usr/sbin/munin10:15:172
16326993416,11cyclictest1818-21fschecks_time10:35:162
16326993412,9cyclictest15916-21ssh12:38:012
1630999345,26cyclictest25814-21sendmail-msp09:00:130
1630999345,26cyclictest25814-21sendmail-msp09:00:130
1632699334,17cyclictest0-21swapper/210:40:132
16326993320,6cyclictest27206-21kthreadcore09:00:222
16326993320,6cyclictest27206-21kthreadcore09:00:212
16326993319,12cyclictest14667-21sh12:00:332
16326993319,11cyclictest3094-21diskmemload09:23:412
16326993318,12cyclictest29537-21ssh11:10:522
16326993318,12cyclictest29537-21ssh11:10:512
16326993317,9cyclictest20886-21latency_hist11:30:022
16326993317,8cyclictest22896-21awk10:25:282
16326993317,12cyclictest17536-21cat07:40:342
16326993317,12cyclictest17536-21cat07:40:332
16326993316,14cyclictest0-21swapper/211:15:132
16326993314,15cyclictest1755-21/usr/sbin/munin08:35:102
16326993226,3cyclictest24927-21apt-get11:50:142
16326993217,12cyclictest16335-21latency_hist07:10:022
16326993216,13cyclictest22910-21/usr/sbin/munin07:50:092
16326993216,12cyclictest28003-21idleruntime07:55:162
1630999323,27cyclictest1557-21sh11:55:140
1630999322,26cyclictest6418-21sendmail-msp08:40:130
16326993120,8cyclictest2306-21switchtime07:25:262
16326993116,13cyclictest307-21ssh10:33:142
16309993119,9cyclictest12-21ksoftirqd/007:25:110
1633499302,24cyclictest25866irq/58-eth011:30:123
16326993015,7cyclictest5678-21ps08:05:172
1631799301,20cyclictest24438-21ssh11:10:141
1631799301,20cyclictest24438-21ssh11:10:141
16309993026,2cyclictest32346-21ssh10:55:120
1633499293,13cyclictest25866irq/58-eth010:45:113
1631799292,19cyclictest8161-21sendmail-msp11:20:141
1631799292,17cyclictest19018-21ssh10:25:141
1631799292,15cyclictest19768-21ssh12:05:141
1631799292,15cyclictest19768-21ssh12:05:131
16317992912,13cyclictest10200-21sendmail-msp12:00:151
16309992917,9cyclictest10875-21sendmail-msp10:20:150
1631799287,13cyclictest21-21ksoftirqd/111:15:141
1631799282,20cyclictest13979-21sendmail-msp12:20:151
1630999288,15cyclictest21244-21ssh11:30:110
1630999284,21cyclictest7764-21sendmail-msp11:00:120
1630999282,23cyclictest24456-21ssh11:10:160
1630999282,23cyclictest24456-21ssh11:10:150
1630999282,22cyclictest0-21swapper/012:05:150
1630999282,22cyclictest0-21swapper/012:05:140
1630999281,21cyclictest900-21snmpd12:35:150
1633499273,13cyclictest0-21swapper/312:15:143
1633499273,13cyclictest0-21swapper/312:15:143
16334992710,10cyclictest321rcuc/310:25:133
16309992718,7cyclictest12-21ksoftirqd/012:30:160
16309992714,10cyclictest25964-21sendmail-msp07:20:150
1630999271,22cyclictest5353-21ssh12:15:150
1630999271,22cyclictest5353-21ssh12:15:140
1633499261,16cyclictest25866irq/58-eth011:55:143
1631799261,17cyclictest5346-21ssh12:15:141
1631799261,17cyclictest5346-21ssh12:15:131
1631799261,10cyclictest15621-21ssh11:04:381
1630999263,18cyclictest12-21ksoftirqd/007:35:120
1630999262,21cyclictest15888-21ssh11:45:120
1630999261,16cyclictest21059-21ssh10:05:130
1633499259,10cyclictest0-21swapper/310:56:063
1633499254,14cyclictest0-21swapper/309:50:183
1633499252,17cyclictest0-21swapper/311:25:383
16334992513,8cyclictest25866irq/58-eth009:35:133
1631799253,16cyclictest32393-21sendmail-msp08:00:161
1631799253,16cyclictest32393-21sendmail-msp08:00:151
1631799252,18cyclictest5154-21sh10:56:061
1631799251,18cyclictest1773-21apt-get08:35:111
1631799251,16cyclictest3580-21ssh10:15:151
1630999254,19cyclictest3478-21ssh09:10:180
1630999251,15cyclictest13979-21sendmail-msp12:20:130
1631799241,19cyclictest15900-21ssh11:45:141
1631799241,17cyclictest28756-21ssh12:10:141
16317992410,10cyclictest21153-21df10:05:161
1630999242,18cyclictest8364-21ssh09:35:130
1630999242,18cyclictest16249-21ssh11:05:140
1630999240,7cyclictest131rcu_preempt12:10:160
16334992315,5cyclictest0-21swapper/308:45:123
1631799231,18cyclictest10106-21cut08:10:161
1631799231,17cyclictest11411-21cut10:20:171
1630999239,11cyclictest16125-21cron08:50:010
16309992313,7cyclictest15624-21sendmail-msp09:40:110
16309992313,7cyclictest10200-21sendmail-msp12:00:120
1630999231,16cyclictest23418-21ssh12:25:140
16334992214,6cyclictest0-21swapper/307:20:143
1631799222,13cyclictest21-21ksoftirqd/108:45:111
1631799221,17cyclictest2066-21sshd07:25:231
1630999228,10cyclictest32393-21sendmail-msp08:00:310
1630999228,10cyclictest32393-21sendmail-msp08:00:300
1630999222,5cyclictest52246usb-storage10:00:060
1630999222,15cyclictest19442-21sendmail-msp08:20:260
1630999222,14cyclictest27269-21timerandwakeup09:45:290
1630999222,14cyclictest27269-21timerandwakeup09:45:280
1630999221,15cyclictest8883-21sendmail-msp10:40:210
1630999221,10cyclictest18367-21find09:20:110
1633499215,10cyclictest25866irq/58-eth009:40:113
1633499212,11cyclictest25866irq/58-eth008:40:133
16334992112,6cyclictest25866irq/58-eth011:50:133
1631799215,9cyclictest131rcu_preempt09:55:131
16317992112,7cyclictest7078-21sshd11:56:191
1631799211,14cyclictest8298-21find07:35:121
16309992113,5cyclictest30827-21/usr/sbin/munin09:50:310
1630999211,16cyclictest8411-21latency08:05:210
1630999211,16cyclictest16341-21ssh10:45:120
1630999211,15cyclictest738-21ssh09:30:200
1630999211,14cyclictest845-21munin-node08:35:130
1630999211,10cyclictest8339-21kthreadcore09:55:200
1633499208,8cyclictest25866irq/58-eth012:00:153
1633499208,7cyclictest321rcuc/309:55:133
1633499203,8cyclictest25866irq/58-eth008:25:123
16334992014,4cyclictest0-21swapper/310:40:203
16334992012,6cyclictest0-21swapper/310:03:073
1631799201,16cyclictest24273-21apt-get10:50:141
1631799201,14cyclictest6418-21sendmail-msp08:40:291
1631799200,18cyclictest28394-21ls11:50:221
1631799200,17cyclictest24592-21ls08:25:131
1630999209,8cyclictest7188-21if_err_eth111:40:170
1630999207,11cyclictest12-21ksoftirqd/011:25:380
1630999201,8cyclictest9078-21latency_hist10:40:020
1630999201,16cyclictest29842-21ssh11:50:320
1630999201,15cyclictest22928-21apt-get07:50:090
1630999201,14cyclictest30316-21ssh11:35:130
1630999201,13cyclictest22656-21sh10:25:250
16309992010,7cyclictest0-21swapper/008:55:170
1630999200,18cyclictest18248-21fschecks_count07:45:180
1633499194,10cyclictest0-21swapper/312:20:153
1633499193,11cyclictest0-21swapper/311:09:393
1633499192,13cyclictest0-21swapper/307:40:243
1633499192,13cyclictest0-21swapper/307:40:243
1633499192,10cyclictest0-21swapper/308:00:043
1633499192,10cyclictest0-21swapper/308:00:043
1633499191,12cyclictest0-21swapper/312:35:323
1633499191,11cyclictest0-21swapper/309:45:143
1633499191,11cyclictest0-21swapper/309:45:133
16334991910,7cyclictest0-21swapper/310:05:243
16334991910,6cyclictest0-21swapper/310:40:033
16334991910,6cyclictest0-21swapper/310:33:063
16334991910,6cyclictest0-21swapper/310:18:343
16334991910,4cyclictest0-21swapper/311:15:293
1631799198,8cyclictest28737-21cut07:20:241
1631799197,8cyclictest32393-21sendmail-msp08:00:011
1631799194,10cyclictest23407-21ssh09:45:121
1631799194,10cyclictest23407-21ssh09:45:121
1631799193,9cyclictest0-21swapper/109:25:131
1631799191,5cyclictest52246usb-storage09:00:241
1631799191,5cyclictest52246usb-storage09:00:231
1631799191,15cyclictest5286-21uniq08:05:151
1631799191,14cyclictest9087-21rm12:35:181
1631799191,12cyclictest16581-21cut07:10:151
1631799190,13cyclictest16499-21cut08:50:171
1630999198,9cyclictest632-21Async17:44:200
1630999193,13cyclictest24279-21ssh10:50:170
16309991911,6cyclictest19392-21sshd08:16:380
1630999191,13cyclictest8121-21cron11:20:000
1633499188,6cyclictest0-21swapper/311:04:583
1633499182,11cyclictest0-21swapper/308:15:163
1633499182,11cyclictest0-21swapper/307:50:123
1633499182,10cyclictest0-21swapper/307:45:043
1633499182,10cyclictest0-21swapper/307:10:143
16334991812,4cyclictest0-21swapper/311:40:113
1633499181,12cyclictest0-21swapper/311:20:153
16334991810,6cyclictest0-21swapper/312:25:333
16334991810,6cyclictest0-21swapper/310:52:523
16334991810,6cyclictest0-21swapper/309:28:533
16334991810,6cyclictest0-21swapper/309:23:293
1631799188,8cyclictest23131-21perl07:50:141
1631799188,7cyclictest15624-21sendmail-msp09:40:171
1631799183,10cyclictest0-21swapper/109:15:141
1631799182,12cyclictest28601-21df_inode10:10:181
1631799182,11cyclictest21-21ksoftirqd/111:30:111
1631799182,11cyclictest0-21swapper/112:29:371
1631799181,15cyclictest23836-21kthreadcore08:55:201
16317991810,5cyclictest7537-21usage_Anybus09:10:351
16317991810,4cyclictest16622-21irqcore11:05:171
1631799180,15cyclictest6830-21ls07:30:241
1631799180,12cyclictest16495-21cut10:45:151
1631799180,11cyclictest8615-21ls09:35:161
1630999183,9cyclictest0-21swapper/010:30:150
1630999183,12cyclictest3599-21sort07:30:170
1630999182,4cyclictest131rcu_preempt10:15:190
16309991810,6cyclictest21083-21cron07:15:020
1633499179,6cyclictest0-21swapper/312:10:243
1633499179,5cyclictest25866irq/58-eth009:30:193
1633499173,9cyclictest25866irq/58-eth010:10:173
1633499172,9cyclictest0-21swapper/307:15:143
1633499172,8cyclictest0-21swapper/308:35:113
16334991710,5cyclictest0-21swapper/309:15:323
16334991710,4cyclictest0-21swapper/309:10:293
1631799179,5cyclictest3493-21latency12:30:251
1631799179,5cyclictest29713-21grep08:30:191
1631799178,7cyclictest12928-21sendmail-msp07:40:091
1631799178,7cyclictest12928-21sendmail-msp07:40:081
1631799172,11cyclictest3381-21ssh11:36:161
1631799172,11cyclictest0-21swapper/109:05:141
1631799172,10cyclictest0-21swapper/107:15:141
1631799171,12cyclictest5743-21awk10:35:311
1630999174,10cyclictest22125-21kthreadcore07:15:190
1630999172,11cyclictest32600-21sed08:00:010
1630999172,11cyclictest14683-21grep08:15:020
16309991711,4cyclictest18240-21latency_hist09:20:020
16309991711,4cyclictest0-21swapper/009:25:160
1633499162,10cyclictest0-21swapper/310:20:143
1633499161,7cyclictest0-21swapper/308:20:193
1631799162,11cyclictest19189-21kthreadcore07:45:201
1631799162,10cyclictest30029-21ssh10:30:261
16317991610,3cyclictest20883-21sh11:30:001
1630999164,9cyclictest27000-21kthreadcore08:25:210
1630999162,11cyclictest16820-21irqcore08:50:180
1630999161,12cyclictest28742-21sed10:10:200
1630999160,13cyclictest32564-21tr09:05:190
1633499151,8cyclictest0-21swapper/309:00:183
1633499151,8cyclictest0-21swapper/309:00:173
1633499151,12cyclictest0-21swapper/308:35:013
1633499151,11cyclictest0-21swapper/307:55:203
1631799152,10cyclictest14816-21cpu08:15:161
1630999151,11cyclictest16384-21latency_hist07:10:010
1633499149,3cyclictest0-21swapper/308:50:183
1633499141,7cyclictest0-21swapper/308:55:143
1633499141,7cyclictest0-21swapper/307:30:143
1631799141,9cyclictest16342-21cat07:10:011
1633499132,7cyclictest0-21swapper/307:25:113
1633499131,9cyclictest0-21swapper/308:15:013
1633499131,6cyclictest0-21swapper/308:05:153
1633499131,10cyclictest0-21swapper/307:40:013
1633499101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional