You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 08:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 22, 2024 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5262999653,10cyclictest24358-21diskmemload21:35:110
526299701,39cyclictest8310-21sendmail-msp20:20:120
5297996647,13cyclictest25866irq/58-eth022:50:163
526299667,8cyclictest131rcu_preempt20:30:120
5262996510,8cyclictest131rcu_preempt19:55:110
526299613,39cyclictest19958-21ssh23:25:110
526299595,8cyclictest131rcu_preempt20:35:110
526299582,50cyclictest4628-21ssh22:35:120
527499571,39cyclictest10734-21sendmail-msp22:00:131
526299532,44cyclictest6668-21sendmail-msp00:00:120
526299522,9cyclictest131rcu_preempt00:25:130
526299522,40cyclictest14620-21sendmail-msp21:00:130
5284995116,30cyclictest10962-21ssh22:58:312
526299513,39cyclictest20857-21ssh22:45:120
526299513,39cyclictest20857-21ssh22:45:120
5284994925,16cyclictest20373-21ssh23:05:142
526299495,34cyclictest21230-21sendmail-msp20:00:160
5262994937,6cyclictest522-21usb-storage22:20:130
5262994931,8cyclictest12-21ksoftirqd/019:50:130
526299482,34cyclictest12964-21ssh22:40:120
5297994638,4cyclictest25866irq/58-eth021:55:103
526299461,33cyclictest9429-21sendmail-msp21:40:120
5284994224,15cyclictest27897-21apt-get00:35:122
5284994224,15cyclictest27897-21apt-get00:35:122
527499421,35cyclictest28187-21ssh23:30:171
5274994212,11cyclictest31048-21ssh00:15:121
5262994219,10cyclictest12-21ksoftirqd/019:25:120
5274994131,5cyclictest13420-21apt-get20:25:111
526299415,3cyclictest131rcu_preempt23:35:130
526299411,4cyclictest131rcu_preempt19:40:150
526299411,33cyclictest14821-21sendmail-msp19:20:120
5284994021,14cyclictest0-21swapper/223:55:112
5274994033,3cyclictest20851-21apt-get22:45:131
5274994033,3cyclictest20851-21apt-get22:45:131
5262994020,13cyclictest8696-21sendmail-msp21:20:140
5284993920,14cyclictest0-21swapper/220:50:132
5284993919,15cyclictest0-21swapper/221:25:142
5284993911,23cyclictest0-21swapper/223:18:312
5274993830,4cyclictest31179-21apt-get20:10:131
5274993827,8cyclictest19746-21apt-get21:05:131
5274993827,8cyclictest19746-21apt-get21:05:131
527499382,27cyclictest522-21usb-storage21:45:111
527499381,29cyclictest11676-21sendmail-msp23:40:131
526299382,31cyclictest26743-21ssh21:50:150
5262993819,13cyclictest27664-21sendmail-msp20:40:130
526299381,30cyclictest5920-21sendmail-msp00:20:130
526299381,30cyclictest4327-21ssh22:55:100
529799372,22cyclictest25866irq/58-eth022:45:123
529799372,22cyclictest25866irq/58-eth022:45:123
528499374,8cyclictest18117-21ssh22:21:552
5284993723,9cyclictest3871-21apt-get23:35:142
5284993722,12cyclictest2650-21ssh21:55:112
5284993721,11cyclictest28900-21apt-get22:30:152
5284993712,21cyclictest10734-21sendmail-msp22:00:112
5274993731,3cyclictest11775-21apt-get19:50:131
5274993729,4cyclictest10004-21apt-get20:55:101
526299375,20cyclictest11990-21sendmail-msp23:00:120
5284993630,3cyclictest19944-21apt-get19:25:122
5284993627,4cyclictest3903-21apt-get22:15:122
5284993627,4cyclictest3903-21apt-get22:15:122
5284993617,15cyclictest0-21swapper/200:00:122
5284993610,12cyclictest6137-21ssh21:16:222
5284993610,12cyclictest6137-21ssh21:16:222
5274993628,5cyclictest18253-21apt-get20:30:121
5274993622,10cyclictest17457-21ssh21:25:141
5274993610,11cyclictest2309-21grep23:55:211
5284993528,3cyclictest321-21apt-get20:45:112
5284993514,18cyclictest2300-21ssh00:15:262
527499359,9cyclictest11220-21ssh23:19:331
527499359,8cyclictest0-21swapper/100:21:451
527499357,11cyclictest24008-21/usr/sbin/munin23:25:291
5274993525,7cyclictest9746-21apt-get21:40:111
526299352,27cyclictest23194-21ssh00:10:190
529799341,17cyclictest25866irq/58-eth021:45:103
528499347,10cyclictest0-21swapper/223:32:412
528499344,14cyclictest23413-21/usr/sbin/munin23:50:252
5284993428,3cyclictest13886-21apt-get00:25:132
527499348,11cyclictest10368-21sh22:57:311
527499346,8cyclictest0-21swapper/122:24:481
5274993411,8cyclictest31396-21ssh21:13:081
528499336,15cyclictest31747-21mailstats22:10:292
528499335,9cyclictest30837-21ssh21:30:582
5284993316,14cyclictest0-21swapper/221:50:162
527499339,7cyclictest2294-21ssh22:52:201
527499338,9cyclictest10797-21sh22:37:231
527499336,8cyclictest0-21swapper/100:25:061
5274993322,8cyclictest28906-21ssh22:30:151
5274993320,9cyclictest20353-21ssh23:05:121
5274993310,7cyclictest13544-21ssh00:03:091
529799322,19cyclictest25866irq/58-eth000:30:163
529799321,18cyclictest25866irq/58-eth021:35:093
5284993220,8cyclictest0-21swapper/220:30:122
5284993216,13cyclictest0-21swapper/220:10:132
5284993216,13cyclictest0-21swapper/219:15:122
5284993211,7cyclictest21407-21ssh23:47:232
527499327,8cyclictest0-21swapper/123:02:441
5274993224,3cyclictest6933-21apt-get19:45:141
5274993221,8cyclictest16585-21apt-get19:55:111
5274993211,8cyclictest24358-21diskmemload23:36:501
5274993210,9cyclictest28163-21ssh00:10:461
5274993210,8cyclictest27078-21ssh00:34:081
526299328,8cyclictest12-21ksoftirqd/021:15:130
526299328,8cyclictest12-21ksoftirqd/021:15:130
526299324,14cyclictest12-21ksoftirqd/000:15:130
526299323,4cyclictest131rcu_preempt23:20:150
5297993115,10cyclictest0-21swapper/300:03:253
5284993114,8cyclictest4930-21memory21:35:202
5284993114,14cyclictest21061-21cut21:05:202
5284993114,14cyclictest21061-21cut21:05:202
527499318,9cyclictest0-21swapper/100:39:171
527499318,9cyclictest0-21swapper/100:39:171
527499318,8cyclictest0-21swapper/121:32:081
527499318,11cyclictest16409-21ssh21:24:021
5274993125,3cyclictest29605-21apt-get19:35:111
5274993125,2cyclictest26731-21apt-get21:50:151
5274993110,7cyclictest20498-21cron22:45:011
529799301,16cyclictest25866irq/58-eth021:15:123
529799301,16cyclictest25866irq/58-eth021:15:123
5284993015,7cyclictest12712-21sshd00:23:312
5284993015,12cyclictest0-21swapper/222:05:132
5284993013,8cyclictest7270-21kthreadcore19:10:222
5284993013,13cyclictest5790-21cat19:40:342
5284993012,9cyclictest0-21swapper/222:45:122
5284993012,9cyclictest0-21swapper/222:45:122
5284993012,10cyclictest0-21swapper/200:30:152
5284993011,4cyclictest3009466rm21:11:162
5284993011,16cyclictest11954-21ssh23:40:092
5284993010,11cyclictest8813-21sh22:35:292
527499308,7cyclictest19518-21apt-get22:05:141
527499308,11cyclictest2364-21ssh23:13:201
527499308,10cyclictest8032-21cat20:15:311
5274993024,3cyclictest1394-21apt-get21:35:101
5274993018,9cyclictest0-21swapper/120:45:111
5297992915,10cyclictest25866irq/58-eth022:40:133
528499298,8cyclictest28203-21diskmemload22:29:372
528499293,21cyclictest0-21swapper/223:29:312
528499293,11cyclictest0-21swapper/200:07:312
5284992921,6cyclictest22146irq/40-dwc2_hso19:50:132
5284992914,8cyclictest14561-21sshd20:55:312
5284992913,8cyclictest395-21grep22:50:332
5284992913,11cyclictest24358-21diskmemload21:20:122
5284992911,15cyclictest26908-21rm00:10:322
5284992910,9cyclictest16676-21ssh21:43:252
527499299,7cyclictest8783-21sh21:57:211
527499298,6cyclictest0-21swapper/100:06:221
5274992921,5cyclictest5157-21apt-get20:50:141
5274992916,9cyclictest10320-21diskstats19:15:151
5274992911,6cyclictest0-21swapper/123:51:571
529799287,12cyclictest0-21swapper/322:55:103
529799282,16cyclictest25866irq/58-eth020:45:123
5297992816,8cyclictest0-21swapper/323:30:173
5284992814,11cyclictest24993-21ntp_states20:00:312
5284992812,8cyclictest19873-21rm23:04:432
5284992812,13cyclictest18122-21cat20:30:012
5284992812,12cyclictest26883-21tail20:35:272
5284992811,8cyclictest18751-21ssh22:42:072
5284992810,8cyclictest1-21systemd21:45:312
5284992810,10cyclictest7079-21fschecks_count19:45:162
527499289,7cyclictest1886-21ssh22:12:461
527499287,9cyclictest9124-21sh22:16:001
527499287,9cyclictest9124-21sh22:16:001
5274992817,9cyclictest21-21ksoftirqd/123:20:321
5274992817,8cyclictest24299-21sh22:25:261
5274992817,8cyclictest0-21swapper/119:10:121
5297992715,8cyclictest25866irq/58-eth023:05:133
528499278,16cyclictest21230-21sendmail-msp20:00:012
5284992714,9cyclictest26625-21kthreadcore19:30:192
5284992714,10cyclictest32696-21latency_hist20:45:012
5284992712,12cyclictest3158-21ssh23:14:322
5284992711,8cyclictest18664-21sshd23:23:192
5284992710,13cyclictest3915-21idleruntime20:15:172
527499277,10cyclictest676-21sh21:15:141
527499277,10cyclictest676-21sh21:15:141
527499276,10cyclictest20126-21cut19:25:161
5274992710,13cyclictest32554-21cron20:45:011
526299272,18cyclictest26776-21ssh00:33:310
526299271,17cyclictest12-21ksoftirqd/020:25:110
5297992614,7cyclictest0-21swapper/323:35:133
5297992610,10cyclictest0-21swapper/300:05:313
529799260,21cyclictest25866irq/58-eth023:25:123
528499269,13cyclictest14891-21sort21:00:142
528499267,10cyclictest8645-21grep20:20:162
5284992614,9cyclictest15041-21apt-get19:20:142
5284992614,9cyclictest0-21swapper/219:35:112
5274992618,5cyclictest24762-21apt-get19:30:121
5274992613,9cyclictest21798-21ssh23:47:581
5274992613,9cyclictest15354-21/usr/sbin/munin19:20:181
526299262,19cyclictest3925-21ssh22:15:140
526299262,19cyclictest3925-21ssh22:15:140
5297992514,8cyclictest0-21swapper/321:50:243
527499258,14cyclictest5548-21ntpq19:40:311
5274992516,6cyclictest23045-21apt-get20:35:121
5274992512,10cyclictest8310-21sendmail-msp20:20:191
526299252,17cyclictest19531-21ssh22:05:140
529799244,13cyclictest0-21swapper/323:40:123
5274992417,5cyclictest14837-21apt-get21:00:141
526299245,13cyclictest30004-1kworker/0:2H00:35:130
526299245,13cyclictest30004-1kworker/0:2H00:35:130
5262992411,6cyclictest0-21swapper/021:30:170
5262992410,9cyclictest629-21tcf-agent23:06:310
529799238,10cyclictest25866irq/58-eth022:15:133
529799238,10cyclictest25866irq/58-eth022:15:133
5297992312,7cyclictest0-21swapper/322:00:163
529799231,13cyclictest25866irq/58-eth022:05:143
5284992312,8cyclictest30924-21sshd20:06:322
5274992311,9cyclictest26324-21cat20:05:111
5274992310,9cyclictest21230-21sendmail-msp20:00:021
5262992313,5cyclictest0-21swapper/021:10:140
529799226,11cyclictest0-21swapper/300:10:313
5297992217,3cyclictest321rcuc/323:15:013
5297992214,5cyclictest0-21swapper/300:25:433
5297992213,6cyclictest0-21swapper/321:30:383
529799221,14cyclictest25866irq/58-eth021:25:143
526299221,18cyclictest28185-21find23:10:090
5297992115,4cyclictest0-21swapper/300:24:413
5297992113,6cyclictest0-21swapper/319:25:173
5297992113,5cyclictest0-21swapper/323:50:143
5297992113,5cyclictest0-21swapper/323:21:563
5297992112,6cyclictest0-21swapper/319:10:163
5297992112,6cyclictest0-21swapper/300:35:043
5297992112,6cyclictest0-21swapper/300:35:043
526299213,15cyclictest11676-21sendmail-msp23:40:160
526299212,4cyclictest12-21ksoftirqd/023:50:010
526299211,17cyclictest15024-21apt-get00:05:130
526299211,15cyclictest31576-21uniq23:55:150
529799202,11cyclictest0-21swapper/320:05:123
5297992015,3cyclictest0-21swapper/322:10:333
5297992014,4cyclictest321rcuc/322:37:463
5297992014,4cyclictest321rcuc/319:50:233
5297992014,4cyclictest0-21swapper/323:48:523
5297992014,4cyclictest0-21swapper/320:35:113
5297992014,4cyclictest0-21swapper/320:20:123
5297992013,5cyclictest0-21swapper/321:44:323
5297992013,5cyclictest0-21swapper/321:22:313
5297992013,5cyclictest0-21swapper/320:55:283
5297992012,6cyclictest0-21swapper/322:20:433
527499208,9cyclictest5281-21sed19:10:021
526299205,9cyclictest12-21ksoftirqd/020:55:110
526299202,15cyclictest10734-21sendmail-msp22:00:300
526299202,13cyclictest3041-21ssh22:53:310
526299201,13cyclictest845-21munin-node19:45:140
5262992010,7cyclictest26331-21apt-get20:05:120
529799199,7cyclictest0-21swapper/323:00:143
529799192,11cyclictest0-21swapper/319:55:123
529799192,10cyclictest0-21swapper/320:10:143
5297991913,4cyclictest321rcuc/322:27:023
5297991913,4cyclictest0-21swapper/321:10:333
5297991913,4cyclictest0-21swapper/320:42:523
5297991913,4cyclictest0-21swapper/319:15:163
5297991913,4cyclictest0-21swapper/300:16:573
5297991912,5cyclictest0-21swapper/323:55:133
5297991912,5cyclictest0-21swapper/323:15:133
5297991912,4cyclictest0-21swapper/321:05:163
5297991912,4cyclictest0-21swapper/321:05:163
5297991911,6cyclictest0-21swapper/322:31:393
526299198,8cyclictest12-21ksoftirqd/020:50:140
526299194,3cyclictest131rcu_preempt23:30:310
526299191,9cyclictest27183-21grep23:50:240
526299191,14cyclictest28917-21ssh22:30:170
5262991910,7cyclictest3362-21cron20:15:010
529799182,10cyclictest0-21swapper/320:15:043
529799182,10cyclictest0-21swapper/319:30:133
5297991813,3cyclictest0-21swapper/319:45:313
5297991812,4cyclictest0-21swapper/320:25:183
5297991812,4cyclictest0-21swapper/319:40:203
5297991812,4cyclictest0-21swapper/319:35:163
526299186,3cyclictest131rcu_preempt19:10:220
526299183,9cyclictest27946-21ssh22:10:150
526299182,11cyclictest0-21swapper/021:05:140
526299182,11cyclictest0-21swapper/021:05:140
526299181,9cyclictest4621-21kthreadcore21:55:210
526299181,12cyclictest3843-21ssh23:15:130
529799172,10cyclictest0-21swapper/320:50:143
5297991711,4cyclictest0-21swapper/321:00:183
5297991711,4cyclictest0-21swapper/320:30:163
526299173,2cyclictest131rcu_preempt19:15:310
526299173,11cyclictest19749-21kthreadcore21:45:180
526299172,13cyclictest1782-21kthreadcore20:45:190
5262991711,4cyclictest3369-21apt-get20:15:040
526299171,14cyclictest22431-21kthreadcore22:25:200
526299171,13cyclictest1774-21munin-run19:40:010
526299171,12cyclictest17657-21tune2fs21:25:170
5297991610,4cyclictest0-21swapper/320:00:313
5297991610,4cyclictest0-21swapper/319:20:113
526299161,10cyclictest12-21ksoftirqd/019:30:120
528499151,6cyclictest5330-21latency_hist19:10:022
526299131,9cyclictest5332-21latency_hist19:10:020
529799101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional