You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-11 - 22:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Fri Apr 11, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
824222470,2sleep30-21swapper/307:05:203
3071221690,5sleep0993799cyclictest08:05:290
1728221420,2sleep00-21swapper/009:53:220
1728221420,2sleep00-21swapper/009:53:220
99379910126,59cyclictest22503-21aten_rbpower_po11:40:190
99379910125,61cyclictest31275-21idleruntime-cro12:35:050
9937999937,37cyclictest0-21swapper/010:00:160
9937999626,56cyclictest15135-21munin-plugin-st07:25:040
317082960,1sleep00-21swapper/011:52:220
9937999525,52cyclictest15093-21latency_hist10:40:050
9937999225,52cyclictest23841-21unixbench_singl07:45:370
9937999225,52cyclictest23841-21unixbench_singl07:45:370
9937998933,41cyclictest0-21swapper/009:15:110
9937998933,41cyclictest0-21swapper/009:15:110
9937998837,11cyclictest18228-21apt-get11:35:110
9937998729,44cyclictest15651-21basename08:55:180
9937998725,50cyclictest20253-21runrttasks09:57:060
9937998725,44cyclictest17778-21grep07:30:290
9937998627,44cyclictest1188-21egrep08:15:200
9937998627,40cyclictest16016-21/usr/sbin/munin10:40:280
9937998623,46cyclictest26716-21/usr/sbin/munin07:55:240
9937998425,46cyclictest22610-21ssh10:51:070
9937998424,55cyclictest17004-21grep12:15:250
9937998326,47cyclictest8374-21taskset08:35:080
9937998324,45cyclictest0-21swapper/009:40:250
9937998234,34cyclictest28206-21ssh11:48:290
9937998234,34cyclictest28206-21ssh11:48:290
9937998223,43cyclictest0-21swapper/012:30:050
9937998222,47cyclictest0-21swapper/010:05:280
9937998132,36cyclictest10404-21sh11:22:050
9937998126,43cyclictest14201-21aten_rbpower_po11:30:190
9937998121,48cyclictest0-21swapper/012:14:080
9937998118,49cyclictest23932-21sh12:24:320
9937998025,44cyclictest6373-21taskset10:26:530
9937998025,43cyclictest19913-21ssh10:48:040
9937998025,40cyclictest27204-21seq10:58:430
9937998024,40cyclictest885-21ssh09:29:160
9937998023,47cyclictest13345-21munin-run07:20:050
9937998023,47cyclictest13345-21munin-run07:20:050
9937998021,47cyclictest25850-21chrt07:52:370
9937997926,44cyclictest0-21swapper/011:56:130
9937997926,41cyclictest5269-21rm10:24:130
9937997923,44cyclictest12784-21ssh11:27:380
9937997921,48cyclictest0-21swapper/009:32:090
9937997825,44cyclictest6819-21taskset12:01:100
9937997822,42cyclictest0-21swapper/011:06:330
9937997822,41cyclictest0-21swapper/011:03:580
9937997820,50cyclictest0-21swapper/010:18:560
9937997729,40cyclictest10035-21ssh12:06:390
9937997727,41cyclictest31932-21ssh12:35:240
9937997724,39cyclictest12723-21ssh09:46:020
9937997721,47cyclictest791-21taskset08:14:590
9937997622,45cyclictest30285-21ssh10:13:360
9937997621,42cyclictest0-21swapper/010:32:130
9937997621,39cyclictest0-21swapper/009:06:410
296832760,2sleep20-21swapper/209:21:232
9937997526,38cyclictest28601-21ssh09:20:270
9937997525,35cyclictest0-21swapper/011:16:550
9937997522,39cyclictest0-21swapper/011:11:440
9937997421,39cyclictest0-21swapper/008:03:380
9937997420,40cyclictest0-21swapper/009:12:200
9937997324,39cyclictest0-21swapper/007:36:010
9937997316,41cyclictest0-21swapper/009:38:020
9937997228,41cyclictest15740-21basename07:25:230
9937997221,37cyclictest0-21swapper/008:32:150
9937997121,42cyclictest18639-21runrttasks09:00:580
9937997121,42cyclictest18639-21runrttasks09:00:580
9937997121,38cyclictest0-21swapper/008:20:420
9937997118,38cyclictest0-21swapper/007:41:550
9937997116,41cyclictest0-21swapper/008:43:520
198352710,1sleep20-21swapper/209:56:092
9937997020,37cyclictest11229-21chrt07:11:320
65682700,2sleep20-21swapper/211:16:092
9937996922,35cyclictest0-21swapper/008:52:280
9937996921,37cyclictest5950-21taskset08:26:280
9937996721,35cyclictest0-21swapper/008:49:350
80652670,1sleep30-21swapper/309:38:553
993899658,25cyclictest0-21swapper/108:35:101
38962590,5sleep2993999cyclictest09:32:342
286872580,3sleep337-21rcuc/311:49:233
286872580,3sleep337-21rcuc/311:49:233
179852580,5sleep1993899cyclictest12:16:101
9939995412,13cyclictest9936-21cyclictest12:15:162
994099527,26cyclictest0-21swapper/308:35:103
9939995210,20cyclictest8050irq/56-dwc_otg_08:35:182
993999505,18cyclictest7850irq/56-dwc_otg12:10:232
9938994921,5cyclictest6530-21aten_rbpower_cu08:30:221
994099481,13cyclictest31553-21uniq10:15:253
9939994821,8cyclictest13579-21apt-get07:20:132
9939994812,11cyclictest21904-21ssh10:00:182
993999478,13cyclictest906-21ssh11:09:122
993999477,22cyclictest8050irq/56-dwc_otg_09:40:352
993999472,7cyclictest8050irq/56-dwc_otg_07:25:162
302522470,2sleep10-21swapper/111:02:351
264492470,2sleep10-21swapper/112:28:461
9939994616,15cyclictest1446-21/usr/sbin/munin08:15:232
9939994611,16cyclictest32423-21taskset11:53:422
993999455,24cyclictest7950irq/56-dwc_otg_08:15:052
993999454,26cyclictest0-21swapper/212:24:382
993899451,3cyclictest7850irq/56-dwc_otg12:35:241
9938994513,8cyclictest24978-21fschecks_count10:55:231
94422453,35sleep00-21swapper/007:07:160
9939994419,13cyclictest18088-21ls07:30:342
9939994412,18cyclictest14363-21ssh11:30:222
9939994411,12cyclictest7756-21ssh10:29:562
9939994410,12cyclictest15217-21ssh10:40:112
993899446,22cyclictest0-21swapper/110:00:171
993999439,23cyclictest30320-21ssh10:13:402
9939994319,6cyclictest25874-21grep10:55:342
9938994323,6cyclictest23-21ksoftirqd/107:55:341
993999429,13cyclictest31520-21ssh12:35:092
993999426,22cyclictest0-21swapper/208:20:192
993999423,29cyclictest8050irq/56-dwc_otg_11:25:062
993999423,22cyclictest0-21swapper/208:50:192
9939994217,11cyclictest301ktimersoftd/208:55:222
993999421,5cyclictest19272-21switchtime11:35:322
9939994211,11cyclictest28056-21ssh11:48:132
9939994211,11cyclictest28056-21ssh11:48:132
993899426,5cyclictest0-21swapper/107:35:171
993899422,21cyclictest0-21swapper/111:43:071
994099411,25cyclictest0-21swapper/308:10:223
994099411,17cyclictest0-21swapper/310:15:053
9939994119,4cyclictest5464-21fschecks_count12:00:232
9939994115,16cyclictest301ktimersoftd/210:37:342
9939994114,17cyclictest12942-21ssh11:27:592
9939994112,18cyclictest26990-21ntp_states07:55:282
9938994116,6cyclictest0-21swapper/111:55:351
199952410,1sleep20-21swapper/207:35:332
994099405,26cyclictest8050irq/56-dwc_otg_07:35:173
994099401,24cyclictest0-21swapper/307:55:193
993999408,7cyclictest10272-21ssh10:32:352
993999405,11cyclictest25611-21ssh11:44:072
9939994012,12cyclictest23993-21taskset07:47:042
9939994012,12cyclictest23993-21taskset07:47:042
9939994010,12cyclictest16625-21ssh09:51:542
9939994010,12cyclictest16625-21ssh09:51:542
9938994021,5cyclictest5020-21sh11:15:171
9938994014,7cyclictest15489-21ntpq09:50:241
9938994014,7cyclictest15489-21ntpq09:50:241
173982400,2sleep10-21swapper/107:30:241
994099392,22cyclictest0-21swapper/312:20:053
994099391,23cyclictest0-21swapper/308:40:203
993999399,21cyclictest25265-21ssh12:26:352
9939993910,18cyclictest24498-21apt-get07:50:192
993899392,22cyclictest8050irq/56-dwc_otg_09:35:241
9938993913,6cyclictest23-21ksoftirqd/109:15:041
258452390,2sleep30-21swapper/310:06:133
994099386,16cyclictest0-21swapper/307:50:223
9940993818,11cyclictest2895-21unixbench_singl11:10:363
9940993815,5cyclictest14828-21sh12:12:563
994099381,30cyclictest0-21swapper/309:16:093
994099381,30cyclictest0-21swapper/309:16:093
994099381,29cyclictest1482-21ls09:30:053
993999389,20cyclictest26561-21ssh10:07:472
993999389,20cyclictest0-21swapper/209:15:372
993999389,20cyclictest0-21swapper/209:15:372
993999384,13cyclictest8050irq/56-dwc_otg_12:30:122
993999383,12cyclictest13388-21date07:20:052
993999383,12cyclictest13388-21date07:20:052
9939993812,18cyclictest10203-21ssh12:06:562
9939993810,7cyclictest5296-21apt-get09:35:182
993899382,24cyclictest7850irq/56-dwc_otg11:35:081
994099374,23cyclictest7383-21ls08:30:373
994099372,14cyclictest15118-21uniq09:50:193
994099372,14cyclictest15118-21uniq09:50:193
9940993714,4cyclictest1362-21fschecks_time08:15:223
994099371,30cyclictest31798-21ls12:35:213
994099371,22cyclictest0-21swapper/309:00:153
994099371,22cyclictest0-21swapper/309:00:153
993999379,19cyclictest19931-21taskset10:48:072
993999377,4cyclictest7950irq/56-dwc_otg_11:11:502
993999376,11cyclictest0-21swapper/209:26:322
993999372,10cyclictest8050irq/56-dwc_otg_11:56:212
993899371,7cyclictest21-21rcuc/107:25:221
993899371,4cyclictest8050irq/56-dwc_otg_07:20:131
9940993625,5cyclictest24257-21ssh10:03:093
994099362,16cyclictest0-21swapper/311:54:333
9940993616,5cyclictest24129-21idleruntime-cro09:15:043
9940993616,12cyclictest22511-21latency_hist07:45:053
993999368,10cyclictest11468-21seq08:41:242
993999363,12cyclictest14658-21latency_hist09:50:062
9939993619,8cyclictest7950irq/56-dwc_otg_10:53:312
9939993611,16cyclictest402-21ssh10:16:172
993899364,14cyclictest8050irq/56-dwc_otg_12:30:141
993899363,3cyclictest0-21swapper/108:05:161
993899363,18cyclictest7950irq/56-dwc_otg_10:00:041
9938993617,7cyclictest14217-21aten_rbpower_po11:30:191
9938993613,9cyclictest221ktimersoftd/109:15:181
9938993613,9cyclictest221ktimersoftd/109:15:181
994099357,19cyclictest22395-21sh11:40:163
994099351,23cyclictest0-21swapper/310:35:343
994099351,22cyclictest0-21swapper/311:35:053
993999358,4cyclictest8050irq/56-dwc_otg_11:01:312
993999357,24cyclictest0-21swapper/208:45:212
993999352,19cyclictest8050irq/56-dwc_otg_08:05:262
993899358,15cyclictest0-21swapper/110:55:061
993899354,5cyclictest7850irq/56-dwc_otg08:25:171
993899354,22cyclictest0-21swapper/109:46:481
993899351,9cyclictest17727-21fschecks_time09:00:191
993899351,9cyclictest17727-21fschecks_time09:00:191
9938993519,5cyclictest221ktimersoftd/108:15:231
9938993518,11cyclictest7850irq/56-dwc_otg07:20:051
9938993518,11cyclictest7850irq/56-dwc_otg07:20:051
993899351,3cyclictest0-21swapper/112:00:371
993899351,24cyclictest24436-21ls07:50:061
993899351,24cyclictest24436-21ls07:50:061
219002350,2sleep20-21swapper/209:10:322
994099349,6cyclictest29142-21cat09:20:343
994099349,16cyclictest0-21swapper/307:50:053
994099349,16cyclictest0-21swapper/307:50:053
994099348,4cyclictest0-21swapper/310:40:243
994099344,20cyclictest11366-21munin-run10:35:043
994099342,5cyclictest19390-21ssh11:35:333
9940993416,11cyclictest13766-21ssh09:48:383
9940993414,5cyclictest13566-21/usr/sbin/munin07:20:203
9939993412,15cyclictest5952-21taskset08:26:302
9939993411,15cyclictest301ktimersoftd/209:06:442
9939993410,3cyclictest0-21swapper/210:21:402
993899349,10cyclictest23-21ksoftirqd/110:35:041
993899347,7cyclictest8050irq/56-dwc_otg_07:50:221
993899345,19cyclictest0-21swapper/111:10:371
9938993422,4cyclictest7850irq/56-dwc_otg11:53:151
9938993418,6cyclictest23-21ksoftirqd/108:55:321
9938993415,13cyclictest7850irq/56-dwc_otg08:10:201
993899341,29cyclictest3087-21ls08:20:211
9938993412,4cyclictest0-21swapper/110:45:451
993899341,23cyclictest14331-21latency08:50:281
993899341,22cyclictest20785-21cut12:20:251
22392340,4sleep17950irq/56-dwc_otg_09:30:281
994099334,8cyclictest0-21swapper/312:00:283
994099332,18cyclictest17240-21grep07:30:223
994099331,22cyclictest27785-21uniq12:30:183
994099331,22cyclictest0-21swapper/309:30:103
994099331,21cyclictest0-21swapper/310:50:143
994099331,14cyclictest0-21swapper/310:24:273
993999336,15cyclictest0-21swapper/208:00:162
993999335,4cyclictest6488-21/usr/sbin/munin08:30:352
993899337,16cyclictest0-21swapper/111:09:201
993899334,4cyclictest7950irq/56-dwc_otg_10:35:361
9938993317,5cyclictest27012-21sh11:46:161
9938993317,5cyclictest27012-21sh11:46:161
993899331,17cyclictest8050irq/56-dwc_otg_10:10:221
993899331,12cyclictest16528-21head10:40:341
182482331,4sleep318263-21apt-get09:55:153
994099326,3cyclictest0-21swapper/311:09:203
994099321,24cyclictest0-21swapper/308:46:013
994099321,14cyclictest14884-21unixbench-2d08:50:363
9940993210,12cyclictest0-21swapper/311:28:533
993999329,14cyclictest22302-21chrt07:44:422
993899329,5cyclictest7950irq/56-dwc_otg_12:10:051
993899326,7cyclictest12545-21grep08:45:271
9938993223,6cyclictest19271-21wget09:05:181
993899322,13cyclictest7850irq/56-dwc_otg11:30:051
9938993220,4cyclictest8050irq/56-dwc_otg_09:22:241
9938993211,5cyclictest0-21swapper/112:10:271
993899321,14cyclictest7850irq/56-dwc_otg09:40:251
964523219,6sleep20-21swapper/207:09:492
994099316,5cyclictest37-21rcuc/307:25:193
994099314,10cyclictest7850irq/56-dwc_otg11:00:333
994099313,17cyclictest31059-21sed08:05:333
994099311,21cyclictest8717-21cut11:20:233
994099311,14cyclictest8050irq/56-dwc_otg_07:15:343
994099311,14cyclictest8050irq/56-dwc_otg_07:15:343
993999314,23cyclictest0-21swapper/207:10:202
993999312,6cyclictest0-21swapper/209:00:152
993999312,6cyclictest0-21swapper/209:00:152
993899317,9cyclictest8050irq/56-dwc_otg_10:20:271
993899315,8cyclictest23-21ksoftirqd/108:01:091
993899315,6cyclictest7950irq/56-dwc_otg_11:20:181
993899314,16cyclictest0-21swapper/110:26:341
993899312,13cyclictest9550irq/48-DMA08:47:301
116962310,2sleep30-21swapper/312:09:443
994099306,14cyclictest0-21swapper/311:55:213
994099301,4cyclictest0-21swapper/308:20:213
994099301,23cyclictest0-21swapper/308:56:063
994099301,22cyclictest0-21swapper/312:20:363
994099301,21cyclictest0-21swapper/312:27:213
994099301,21cyclictest0-21swapper/309:40:413
994099301,20cyclictest0-21swapper/311:18:523
994099301,19cyclictest5536-21switchtime08:25:313
994099301,11cyclictest0-21swapper/310:45:063
993899307,4cyclictest31576-21fschecks_count10:15:251
993899305,15cyclictest0-21swapper/109:28:381
9938993013,5cyclictest10008-21basename07:10:221
994099297,5cyclictest7850irq/56-dwc_otg10:55:253
9940992912,11cyclictest0-21swapper/310:26:163
9938992910,9cyclictest7850irq/56-dwc_otg10:05:501
994099284,7cyclictest9968-21apt-get07:10:203
994099281,14cyclictest19734-21latency09:05:243
9938992811,5cyclictest20979-21cpuspeed07:40:201
994099272,16cyclictest0-21swapper/308:00:143
93232274,15sleep10-21swapper/107:05:431
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional