You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-23 - 20:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Nov 23, 2024 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30447210086,9sleep10-21swapper/107:05:381
3112629986,9sleep30-21swapper/307:07:133
3120228766,6sleep20-21swapper/207:08:132
3131528364,14sleep00-21swapper/007:09:390
243702260,1sleep217343-21kworker/2:112:01:152
3142999244,8cyclictest39-21ksoftirqd/311:51:593
31429992322,1cyclictest0-21swapper/307:38:463
31429992217,4cyclictest39-21ksoftirqd/309:50:023
3142199220,22cyclictest0-21swapper/111:43:301
3141699217,7cyclictest11866-21perf08:45:000
31416992118,2cyclictest0-21swapper/011:34:350
128572210,1chrt161rcuc/009:26:550
48192200,0chrt0-21swapper/009:20:170
3142599201,18cyclictest12268-21ntp_kernel_pll_07:35:212
3142599200,2cyclictest0-21swapper/212:05:132
3142599200,20cyclictest0-21swapper/211:15:002
3142599200,20cyclictest0-21swapper/208:05:122
3141699200,2cyclictest141rcu_preempt11:59:500
64882190,0sleep00-21swapper/012:10:230
3142999198,2cyclictest39-21ksoftirqd/311:35:283
3142999196,12cyclictest39-21ksoftirqd/311:15:123
3142999196,12cyclictest39-21ksoftirqd/311:15:123
3142999190,18cyclictest0-21swapper/309:25:053
31425991915,4cyclictest0-21swapper/212:21:312
3141699192,2cyclictest141rcu_preempt09:32:370
31429991817,1cyclictest15497-21apache209:10:253
31429991811,7cyclictest39-21ksoftirqd/310:25:143
3142999180,1cyclictest0-21swapper/311:57:313
31416991814,3cyclictest13-21ksoftirqd/008:40:000
3141699180,12cyclictest0-21swapper/010:55:050
31429991714,2cyclictest7744-21snmpd12:05:153
3142199176,8cyclictest5358-21grep10:15:181
3141699176,5cyclictest21640-21munin-run09:05:000
3142999165,10cyclictest0-21swapper/311:47:283
31429991612,3cyclictest39-21ksoftirqd/310:55:213
31429991612,2cyclictest0-21swapper/309:24:023
31429991611,3cyclictest39-21ksoftirqd/311:30:113
3142599163,2cyclictest141rcu_preempt12:37:202
3142599160,2cyclictest0-21swapper/209:23:352
3142599160,1cyclictest0-21swapper/210:46:412
31416991616,0cyclictest0-21swapper/010:30:030
3142999159,5cyclictest5011-21/usr/sbin/munin12:30:203
3142999158,7cyclictest0-21swapper/311:10:153
31429991513,1cyclictest0-21swapper/311:00:193
31429991512,3cyclictest39-21ksoftirqd/310:05:223
31429991511,2cyclictest589-21grep10:10:323
31429991510,4cyclictest24011-21diskmemload12:18:323
31425991513,2cyclictest32-21ksoftirqd/209:32:452
3142599150,14cyclictest0-21swapper/211:16:542
3142599150,14cyclictest0-21swapper/211:16:542
3142199150,13cyclictest0-21swapper/112:37:471
3141699151,6cyclictest2693-21cut08:25:140
11062150,1chrt161rcuc/007:12:110
31429991413,0cyclictest24011-21diskmemload10:15:273
3142999141,1cyclictest141rcu_preempt10:00:123
31429991410,3cyclictest1837-21grep08:20:263
3142599141,8cyclictest6609-21/usr/sbin/munin11:50:262
31425991411,2cyclictest22764-21munin-plugin-st10:30:002
3142599140,1cyclictest0-21swapper/210:43:302
3142599140,14cyclictest0-21swapper/211:48:112
3142599140,14cyclictest0-21swapper/209:16:202
3142599140,10cyclictest0-21swapper/210:13:482
31421991410,3cyclictest13794-21rm11:55:161
3142199140,1cyclictest0-21swapper/111:48:591
3142199140,1cyclictest0-21swapper/109:54:421
3141699143,4cyclictest224-21systemd-journal09:17:300
31416991410,3cyclictest4323-21sleep10:15:000
3142999139,3cyclictest30470-21grep08:15:163
3142999139,3cyclictest16897-21wc09:30:313
3142999138,4cyclictest39-21ksoftirqd/308:55:003
3142999137,6cyclictest0-21swapper/307:30:153
3142999135,8cyclictest0-21swapper/310:31:033
3142999131,4cyclictest39-21ksoftirqd/312:23:273
31429991312,1cyclictest39-21ksoftirqd/312:37:313
31429991312,1cyclictest39-21ksoftirqd/311:42:203
31429991311,1cyclictest0-21swapper/312:00:153
31429991310,2cyclictest24621-21wc08:00:273
31429991310,2cyclictest24011-21diskmemload10:24:303
3142999130,9cyclictest39-21ksoftirqd/311:24:263
3142599133,9cyclictest37850irq/131-snd_hda09:25:122
3142599131,4cyclictest224-21systemd-journal08:44:302
31425991310,2cyclictest440-21grep08:20:142
31425991310,2cyclictest30930-21sed10:35:272
31425991310,2cyclictest22384-21grep11:40:192
31425991310,2cyclictest19651-21grep10:00:152
31425991310,2cyclictest14050-21grep12:15:192
31425991310,1cyclictest0-21swapper/209:12:212
3142599130,6cyclictest224-21systemd-journal11:39:552
3142599130,12cyclictest22809-21awk09:35:192
3142599130,11cyclictest4250-21tr07:20:152
3142199139,3cyclictest32557-21tail09:15:271
3142199139,3cyclictest10495-21tr11:10:181
3142199136,1cyclictest24011-21diskmemload11:35:181
3142199134,8cyclictest15351-21/usr/sbin/munin09:30:231
3142199132,10cyclictest12725-21ls12:15:001
31421991310,2cyclictest556-21sed10:10:331
31421991310,2cyclictest3320-21head07:15:271
31421991310,2cyclictest30185-21grep12:05:181
31421991310,1cyclictest14266-21grep07:40:171
3142199130,1cyclictest0-21swapper/110:07:081
3141699136,5cyclictest0-21swapper/011:20:130
3141699135,3cyclictest29496-21apt-get12:05:160
3141699133,9cyclictest5943-21ntp_states10:15:230
31416991312,1cyclictest0-21swapper/009:11:240
31416991310,2cyclictest4471-21ssh11:05:210
31416991310,2cyclictest3241-21sed07:15:270
31416991310,2cyclictest26271-21cut08:05:160
31416991310,2cyclictest24215-21head08:00:210
31416991310,2cyclictest23380-21wc09:05:250
31416991310,2cyclictest18822-21grep08:55:260
31416991310,2cyclictest12526-21wc10:20:270
3142999129,2cyclictest9954-21tr08:40:133
3142999129,2cyclictest5923-21head12:10:183
3142999129,2cyclictest32611-21head09:15:273
3142999129,2cyclictest31596-21chrt12:25:373
3142999129,2cyclictest21148-21sed09:00:273
3142999129,2cyclictest12774-21cut08:45:153
3142999129,2cyclictest10066-21tail10:45:273
3142999128,3cyclictest28633-21sed09:40:193
3142999128,3cyclictest28001-21cut08:10:143
3142999126,2cyclictest224-21systemd-journal07:12:213
3142999125,3cyclictest224-21systemd-journal11:06:033
3142999124,8cyclictest0-21swapper/310:40:273
3142999124,4cyclictest39-21ksoftirqd/309:08:203
3142999121,5cyclictest224-21systemd-journal07:54:213
31429991210,1cyclictest7064-21cron08:35:013
3142999120,8cyclictest39-21ksoftirqd/310:52:313
3142599129,2cyclictest6347-21tail12:10:212
3142599129,2cyclictest4498-21ls11:05:212
3142599129,2cyclictest28134-21tr08:10:152
3142599129,2cyclictest24256-21grep11:20:152
3142599129,2cyclictest23170-21grep10:55:222
3142599129,2cyclictest2119-21sed09:45:202
3142599129,2cyclictest21004-21wc09:00:262
3142599129,2cyclictest18910-21cron10:00:002
3142599129,2cyclictest18146-21wc07:50:002
3142599128,3cyclictest4297-21head08:25:252
3142599128,3cyclictest31018-21grep08:15:182
3142599128,3cyclictest2330-21grep07:15:192
3142599128,2cyclictest0-21swapper/208:45:142
31425991210,2cyclictest32-21ksoftirqd/212:31:302
31425991210,1cyclictest8955-21grep08:35:252
3142199129,2cyclictest8959-21grep08:35:251
3142199129,2cyclictest7847-21basename10:45:001
3142199129,2cyclictest5695-21head11:30:141
3142199129,2cyclictest4529-21sh11:30:001
3142199129,2cyclictest4227-21cat11:05:191
3142199129,2cyclictest32257-21taskset12:26:091
3142199129,2cyclictest23230-21grep12:20:271
3142199129,2cyclictest23137-21cat12:00:271
3142199129,2cyclictest22496-21grep09:05:171
3142199129,2cyclictest17918-21uniq10:25:181
3142199129,2cyclictest10932-21ssh10:20:141
3142199128,3cyclictest24788-21ls10:30:301
3142199124,6cyclictest18306-21/usr/sbin/munin07:50:131
3142199121,8cyclictest6291-21ls11:50:221
3141699129,2cyclictest8491-21cron07:30:000
3141699129,2cyclictest31013-21cut08:15:180
3141699129,2cyclictest2344-21basename08:24:590
3141699129,2cyclictest16853-21cron08:55:010
3141699128,3cyclictest27015-21sh11:43:340
3141699128,3cyclictest17427-21grep11:15:160
3141699128,3cyclictest17427-21grep11:15:150
3141699128,2cyclictest31731-21ssh11:25:500
3141699125,4cyclictest13218-21cut12:35:160
3141699124,7cyclictest12945-21idleruntime-cro09:55:000
3141699124,2cyclictest224-21systemd-journal12:30:000
3141699123,7cyclictest1291-21runrttasks10:52:410
31416991210,1cyclictest18406-21grep07:50:110
3141699120,11cyclictest26090-21grep10:05:180
3142999118,2cyclictest7587-21cut07:25:213
3142999118,2cyclictest4740-21grep07:20:183
3142999118,2cyclictest3715-21chrt09:46:083
3142999118,2cyclictest3315-21head07:15:273
3142999118,2cyclictest26744-21wc08:05:223
3142999118,2cyclictest24057-21ssh09:36:053
3142999118,2cyclictest21784-21sed07:55:193
3142999118,2cyclictest15373-21ssh09:55:323
3142999117,3cyclictest17028-21wc07:45:213
3142999116,4cyclictest8845-21wc08:35:243
3142999114,5cyclictest0-21swapper/311:25:203
3142999113,8cyclictest0-21swapper/310:35:273
31429991110,1cyclictest0-21swapper/308:55:143
3142999110,10cyclictest3936-21cut08:25:213
3142999110,10cyclictest14099-21sed07:40:163
3142599119,1cyclictest28512-21grep09:40:182
3142599118,2cyclictest9367-21grep07:30:182
3142599118,2cyclictest7116-21grep07:25:162
3142599118,2cyclictest6755-21sed11:30:232
3142599118,2cyclictest31729-21cut07:10:132
3142599118,2cyclictest31173-21sh11:25:212
3142599118,2cyclictest30135-21grep12:25:172
3142599118,2cyclictest28968-21tr11:00:152
3142599118,2cyclictest25554-21ssh10:31:012
3142599118,2cyclictest25291-21ssh10:05:122
3142599118,2cyclictest24036-21basename09:10:012
3142599118,2cyclictest15361-21ssh11:55:352
3142599118,2cyclictest11082-21cut10:20:152
3142599118,2cyclictest10177-21ssh09:51:472
3142599117,3cyclictest6472-21cat08:30:272
3142599117,2cyclictest22991-21cat08:00:002
3142199118,2cyclictest893-21sed07:10:261
3142199118,2cyclictest8507-21sh07:30:011
3142199118,2cyclictest5908-21taskset09:20:271
3142199118,2cyclictest3414-21basename08:25:161
3142199118,2cyclictest32119-21rm11:01:591
3142199118,2cyclictest30530-21ls11:25:001
3142199118,2cyclictest28048-21grep09:40:151
3142199118,2cyclictest24664-21cron10:04:591
3142199118,2cyclictest23374-21tr08:00:161
3142199118,2cyclictest22206-21ls10:55:151
3142199118,2cyclictest21736-21ssh09:35:131
3142199118,2cyclictest21407-21taskset09:02:171
3142199118,2cyclictest20390-21ssh11:16:491
3142199118,2cyclictest20390-21ssh11:16:491
3142199118,2cyclictest19143-21rm10:52:471
3142199118,2cyclictest17664-21grep07:45:291
3142199118,2cyclictest15897-21awk08:50:221
3142199118,2cyclictest1522-21awk09:45:171
3142199118,2cyclictest14447-21cat12:15:221
3142199118,2cyclictest13807-21expr09:55:151
3141699119,1cyclictest8562-21tr10:45:150
3141699118,2cyclictest4754-21tail07:20:180
3141699118,2cyclictest30055-21cut11:45:180
3141699118,2cyclictest2959-21wget10:40:140
3141699118,2cyclictest26826-21sh12:23:200
3141699118,2cyclictest22385-21head09:35:160
3141699118,2cyclictest21802-21grep07:55:190
3141699118,2cyclictest21672-21grep12:00:160
3141699118,2cyclictest19465-21ssh10:25:480
3141699118,2cyclictest18799-21ssh12:18:210
3141699118,2cyclictest17784-21ssh11:37:380
3141699118,2cyclictest16168-21sort07:45:160
3141699118,2cyclictest15602-21ssh09:55:440
3141699118,2cyclictest12222-21sh11:54:470
3141699117,2cyclictest29557-21head11:00:170
3141699116,5cyclictest1462-21munin-node08:34:450
3141699115,3cyclictest5023-21ssh12:30:150
3141699110,10cyclictest156872sleep011:14:020
3142599108,1cyclictest25312-21cut08:05:002
3142599108,1cyclictest15003-21cat08:50:162
3142599107,2cyclictest4591-21ssh10:15:062
3142599107,2cyclictest18507-21grep08:55:222
3142599107,2cyclictest16793-21grep10:50:292
3142599107,2cyclictest13862-21expr07:40:142
3142199108,1cyclictest67772sleep108:31:211
3142199108,1cyclictest32675-21tr08:20:131
3142199108,1cyclictest320892sleep108:16:481
3142199108,1cyclictest313732sleep110:35:471
3142199108,1cyclictest296112sleep108:10:301
3142199108,1cyclictest13949-21taskset08:46:421
3142199108,1cyclictest105312sleep107:30:431
3142199108,1cyclictest10158-21expr08:40:151
3142199107,2cyclictest4791-21grep07:20:181
3142199107,2cyclictest25952-21cat08:05:141
3142199107,2cyclictest24891-21rm09:10:191
3142199107,2cyclictest19258-21perf09:00:001
3142199107,2cyclictest11871-21grep07:35:161
3142199107,2cyclictest10270-21ssh10:45:341
3142199100,9cyclictest5362-21expr12:30:181
3142199100,9cyclictest10694-21basename09:25:171
3141699108,1cyclictest12791-21sed07:35:280
3141699107,2cyclictest3586-21ssh09:45:580
3141699107,2cyclictest29987-21ssh09:40:510
3141699107,2cyclictest29221-21ssh10:35:140
3141699107,2cyclictest14212-21cut07:40:170
3141699107,2cyclictest12702-21fschecks_count08:45:150
3141699107,2cyclictest10607-21chrt07:31:420
3141699105,4cyclictest21181-21grep10:00:330
314259997,1cyclictest19748-21switchtime07:50:232
314219998,1cyclictest21056-21expr07:55:141
314169996,2cyclictest28631-21grep08:10:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional