You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Thu Apr 03, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2658825036,0sleep00-21swapper/018:44:240
291350440,0irq/25-eth10-21swapper/318:46:343
2683223425,0sleep20-21swapper/218:47:142
2683123228,0sleep10-21swapper/118:47:131
269909976,0cyclictest27466-21tr20:23:331
35160,0ktimersoftd/30-21swapper/319:58:323
339960,0migration/331488-21sh22:05:573
339960,0migration/33091-21kthreadcore20:53:323
339960,0migration/324357-21sh22:24:153
339960,0migration/321305-21sh21:55:073
339960,0migration/319001-21munin-node21:28:323
339960,0migration/311229-21sh23:04:083
28415060,0irq/27-eth0-tx-5053-21ssh21:44:142
28415060,0irq/27-eth0-tx-3370-21munin-node19:43:280
28415060,0irq/27-eth0-tx-16080-21ssh21:00:292
28415060,0irq/27-eth0-tx-0-21swapper/223:55:382
28415060,0irq/27-eth0-tx-0-21swapper/223:36:052
28415060,0irq/27-eth0-tx-0-21swapper/223:25:112
28415060,0irq/27-eth0-tx-0-21swapper/223:17:552
28415060,0irq/27-eth0-tx-0-21swapper/223:10:192
28415060,0irq/27-eth0-tx-0-21swapper/223:03:272
28415060,0irq/27-eth0-tx-0-21swapper/222:41:412
28415060,0irq/27-eth0-tx-0-21swapper/222:24:342
28415060,0irq/27-eth0-tx-0-21swapper/222:18:522
28415060,0irq/27-eth0-tx-0-21swapper/222:05:392
28415060,0irq/27-eth0-tx-0-21swapper/221:55:522
28415060,0irq/27-eth0-tx-0-21swapper/221:40:082
28415060,0irq/27-eth0-tx-0-21swapper/221:38:032
28415060,0irq/27-eth0-tx-0-21swapper/221:24:042
28415060,0irq/27-eth0-tx-0-21swapper/221:09:422
28415060,0irq/27-eth0-tx-0-21swapper/220:50:282
28405060,0irq/26-eth0-rx-20019-21perl22:48:233
28405060,0irq/26-eth0-rx-10583-21iostat_ios22:38:343
269929966,0cyclictest0-21swapper/221:07:062
269929964,0cyclictest27511-21grep19:58:322
269909966,0cyclictest0-21swapper/123:13:311
269909960,0cyclictest5526-21irqrtprio20:08:351
269899966,0cyclictest0-21swapper/022:47:040
269899960,0cyclictest25138-21diskmemload22:02:080
8225050,0irq/24-0000:00:7938-21latency_hist18:58:142
8225050,0irq/24-0000:00:0-21swapper/223:39:142
339950,0migration/3887-21munin-node21:13:363
339950,0migration/3696-21kthreadcore20:03:333
339950,0migration/36222-21munin-node20:33:303
339950,0migration/34626-21kthreadcore19:18:313
339950,0migration/330599-21taskset20:26:033
339950,0migration/329177-21aten_r1power_po19:38:263
339950,0migration/326975-21kthreadcore00:08:273
339950,0migration/322577-21kthreadcore23:13:313
339950,0migration/319911-21kthreadcore19:03:333
339950,0migration/314559-21kthreadcore19:48:353
339950,0migration/31200-21kthreadcore20:28:353
339950,0migration/311059-21taskset20:13:213
28415050,0irq/27-eth0-tx-8166-21munin-node18:58:330
28415050,0irq/27-eth0-tx-27204-21munin-node18:48:310
28415050,0irq/27-eth0-tx-25571-21ssh22:48:392
28415050,0irq/27-eth0-tx-22635-21ssh21:58:242
28415050,0irq/27-eth0-tx-0-21swapper/223:49:412
28415050,0irq/27-eth0-tx-0-21swapper/223:46:432
28415050,0irq/27-eth0-tx-0-21swapper/222:53:242
28415050,0irq/27-eth0-tx-0-21swapper/222:44:062
28415050,0irq/27-eth0-tx-0-21swapper/222:30:582
28415050,0irq/27-eth0-tx-0-21swapper/222:14:412
28415050,0irq/27-eth0-tx-0-21swapper/222:11:142
28415050,0irq/27-eth0-tx-0-21swapper/221:48:532
28415050,0irq/27-eth0-tx-0-21swapper/200:13:292
28405050,0irq/26-eth0-rx-9882-21ssh23:00:463
28405050,0irq/26-eth0-rx-6493-21ssh00:13:503
28405050,0irq/26-eth0-rx-32271-21latency_hist22:08:153
28405050,0irq/26-eth0-rx-315-21perl23:48:243
28405050,0irq/26-eth0-rx-21821-21perl21:33:273
28405050,0irq/26-eth0-rx-21677-21ssh23:37:153
28405050,0irq/26-eth0-rx-15720-21ssh22:17:583
28405050,0irq/26-eth0-rx-15102-21ssh23:57:583
28405050,0irq/26-eth0-rx-10294-21sh21:20:493
28405050,0irq/26-eth0-rx-0-21swapper/323:58:313
28405050,0irq/26-eth0-rx-0-21swapper/323:38:163
28405050,0irq/26-eth0-rx-0-21swapper/323:27:013
28405050,0irq/26-eth0-rx-0-21swapper/323:20:353
28405050,0irq/26-eth0-rx-0-21swapper/323:20:353
28405050,0irq/26-eth0-rx-0-21swapper/323:11:083
28405050,0irq/26-eth0-rx-0-21swapper/322:54:323
28405050,0irq/26-eth0-rx-0-21swapper/322:44:133
28405050,0irq/26-eth0-rx-0-21swapper/322:31:393
28405050,0irq/26-eth0-rx-0-21swapper/321:58:353
28405050,0irq/26-eth0-rx-0-21swapper/321:48:273
28405050,0irq/26-eth0-rx-0-21swapper/321:42:093
28405050,0irq/26-eth0-rx-0-21swapper/321:06:033
28405050,0irq/26-eth0-rx-0-21swapper/321:01:213
28405050,0irq/26-eth0-rx-0-21swapper/320:38:313
28405050,0irq/26-eth0-rx-0-21swapper/300:04:393
269929955,0cyclictest0-21swapper/221:14:022
269909954,0cyclictest16087-21kworker/1:222:49:381
269909954,0cyclictest16087-21kworker/1:221:58:421
269899955,0cyclictest0-21swapper/023:51:140
269899955,0cyclictest0-21swapper/018:53:350
269899950,0cyclictest0-21swapper/021:44:380
259950,0migration/23300-21munin-node20:03:372
259950,0migration/226209-21kthreadcore19:08:352
259950,0migration/217699-21munin-node20:18:272
259950,0migration/211744-21missed_timers23:28:282
249950,0watchdog/213259-21munin-node21:28:252
249950,0watchdog/20-21swapper/223:59:492
249950,0watchdog/20-21swapper/223:21:052
249950,0watchdog/20-21swapper/223:21:052
249950,0watchdog/20-21swapper/222:59:132
249950,0watchdog/20-21swapper/222:37:532
249950,0watchdog/20-21swapper/221:21:092
249950,0watchdog/20-21swapper/220:53:252
249950,0watchdog/20-21swapper/220:40:052
249950,0watchdog/20-21swapper/220:29:372
249950,0watchdog/20-21swapper/220:23:412
249950,0watchdog/20-21swapper/219:33:372
249950,0watchdog/20-21swapper/200:03:292
179950,0migration/18791-21kthreadcore23:28:271
179950,0migration/18632-21kthreadcore19:43:341
179950,0migration/124176-21kthreadcore21:33:311
179950,0migration/120308-21kthreadcore23:58:331
179950,0migration/115138-21kthreadcore23:08:331
179950,0migration/114348-21kthreadcore20:13:321
169950,0watchdog/131558-21ssh23:47:171
149950,0migration/032354-21latency_hist22:08:150
149950,0migration/030552-21kthreadcore23:18:320
149950,0migration/030552-21kthreadcore23:18:320
149950,0migration/027960-21munin-node23:38:370
149950,0migration/024935-21kthreadcore19:58:310
149950,0migration/024762-21latency22:48:280
149950,0migration/017872-21sh21:04:460
149950,0migration/016560-21kthreadcore23:33:320
149950,0migration/016081-21kthreadcore21:28:270
339940,0migration/36914-21kthreadcore20:08:363
339940,0migration/329231-21kthreadcore18:48:333
339940,0migration/328096-21taskset19:13:173
339940,0migration/325908-21kthreadcore19:33:303
339940,0migration/318429-21gltestperf20:18:293
339940,0migration/316287-21fschecks_count19:28:313
339940,0migration/31521-21runrttasks22:33:243
28415040,0irq/27-eth0-tx-16829-21telnet19:53:270
28415040,0irq/27-eth0-tx-0-21swapper/020:23:360
28405040,0irq/26-eth0-rx-0-21swapper/323:44:463
28405040,0irq/26-eth0-rx-0-21swapper/323:28:263
28405040,0irq/26-eth0-rx-0-21swapper/322:18:333
28405040,0irq/26-eth0-rx-0-21swapper/321:45:183
28405040,0irq/26-eth0-rx-0-21swapper/321:23:223
28405040,0irq/26-eth0-rx-0-21swapper/321:08:253
28405040,0irq/26-eth0-rx-0-21swapper/320:43:313
28405040,0irq/26-eth0-rx-0-21swapper/319:23:323
28405040,0irq/26-eth0-rx-0-21swapper/319:08:283
269929944,0cyclictest18585-21kworker/2:119:23:242
269929944,0cyclictest18585-21kworker/2:119:18:222
269929944,0cyclictest18585-21kworker/2:119:13:162
269929944,0cyclictest18585-21kworker/2:119:03:462
269929944,0cyclictest18585-21kworker/2:118:58:162
269929944,0cyclictest15498-21kworker/2:220:43:202
269929944,0cyclictest15498-21kworker/2:220:13:182
269929944,0cyclictest15498-21kworker/2:220:08:162
269929944,0cyclictest15498-21kworker/2:219:53:182
269929944,0cyclictest15498-21kworker/2:219:48:242
269929944,0cyclictest15498-21kworker/2:219:43:182
269929944,0cyclictest15498-21kworker/2:219:38:162
269929944,0cyclictest15498-21kworker/2:219:28:262
269929944,0cyclictest15498-21kworker/2:200:08:142
269909944,0cyclictest23896-21kworker/1:020:28:181
269909944,0cyclictest23896-21kworker/1:019:38:401
269909944,0cyclictest23896-21kworker/1:019:23:521
269909944,0cyclictest23896-21kworker/1:019:08:281
269909944,0cyclictest23896-21kworker/1:019:03:201
269909944,0cyclictest16087-21kworker/1:222:53:221
269909944,0cyclictest16087-21kworker/1:222:43:301
269909944,0cyclictest16087-21kworker/1:222:24:141
269909944,0cyclictest16087-21kworker/1:222:14:081
269909944,0cyclictest16087-21kworker/1:222:04:381
269909944,0cyclictest16087-21kworker/1:221:08:281
269909944,0cyclictest16087-21kworker/1:221:03:241
269909944,0cyclictest10728-21kworker/1:123:40:541
269909944,0cyclictest0-21swapper/122:12:301
269909943,0cyclictest9098-21awk18:58:301
269909943,0cyclictest6445-21snmpd22:58:251
269909943,0cyclictest6445-21snmpd21:48:391
269909943,0cyclictest6445-21snmpd19:49:231
269909943,0cyclictest23896-21kworker/1:020:18:281
269909943,0cyclictest23896-21kworker/1:018:48:141
269909943,0cyclictest2380-21awk18:53:331
269909943,0cyclictest16657-21awk19:28:331
269909943,0cyclictest16087-21kworker/1:221:23:401
269909943,0cyclictest15338-21awk23:33:311
269909943,0cyclictest10728-21kworker/1:123:24:201
269909943,0cyclictest10728-21kworker/1:123:06:521
269909943,0cyclictest10728-21kworker/1:100:16:141
269909943,0cyclictest10728-21kworker/1:100:04:001
269899944,0cyclictest0-21swapper/023:27:360
269899943,0cyclictest9021-21awk22:13:290
269899943,0cyclictest8685-21awk23:53:310
269899943,0cyclictest6445-21snmpd19:37:380
269899943,0cyclictest6445-21snmpd19:29:380
269899943,0cyclictest6445-21snmpd19:25:380
269899943,0cyclictest6445-21snmpd19:18:230
269899943,0cyclictest5344-21awk20:08:340
269899943,0cyclictest4350-21awk21:18:320
269899943,0cyclictest4148-21awk20:28:380
269899943,0cyclictest3425-21lldpd23:43:280
269899943,0cyclictest3425-21lldpd21:08:490
269899943,0cyclictest3425-21lldpd20:04:170
269899943,0cyclictest3110-21irqbalance22:23:490
269899943,0cyclictest29898-21awk19:38:290
269899943,0cyclictest22073-21awk21:33:280
269899943,0cyclictest19595-21awk20:43:320
269899943,0cyclictest15770-21awk19:03:310
269899943,0cyclictest11902-21awk20:13:300
269899941,0cyclictest0-21swapper/020:48:300
269899940,0cyclictest0-21swapper/000:02:450
259940,0migration/28664-21kthreadcore20:33:322
249940,0watchdog/20-21swapper/218:48:292
179940,0migration/18320-21fschecks_time23:53:291
179940,0migration/16800-21kthreadcore21:18:341
179940,0migration/14627-21kthreadcore20:53:321
179940,0migration/131376-21timerwakeupswit20:48:411
179940,0migration/130768-21kthreadcore19:13:331
179940,0migration/127961-21kthreadcore00:08:281
179940,0migration/126210-21kthreadcore19:33:301
179940,0migration/126011-21aten_r1power_en22:28:211
179940,0migration/124127-21gltestperf19:58:281
179940,0migration/118662-21kthreadcore22:18:331
179940,0migration/116838-21aten_r1power_po19:53:271
179940,0migration/116540-21kthreadcore21:28:271
179940,0migration/116146-21sh22:39:591
179940,0migration/115406-21vmstat20:58:411
179940,0migration/114696-21kthreadcore20:38:311
179940,0migration/111203-21timerwakeupswit20:33:381
169940,0watchdog/18102-21ssh22:34:451
169940,0watchdog/132520-21grep23:18:331
169940,0watchdog/132520-21grep23:18:331
169940,0watchdog/130766-21grep20:03:291
169940,0watchdog/128411-21gltestperf21:13:291
169940,0watchdog/124191-21open_inodes20:43:371
169940,0watchdog/11575-21cut21:38:371
169940,0watchdog/10-21swapper/121:53:331
149940,0migration/09932-21fschecks_time20:58:300
149940,0migration/06733-21sh00:14:350
149940,0migration/02808-21latency_hist22:58:150
149940,0migration/026224-21kthreadcore00:08:270
149940,0migration/023599-21timerwakeupswit20:18:380
149940,0migration/022770-21kthreadcore19:08:340
149940,0migration/018180-21taskset20:39:530
149940,0migration/015509-21idleruntime21:53:300
149940,0migration/014312-21sh21:53:010
339930,0migration/37589-21runrttasks18:54:213
28405030,0irq/26-eth0-rx-16726-21runrttasks19:53:183
28405030,0irq/26-eth0-rx-0-21swapper/320:49:133
28405030,0irq/26-eth0-rx-0-21swapper/319:43:313
269909933,0cyclictest23896-21kworker/1:019:18:161
269909933,0cyclictest10728-21kworker/1:123:48:221
269909933,0cyclictest0-21swapper/121:43:201
269899933,0cyclictest0-21swapper/022:33:590
269899933,0cyclictest0-21swapper/019:14:050
269899932,0cyclictest991-21sensors21:13:370
269899932,0cyclictest6445-21snmpd22:54:400
269899932,0cyclictest6445-21snmpd00:07:400
269899932,0cyclictest5789-21ssh22:08:310
269899932,0cyclictest3425-21lldpd21:26:190
269899932,0cyclictest16106-21munin-node22:18:380
269899932,0cyclictest13670-21sh23:32:060
269899932,0cyclictest11025-21awk19:48:330
269899930,0cyclictest6026-21perl20:33:290
269899930,0cyclictest26563-21perl22:28:230
149930,0migration/09724-21diskmemload22:38:300
149930,0migration/08916-21sh20:58:090
149930,0migration/032399-21kthreadcore21:38:370
149930,0migration/027148-21sh23:14:150
28405020,0irq/26-eth0-rx-0-21swapper/318:59:303
269899921,0cyclictest3425-21lldpd23:03:260
269899920,0cyclictest12810-21latency_hist23:08:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional