You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 03, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31027210375,9sleep10-21swapper/119:05:501
31059210275,8sleep30-21swapper/319:06:173
2922228565,15sleep20-21swapper/219:05:122
2923028267,10sleep00-21swapper/019:05:170
3143299380,37cyclictest0-21swapper/321:55:113
31432992513,6cyclictest39-21ksoftirqd/323:15:193
31432992513,6cyclictest39-21ksoftirqd/323:15:183
31419992423,1cyclictest0-21swapper/022:29:580
31419992418,1cyclictest24001-21diskmemload22:19:460
3143299220,22cyclictest0-21swapper/300:18:553
31428992220,2cyclictest37850irq/131-snd_hda23:45:272
31428992220,2cyclictest37850irq/131-snd_hda22:56:452
31428992220,2cyclictest37850irq/131-snd_hda22:49:432
31428992220,2cyclictest37850irq/131-snd_hda22:42:242
31428992220,2cyclictest37850irq/131-snd_hda22:01:142
31428992220,2cyclictest37850irq/131-snd_hda00:34:082
31428992220,2cyclictest37850irq/131-snd_hda00:29:162
31428992220,2cyclictest37850irq/131-snd_hda00:11:282
31428992220,2cyclictest37850irq/131-snd_hda00:07:052
31428992220,1cyclictest37850irq/131-snd_hda23:57:382
31428992220,1cyclictest37850irq/131-snd_hda23:55:002
31428992220,1cyclictest37850irq/131-snd_hda23:37:252
31428992220,1cyclictest37850irq/131-snd_hda23:25:012
31428992220,1cyclictest37850irq/131-snd_hda23:15:082
31428992220,1cyclictest37850irq/131-snd_hda23:15:072
31428992220,1cyclictest37850irq/131-snd_hda23:10:062
31428992220,1cyclictest37850irq/131-snd_hda23:05:042
31428992220,1cyclictest37850irq/131-snd_hda23:02:522
31428992220,1cyclictest37850irq/131-snd_hda22:31:222
31428992220,1cyclictest37850irq/131-snd_hda22:26:072
31428992220,1cyclictest37850irq/131-snd_hda22:20:012
31428992220,1cyclictest37850irq/131-snd_hda22:15:422
31428992220,1cyclictest37850irq/131-snd_hda21:59:202
31428992220,1cyclictest37850irq/131-snd_hda21:47:002
31428992220,1cyclictest37850irq/131-snd_hda21:26:572
31428992220,1cyclictest37850irq/131-snd_hda21:22:492
31428992220,1cyclictest37850irq/131-snd_hda21:19:162
31428992220,1cyclictest37850irq/131-snd_hda21:10:182
31428992219,2cyclictest37850irq/131-snd_hda21:41:522
3142299220,21cyclictest0-21swapper/123:19:241
3142299220,21cyclictest0-21swapper/123:19:231
3142299220,18cyclictest0-21swapper/100:17:581
3141999220,2cyclictest0-21swapper/022:50:540
31428992120,1cyclictest37850irq/131-snd_hda23:30:182
31428992120,1cyclictest37850irq/131-snd_hda23:20:042
31428992120,1cyclictest37850irq/131-snd_hda23:20:042
31428992120,1cyclictest37850irq/131-snd_hda21:31:392
31428992120,1cyclictest37850irq/131-snd_hda00:38:582
31428992120,1cyclictest37850irq/131-snd_hda00:20:532
31428992120,1cyclictest37850irq/131-snd_hda00:02:182
31428992119,2cyclictest37850irq/131-snd_hda22:50:092
31428992119,2cyclictest37850irq/131-snd_hda22:36:182
31428992119,2cyclictest37850irq/131-snd_hda22:05:182
31428992119,2cyclictest37850irq/131-snd_hda21:50:092
31428992119,2cyclictest37850irq/131-snd_hda00:15:532
31428992119,1cyclictest37850irq/131-snd_hda22:12:392
31428992118,2cyclictest37850irq/131-snd_hda21:36:092
31428992118,2cyclictest37850irq/131-snd_hda21:36:082
31419992119,1cyclictest24001-21diskmemload23:20:560
31419992119,1cyclictest24001-21diskmemload23:20:550
31419992119,1cyclictest0-21swapper/022:20:180
31419992118,2cyclictest13177-21ssh21:50:100
313502210,2chrt29054-21kworker/2:023:40:042
3142299200,18cyclictest0-21swapper/122:01:411
31419992016,3cyclictest19791-21memory23:55:210
31419992016,1cyclictest24001-21diskmemload22:31:390
31419992013,4cyclictest0-21swapper/023:46:450
3141999200,19cyclictest0-21swapper/000:15:590
31432991917,2cyclictest20754-21kworker/u8:223:35:373
31419991917,1cyclictest0-21swapper/019:45:140
31419991916,2cyclictest0-21swapper/023:06:160
31419991916,2cyclictest0-21swapper/022:05:380
31419991915,3cyclictest0-21swapper/000:10:100
31419991913,2cyclictest0-21swapper/020:55:130
31419991912,7cyclictest0-21swapper/022:10:320
3143299180,1cyclictest141rcu_preempt21:46:313
3143299180,17cyclictest0-21swapper/322:51:023
31422991812,5cyclictest25-21ksoftirqd/122:23:371
3141999187,8cyclictest0-21swapper/020:20:140
31419991812,5cyclictest0-21swapper/000:31:330
3143299170,17cyclictest0-21swapper/322:16:423
3142299170,6cyclictest0-21swapper/120:15:151
3142299170,15cyclictest0-21swapper/121:58:501
3141999174,10cyclictest0-21swapper/000:09:410
31419991713,4cyclictest0-21swapper/021:24:090
31419991710,4cyclictest0-21swapper/019:20:170
3141999170,2cyclictest0-21swapper/021:33:430
3143299162,2cyclictest141rcu_preempt23:50:093
31432991615,1cyclictest0-21swapper/322:20:203
31432991614,1cyclictest39-21ksoftirqd/322:00:083
3143299161,1cyclictest141rcu_preempt19:35:183
3142299160,1cyclictest0-21swapper/121:35:151
3142299160,1cyclictest0-21swapper/121:35:141
3141999169,3cyclictest0-21swapper/022:55:300
3141999161,11cyclictest16722-21munin-run23:30:010
31419991610,6cyclictest0-21swapper/023:01:070
3143299159,4cyclictest39-21ksoftirqd/323:12:313
3143299151,1cyclictest141rcu_preempt21:14:473
3143299150,15cyclictest0-21swapper/322:38:353
31428991510,4cyclictest17568-21grep19:45:282
3142299153,10cyclictest24001-21diskmemload21:27:451
31422991514,1cyclictest0-21swapper/123:25:241
3142299150,2cyclictest0-21swapper/121:49:131
3141999159,1cyclictest0-21swapper/022:43:340
3141999158,3cyclictest0-21swapper/023:50:080
3141999157,5cyclictest25833-21open_inodes21:10:240
3141999157,3cyclictest0-21swapper/021:05:160
3141999156,8cyclictest0-21swapper/021:15:130
31419991513,1cyclictest0-21swapper/022:47:300
31419991513,1cyclictest0-21swapper/021:40:180
31419991511,3cyclictest0-21swapper/021:26:380
3141999150,1cyclictest24001-21diskmemload21:58:270
31432991410,3cyclictest18424-21cut00:20:143
3143299140,14cyclictest0-21swapper/323:46:023
31422991410,3cyclictest27942-21grep20:10:161
3141999149,3cyclictest0-21swapper/021:35:010
3141999149,3cyclictest0-21swapper/021:35:000
3141999149,1cyclictest0-21swapper/000:25:280
3141999148,6cyclictest0-21swapper/023:33:200
31419991410,3cyclictest13156-21sed20:45:180
3141999140,1cyclictest0-21swapper/000:03:330
3143299139,3cyclictest5051-21wc20:30:003
3143299139,3cyclictest11039-21cat20:40:203
3143299138,2cyclictest39-21ksoftirqd/322:45:053
31432991311,1cyclictest39-21ksoftirqd/322:13:333
31432991310,2cyclictest9813-21cat20:40:003
31432991310,2cyclictest4925-21cat00:10:003
31432991310,2cyclictest23339-21grep20:00:143
31432991310,2cyclictest1512-21cron00:35:003
3143299130,3cyclictest0-21swapper/321:26:083
3143299130,13cyclictest0-21swapper/322:55:013
3142899139,3cyclictest4785-21grep19:20:172
3142899139,3cyclictest20250-21head21:00:162
3142299139,3cyclictest8545-21cron19:30:011
31422991310,2cyclictest870-21chrt19:10:271
31422991310,2cyclictest5651-21cron21:20:011
31422991310,2cyclictest3337-21sed19:15:301
31422991310,2cyclictest29779-21grep22:50:121
31422991310,2cyclictest19017-21grep21:30:141
31422991310,2cyclictest12153-21sort20:44:591
3141999139,3cyclictest0-21swapper/020:00:130
3141999132,7cyclictest5875-21fschecks_count20:30:210
31419991312,1cyclictest0-21swapper/023:41:350
31419991310,3cyclictest0-21swapper/022:37:200
31419991310,2cyclictest8559-21idleruntime-cro19:30:000
31419991310,2cyclictest29489-21sed20:10:300
31419991310,2cyclictest23707-21grep23:10:120
31419991310,2cyclictest20764-21grep19:55:130
3143299129,2cyclictest7430-21lockfile-remove00:40:013
3143299129,2cyclictest6045-21grep00:10:133
3143299129,2cyclictest32316-21tail21:15:173
3143299129,2cyclictest25346-21rm23:59:593
3143299129,2cyclictest2490-21tr22:30:113
3143299129,2cyclictest22122-21grep21:05:163
3143299129,2cyclictest21617-21cron21:04:593
3143299129,2cyclictest21345-21sort19:55:163
3143299129,2cyclictest20026-21grep19:50:293
3143299129,2cyclictest13722-21tr19:40:143
3143299128,2cyclictest39-21ksoftirqd/323:07:053
3143299123,4cyclictest224-21systemd-journal22:40:303
31432991211,1cyclictest0-21swapper/321:35:153
31432991211,1cyclictest0-21swapper/321:35:143
31432991210,1cyclictest39-21ksoftirqd/321:21:303
3143299120,3cyclictest0-21swapper/322:05:353
3142899129,2cyclictest4047-21grep20:25:232
3142899129,2cyclictest12912-21head19:35:282
3142899129,2cyclictest1238-21grep20:20:192
3142899128,3cyclictest25037-21taskset20:03:082
3142899128,3cyclictest15026-21cat19:40:252
3142899128,3cyclictest14078-21grep20:45:282
3142299129,2cyclictest26575-21uname20:05:221
3142299129,2cyclictest25600-21grep23:35:191
3142299129,2cyclictest24690-21cut21:10:171
3142299129,2cyclictest20425-21cron19:55:001
3142299129,2cyclictest16965-21grep22:40:151
3142299129,2cyclictest15744-21tr22:15:181
3142299129,2cyclictest11869-21sort19:35:191
3142299127,4cyclictest25-21ksoftirqd/100:01:081
3142299122,8cyclictest6724-21ssh22:56:171
3142299121,1cyclictest0-21swapper/121:20:381
3142299120,11cyclictest4178-21ssh00:35:501
3141999129,2cyclictest3076-21grep20:25:150
3141999129,2cyclictest2799-21cut00:35:200
3141999129,2cyclictest25643-21tr23:35:210
3141999129,2cyclictest18617-21tr19:50:170
3141999129,2cyclictest15978-21grep20:50:260
3141999129,1cyclictest0-21swapper/023:17:360
3141999129,1cyclictest0-21swapper/023:17:350
3141999126,2cyclictest224-21systemd-journal19:15:000
3141999124,4cyclictest224-21systemd-journal20:35:280
31419991210,1cyclictest0-21swapper/019:30:200
3143299119,1cyclictest4642-21cut19:20:173
3143299119,1cyclictest15499-21head20:50:203
3143299118,2cyclictest7681-21grep19:25:233
3143299118,2cyclictest715-21ssh22:28:593
3143299118,2cyclictest6970-21grep20:30:303
3143299118,2cyclictest641-21apt21:40:153
3143299118,2cyclictest5185-21expr23:20:153
3143299118,2cyclictest5185-21expr23:20:143
3143299118,2cyclictest32647-21wget20:20:133
3143299118,2cyclictest32313-21fschecks_time19:10:173
3143299118,2cyclictest30659-21expr20:15:173
3143299118,2cyclictest25950-21tr00:00:163
3143299118,2cyclictest19920-21strings21:30:203
3143299118,2cyclictest18101-21sed23:30:253
3143299118,2cyclictest16660-21grep19:45:193
3143299118,2cyclictest16618-21ssh23:04:153
3143299118,2cyclictest15753-21sh21:51:013
3143299118,2cyclictest13026-21sh23:26:273
3143299117,2cyclictest39-21ksoftirqd/300:25:303
31432991111,0cyclictest0-21swapper/323:40:203
3143299110,7cyclictest0-21swapper/320:10:203
3142899118,2cyclictest400-21grep19:10:222
3142899118,2cyclictest30215-21apt-get20:15:152
3142899118,2cyclictest29781-21chrt20:13:022
3142899118,2cyclictest19274-21sendmail-msp21:00:002
3142899118,2cyclictest15006-21grep20:50:182
3142899118,2cyclictest10716-21grep20:40:182
31428991110,1cyclictest37850irq/131-snd_hda19:55:122
3142899110,10cyclictest8447-21chrt19:28:482
3142299119,1cyclictest4699-31sort00:10:351
3142299118,2cyclictest8558-21grep22:10:131
3142299118,2cyclictest7398-21ssh23:46:101
3142299118,2cyclictest6575-21head23:20:271
3142299118,2cyclictest6575-21head23:20:261
3142299118,2cyclictest5250-21apache_processe20:30:151
3142299118,2cyclictest3773-21ssh22:06:351
3142299118,2cyclictest3405-21sed22:30:151
3142299118,2cyclictest32405-21sed23:40:191
3142299118,2cyclictest30957-21ssh22:26:531
3142299118,2cyclictest3089-21tr20:25:161
3142299118,2cyclictest26566-21ssh23:11:591
3142299118,2cyclictest24785-21ssh00:26:181
3142299118,2cyclictest24145-21ssh22:45:351
3142299118,2cyclictest23224-21wget20:00:141
3142299118,2cyclictest21294-21grep19:55:151
3142299118,2cyclictest19127-21ssh23:06:341
3142299118,2cyclictest18478-21grep23:30:261
3142299118,2cyclictest17751-21cut20:55:171
3142299118,2cyclictest1697-21uname21:40:221
3142299118,2cyclictest16694-21tail19:45:191
3142299118,2cyclictest15117-21ssh21:50:231
3142299118,2cyclictest15069-21sh23:51:511
3142299118,2cyclictest11001-21apache_processe23:00:151
3142299117,3cyclictest18841-21grep23:55:151
3142299117,3cyclictest18284-21ssh00:20:131
3142299117,2cyclictest9602-21cut22:35:141
3142299117,2cyclictest22602-21tail21:05:191
3142299110,10cyclictest343-21ssh00:05:381
3141999119,2cyclictest0-21swapper/020:05:140
3141999118,3cyclictest0-21swapper/022:01:410
3141999118,2cyclictest7856-21ssh21:45:160
3141999118,2cyclictest31654-21cat20:15:260
3141999118,2cyclictest2859-21grep19:15:240
3141999118,2cyclictest20990-21grep21:00:250
3141999117,3cyclictest13-21ksoftirqd/020:40:140
3141999117,1cyclictest0-21swapper/000:22:140
3141999110,10cyclictest12656-21cat19:35:270
3143299108,1cyclictest8772-21chrt19:30:003
3143299108,1cyclictest26422-21wc20:05:213
3143299108,1cyclictest2500-21cat19:15:223
3143299108,1cyclictest17918-21which20:55:183
3143299108,1cyclictest142532sleep320:47:063
3142899107,2cyclictest7584-21taskset20:35:092
3142899107,2cyclictest2721-21sed19:15:232
3142899107,2cyclictest26556-21cut20:05:222
3142299108,1cyclictest21300-21taskset21:00:571
3142299108,1cyclictest135422sleep119:40:041
3142299107,2cyclictest4803-21head19:20:171
3142299107,2cyclictest28850-21cut00:30:131
3142299107,2cyclictest1280-21grep20:20:191
3142299107,2cyclictest12497-21awk20:45:161
3141999108,1cyclictest13995-21expr19:40:150
314289997,1cyclictest22097-21grep21:05:172
314289997,1cyclictest20029-21sed19:50:292
314289996,2cyclictest5607-21expr20:30:182
314289990,4cyclictest9445-21cut19:30:192
314229997,1cyclictest16128-21tr20:50:261
314229997,1cyclictest10596-21chrt19:30:541
314229996,2cyclictest8764-21ntpq20:35:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional