You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 03:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot5s.osadl.org (updated Thu Apr 03, 2025 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2390704995742,10cyclictest2421807-21sendmail20:40:003
2390704995140,6cyclictest2478675-21idleruntime-cro23:50:003
2390690994835,9cyclictest121ktimers/020:20:130
2390690994734,10cyclictest121ktimers/000:15:120
2390700994313,10cyclictest2472467-21cat23:25:012
2390700994311,14cyclictest2420667-21kernelversion20:35:192
2390690994333,7cyclictest121ktimers/021:40:140
2390704994231,8cyclictest440-21cron21:50:003
2390696994226,12cyclictest231ktimers/122:35:121
2390690994236,3cyclictest2404819-21idleruntime-cro19:49:590
2390700994113,9cyclictest2447348-21/usr/sbin/munin22:00:192
2390696994133,5cyclictest2466147-21cron23:05:001
2390696994133,5cyclictest2441321-21cron21:40:001
2390696994120,5cyclictest131rcu_preempt22:30:001
239069099418,13cyclictest2479303-21grep23:50:130
2390700994013,9cyclictest2490777-21/usr/sbin/munin00:30:132
2390696994032,5cyclictest2425197-21ls20:50:001
2390700993912,9cyclictest2435923-21cpuspeed21:20:162
2390704993831,4cyclictest2468679-21cron23:15:013
2390704993827,7cyclictest2452324-21unixbench_multi22:15:273
239070099388,12cyclictest2477547-21timerwakeupswit23:40:262
2390700993829,5cyclictest2407744-21unixbench_multi19:55:232
2390704993729,4cyclictest0-21swapper/320:45:273
2390704993728,5cyclictest0-21swapper/319:35:183
2390700993729,4cyclictest0-21swapper/219:40:262
2390700993723,11cyclictest2452561-21sshd22:17:302
2390696993729,5cyclictest2470384-21apt-get23:20:001
2390696993719,7cyclictest2413111-21kworker/u8:2+events_unbound20:28:251
239069099375,13cyclictest0-21swapper/023:40:300
2390690993726,7cyclictest2463116-21grep22:55:140
2390690993711,9cyclictest2440828-21ls21:35:260
239070099369,9cyclictest2399714-21latency_hist19:35:012
239070099369,9cyclictest2399714-21latency_hist19:35:002
239070099369,10cyclictest2483539-21unixbench_multi00:00:262
2390700993625,6cyclictest2394199-21grep19:15:282
2390700993623,9cyclictest2460798-21sshd22:49:302
2390700993623,10cyclictest2395414-21/usr/sbin/munin19:20:222
2390700993618,12cyclictest2445685-21/usr/sbin/munin21:55:122
2390696993629,4cyclictest2445342-21cron21:55:011
2390696993626,7cyclictest2443038-21munin-plugin-st21:45:011
2390696993623,5cyclictest2413111-21kworker/u8:2+events_unbound20:34:221
2390696993615,10cyclictest0-21swapper/100:29:281
239069099367,11cyclictest2409914-21latency_hist20:05:010
2390690993627,5cyclictest0-21swapper/023:15:140
2390704993524,7cyclictest0-21swapper/321:30:253
239070099357,10cyclictest2400041-21cstates19:35:152
2390700993529,3cyclictest2410146-21apt-get20:05:142
2390700993529,3cyclictest2410146-21apt-get20:05:142
2390700993526,5cyclictest0-21swapper/200:35:272
2390700993524,6cyclictest0-21swapper/219:15:012
2390700993523,9cyclictest2475546-21/usr/sbin/munin23:35:222
2390700993522,10cyclictest2428532-21cron20:57:002
2390700993522,10cyclictest2428532-21cron20:57:002
2390700993522,10cyclictest2404427-21users19:45:272
2390696993515,6cyclictest131rcu_preempt19:44:511
2390696993513,10cyclictest131rcu_preempt21:15:171
2390690993526,5cyclictest0-21swapper/021:00:000
2390690993526,5cyclictest0-21swapper/020:59:590
2390690993523,8cyclictest2435116-21grep21:15:250
2390704993424,6cyclictest2399241-21grep19:30:283
2390704993424,6cyclictest2399241-21grep19:30:273
239070099347,10cyclictest2441271-21chrt21:39:332
2390700993425,4cyclictest0-21swapper/219:50:282
2390700993424,7cyclictest2489458-21cron00:22:012
2390700993423,7cyclictest2459092-21/usr/sbin/munin22:40:182
2390700993421,10cyclictest2431427-21/usr/sbin/munin21:05:242
2390700993421,10cyclictest2412402-21/usr/sbin/munin20:10:242
2390700993421,10cyclictest2412402-21/usr/sbin/munin20:10:232
2390700993421,10cyclictest2409132-21/usr/sbin/munin20:00:232
2390700993415,15cyclictest2443018-21cron21:45:012
239069699348,8cyclictest24-21ksoftirqd/120:55:271
239069699348,8cyclictest24-21ksoftirqd/120:55:271
2390696993428,4cyclictest2401037-21kworker/u8:0+events_unbound19:39:321
2390696993424,6cyclictest0-21swapper/120:05:001
2390696993417,4cyclictest131rcu_preempt23:30:231
2390696993417,4cyclictest131rcu_preempt00:35:451
2390696993414,6cyclictest131rcu_preempt19:30:281
2390696993414,6cyclictest131rcu_preempt19:30:281
2390696993414,4cyclictest131rcu_preempt21:29:011
2390696993413,4cyclictest131rcu_preempt22:55:081
2390696993412,9cyclictest131rcu_preempt19:16:391
239069099347,9cyclictest2482270-21latency_hist00:00:010
239069099342,9cyclictest11-21ksoftirqd/020:46:110
2390704993326,4cyclictest0-21swapper/321:00:253
2390704993324,4cyclictest0-21swapper/319:15:023
2390704993323,6cyclictest0-21swapper/323:40:263
239070099335,19cyclictest0-21swapper/220:45:242
2390700993327,3cyclictest2432436-21apt-get21:10:162
2390700993326,4cyclictest2443883-21apt-get21:50:072
2390700993326,4cyclictest2425752-21df_inode20:50:192
2390700993326,4cyclictest2425752-21df_inode20:50:192
2390700993324,6cyclictest2478665-21auditctl23:50:002
2390700993324,4cyclictest0-21swapper/220:30:302
2390700993323,7cyclictest2450452-21/usr/sbin/munin22:10:202
2390700993323,7cyclictest2397965-21sshd19:29:272
2390700993320,9cyclictest2437715-21sshd21:27:292
2390700993320,10cyclictest2473508-21/usr/sbin/munin23:30:202
2390700993318,11cyclictest2464694-21/usr/sbin/munin23:00:202
2390700993314,15cyclictest0-21swapper/223:27:342
2390696993320,9cyclictest221rcuc/120:20:001
2390696993316,4cyclictest131rcu_preempt19:49:111
2390696993315,9cyclictest131rcu_preempt21:30:471
2390696993314,3cyclictest0-21swapper/122:11:491
2390696993312,3cyclictest131rcu_preempt21:10:211
2390690993326,3cyclictest0-21swapper/022:20:000
2390690993323,6cyclictest0-21swapper/020:40:010
239069099331,18cyclictest0-21swapper/019:40:090
2390690993310,18cyclictest2461040-21latency_hist22:50:020
2390704993225,4cyclictest0-21swapper/320:50:293
2390704993225,4cyclictest0-21swapper/320:50:293
2390704993223,6cyclictest0-21swapper/300:10:013
2390704993223,5cyclictest24620872sleep322:50:213
2390700993226,3cyclictest2468918-21apt-get23:15:112
2390700993224,4cyclictest0-21swapper/220:19:592
2390700993222,7cyclictest2487079-21/usr/sbin/munin00:15:222
2390700993222,7cyclictest2481929-21cron23:56:002
2390700993222,7cyclictest2453549-21/usr/sbin/munin22:20:242
2390700993222,7cyclictest2453549-21/usr/sbin/munin22:20:232
2390700993222,7cyclictest2422801-21/usr/sbin/munin20:40:252
2390700993221,8cyclictest2479556-21/usr/sbin/munin23:50:172
2390700993221,7cyclictest2467403-21/usr/sbin/munin23:10:112
2390700993221,7cyclictest0-21swapper/221:15:262
2390700993220,9cyclictest2463320-21/usr/sbin/munin22:55:172
2390700993220,9cyclictest2429172-21/usr/sbin/munin21:00:152
2390700993219,10cyclictest2484357-21cron00:07:002
2390700993219,10cyclictest2417389-21meminfo20:25:202
2390700993217,11cyclictest2458090-21grep22:35:252
239069699329,10cyclictest0-21swapper/123:40:391
239069699329,10cyclictest0-21swapper/122:00:071
239069699328,3cyclictest2413295-21apt20:15:001
239069699328,3cyclictest2413295-21apt20:15:001
2390696993224,6cyclictest2411401-21kworker/u8:0+events_unbound20:09:061
2390696993224,6cyclictest2411401-21kworker/u8:0+events_unbound20:09:051
2390696993223,5cyclictest231ktimers/123:55:131
2390696993221,4cyclictest131rcu_preempt22:17:281
2390696993218,6cyclictest2462411-21kworker/u8:2+events_unbound23:24:141
2390696993218,4cyclictest131rcu_preempt23:48:021
2390696993216,4cyclictest131rcu_preempt23:10:181
2390696993215,5cyclictest131rcu_preempt22:35:011
2390696993214,5cyclictest131rcu_preempt20:50:191
2390696993214,5cyclictest131rcu_preempt20:50:191
2390696993213,10cyclictest0-21swapper/120:22:321
2390696993212,4cyclictest131rcu_preempt20:39:211
239069099324,8cyclictest0-21swapper/021:12:570
2390690993219,10cyclictest2406943-21/usr/sbin/munin19:55:150
2390690993217,3cyclictest2464697-21apt-get23:00:140
2390704993124,4cyclictest0-21swapper/323:50:223
2390704993122,5cyclictest0-21swapper/323:35:253
2390704993120,7cyclictest2446572-21grep21:55:253
2390704993119,7cyclictest2456346-21grep22:30:203
2390700993121,8cyclictest2462325-21sshd22:52:312
2390700993121,7cyclictest2456432-21sshd22:30:312
2390700993121,7cyclictest2438270-21/usr/sbin/munin21:30:132
239069699319,12cyclictest24-21ksoftirqd/123:50:181
239069699317,4cyclictest0-21swapper/100:30:161
239069699314,7cyclictest24-21ksoftirqd/100:05:011
2390696993120,8cyclictest231ktimers/121:20:231
2390696993120,4cyclictest131rcu_preempt23:35:191
2390696993119,8cyclictest231ktimers/121:05:261
2390696993117,7cyclictest2484345-21kworker/u8:2+events_unbound00:17:571
2390696993114,5cyclictest131rcu_preempt20:00:031
2390696993114,4cyclictest131rcu_preempt21:02:361
2390696993114,11cyclictest131rcu_preempt19:25:191
2390696993112,4cyclictest131rcu_preempt20:40:221
239069099319,17cyclictest0-21swapper/000:25:000
239069099315,24cyclictest0-21swapper/019:15:020
239069099313,17cyclictest637-21runrttasks22:03:150
2390690993124,4cyclictest2430871-21cut21:05:180
2390690993121,7cyclictest2413837-21/usr/sbin/munin20:15:190
2390690993121,6cyclictest0-21swapper/020:15:000
2390690993121,6cyclictest0-21swapper/020:15:000
2390690993121,6cyclictest0-21swapper/000:15:010
2390690993120,7cyclictest2438639-21/usr/sbin/munin21:30:200
2390690993119,9cyclictest2491300-21/usr/sbin/munin00:30:160
2390690993118,10cyclictest2429704-21/usr/sbin/munin21:00:230
2390704993022,4cyclictest0-21swapper/319:15:293
2390704993019,8cyclictest2470513-21latency_hist23:20:013
2390700993020,7cyclictest2484957-21diskstats00:10:122
2390700993019,8cyclictest2416616-21cron20:24:012
2390700993015,12cyclictest2490408-21sendmail-mta00:28:172
2390696993021,6cyclictest0-21swapper/122:10:011
2390696993019,7cyclictest0-21swapper/122:50:201
2390696993019,3cyclictest131rcu_preempt00:06:421
2390696993017,4cyclictest131rcu_preempt19:23:341
2390696993016,4cyclictest131rcu_preempt19:50:311
2390696993014,10cyclictest131rcu_preempt00:10:091
2390696993012,11cyclictest131rcu_preempt00:20:251
2390690993023,3cyclictest0-21swapper/022:00:010
2390690993020,7cyclictest0-21swapper/000:40:000
2390690993017,10cyclictest2472601-21cron23:26:000
2390690993017,10cyclictest2449722-21/usr/sbin/munin22:10:090
2390690993014,12cyclictest2411207-21head20:05:260
2390690993014,12cyclictest2411207-21head20:05:260
239069099301,21cyclictest0-21swapper/021:47:230
2390704992921,4cyclictest0-21swapper/322:00:313
2390704992920,5cyclictest0-21swapper/320:20:013
2390704992920,4cyclictest0-21swapper/300:00:183
2390700992920,6cyclictest2466742-21cron23:07:012
2390696992923,4cyclictest2462411-21kworker/u8:2+events_unbound23:25:211
2390696992915,4cyclictest131rcu_preempt23:05:061
2390696992913,4cyclictest131rcu_preempt22:45:281
239069099298,13cyclictest0-21swapper/020:25:230
2390690992920,6cyclictest121ktimers/000:00:150
2390690992920,5cyclictest2468278-21grep23:10:210
2390690992919,7cyclictest2419757-21cron20:31:000
2390690992916,10cyclictest0-21swapper/022:30:070
2390690992915,11cyclictest2475021-21/usr/sbin/munin23:35:160
2390690992914,11cyclictest2458034-21grep22:35:250
2390690992911,13cyclictest2455304-21latency_hist22:30:000
2390690992911,12cyclictest0-21swapper/023:20:130
2390704992821,4cyclictest0-21swapper/320:55:263
2390704992821,4cyclictest0-21swapper/320:55:263
2390704992820,4cyclictest0-21swapper/321:15:253
2390704992815,8cyclictest0-21swapper/319:25:003
239070099283,15cyclictest0-21swapper/222:06:022
2390700992814,11cyclictest2454718-21seq22:25:152
239069699289,5cyclictest131rcu_preempt21:48:471
2390696992821,4cyclictest231ktimers/122:20:551
2390696992821,4cyclictest231ktimers/122:20:541
2390696992812,5cyclictest131rcu_preempt21:55:231
239069699281,17cyclictest0-21swapper/119:10:191
239069099289,14cyclictest2490555-21latency_hist00:30:010
2390690992818,7cyclictest2473623-21memory23:30:230
2390690992818,7cyclictest2461702-21/usr/sbin/munin22:50:180
2390690992818,7cyclictest2400035-21/usr/sbin/munin19:35:160
2390690992818,7cyclictest2393708-21/usr/sbin/munin19:15:250
2390690992817,9cyclictest2422222-21/usr/sbin/munin20:40:180
2390690992817,8cyclictest2437592-21cron21:26:010
2390690992817,7cyclictest2454079-21grep22:20:300
2390690992817,7cyclictest2454079-21grep22:20:300
2390690992810,14cyclictest2395563-21mailstats19:20:260
2390704992719,5cyclictest0-21swapper/322:40:273
2390704992719,4cyclictest0-21swapper/323:04:593
2390704992719,4cyclictest0-21swapper/321:40:273
2390704992719,4cyclictest0-21swapper/320:00:163
2390704992718,6cyclictest391ktimers/300:10:133
2390704992718,6cyclictest2449813-21grep22:10:113
2390704992717,6cyclictest0-21swapper/300:20:013
239070099273,16cyclictest0-21swapper/221:45:142
2390696992715,4cyclictest131rcu_preempt22:40:251
2390690992719,4cyclictest2478658-21cron23:49:590
2390690992718,5cyclictest2444958-21users21:50:220
2390690992717,7cyclictest2435955-21df_inode21:20:160
2390690992717,7cyclictest2399095-21/usr/sbin/munin19:30:270
2390690992717,7cyclictest2399095-21/usr/sbin/munin19:30:270
2390690992717,7cyclictest2397928-21cron19:29:000
2390690992716,8cyclictest2458966-21/usr/sbin/munin22:40:180
2390690992713,10cyclictest2425796-21fschecks_count20:50:200
2390690992713,10cyclictest2425796-21fschecks_count20:50:190
2390704992619,4cyclictest0-21swapper/320:30:283
2390704992618,5cyclictest0-21swapper/320:40:273
2390704992618,4cyclictest0-21swapper/300:30:123
2390704992614,8cyclictest2482296-21basename00:00:003
2390690992617,6cyclictest121ktimers/023:05:040
2390690992616,7cyclictest2449293-21cron22:08:590
2390690992616,7cyclictest2405042-21/usr/sbin/munin19:50:150
2390704992517,4cyclictest391ktimers/323:25:013
2390704992513,8cyclictest0-21swapper/300:40:003
239069099251,18cyclictest0-21swapper/000:06:290
2390704992416,4cyclictest2474631-21auditctl23:35:003
2390704992416,4cyclictest0-21swapper/320:00:013
2390704992317,3cyclictest0-21swapper/319:40:193
2390704992315,4cyclictest0-21swapper/322:35:243
239070499231,12cyclictest0-21swapper/323:07:023
2390704992217,3cyclictest2449548-21latency_hist22:10:013
2390704992216,3cyclictest0-21swapper/322:20:223
2390704992216,3cyclictest0-21swapper/322:20:223
2390704992214,3cyclictest0-21swapper/320:25:173
239070499212,13cyclictest2445346-21sh21:55:013
239070499211,9cyclictest2432572-21grep21:10:203
2390704992115,3cyclictest2455126-21idleruntime-cro22:30:003
2390704992115,3cyclictest0-21swapper/320:05:183
2390704992115,3cyclictest0-21swapper/320:05:173
2390704992115,3cyclictest0-21swapper/319:50:193
2390704992115,3cyclictest0-21swapper/319:25:143
2390704992114,4cyclictest391ktimers/321:20:263
2390704992114,3cyclictest0-21swapper/321:35:223
2390704992113,5cyclictest0-21swapper/300:20:263
2390704992110,8cyclictest2404838-21sort19:50:003
239070499207,3cyclictest6250watchdogd00:25:023
2390704992013,4cyclictest2462538-21apt-get22:55:063
239070499201,15cyclictest2437922-21cron21:30:013
2390704991914,3cyclictest2432197-21latency_hist21:10:013
2390704991913,4cyclictest391ktimers/320:20:083
2390704991912,4cyclictest2472929-21cron23:30:003
2390704991911,4cyclictest0-21swapper/322:50:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional