You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 14:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Sat Jul 27, 2024 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
289710799508499,6cyclictest2974455-21turbostat22:54:553
289710299507500,4cyclictest2946310-21turbostat21:34:552
289710799505497,5cyclictest2927292-21turbostat20:35:003
289710299501494,4cyclictest2901045-21turbostat19:24:552
289710799471449,19cyclictest2964088-21turbostat22:24:553
289710799465457,5cyclictest2906221-21turbostat19:39:563
289710799465444,8cyclictest2941128-21turbostat21:19:543
289710799464457,4cyclictest2949772-21turbostat21:40:003
289710799463441,19cyclictest2904500-21turbostat19:34:553
289710799461454,4cyclictest2937671-21turbostat21:09:553
289710799459439,17cyclictest2984821-21turbostat23:20:003
289710799457448,6cyclictest3003902-21turbostat00:15:003
289710299456447,6cyclictest3000406-21turbostat00:09:542
289710299455447,5cyclictest2990006-21turbostat23:39:552
289710299454447,4cyclictest2920387-21turbostat20:19:552
289710799451444,4cyclictest2930757-21turbostat20:49:543
289710799451444,4cyclictest2913455-21turbostat19:59:553
289710799451442,6cyclictest2953420-21turbostat21:50:003
289710799451442,6cyclictest2953420-21turbostat21:50:003
289709299451434,5cyclictest2981364-21turbostat23:14:550
289710799450442,4cyclictest2998651-21turbostat00:04:553
289710799448440,5cyclictest2948046-21turbostat21:35:003
289710799447440,4cyclictest2988288-21turbostat23:30:003
289710799447439,5cyclictest2957014-21turbostat22:04:553
289710799445438,4cyclictest2949772-21turbostat21:44:543
289710799445437,5cyclictest2923830-21turbostat20:29:563
289710799445437,5cyclictest2923830-21turbostat20:29:563
289710799444437,4cyclictest2934208-21turbostat20:55:013
289710799444434,7cyclictest2898812-21turbostat19:19:553
289710799443434,6cyclictest2962356-21turbostat22:15:003
289710799443434,6cyclictest2962356-21turbostat22:15:003
289710299442435,4cyclictest2967536-21turbostat22:34:552
289710799441419,19cyclictest2984821-21turbostat23:24:553
289710299441432,6cyclictest2927292-21turbostat20:39:552
289710799440432,5cyclictest2946310-21turbostat21:30:003
289710799440432,5cyclictest2913455-21turbostat19:55:003
289710799440431,6cyclictest2920387-21turbostat20:15:003
289710799438430,5cyclictest2958745-21turbostat22:09:543
289710799436429,4cyclictest2965815-21turbostat22:29:553
289710799435428,4cyclictest2970995-21turbostat22:44:563
289709499434422,8cyclictest2942859-21turbostat21:24:561
289709499434422,8cyclictest2942859-21turbostat21:24:561
289710799432424,5cyclictest2993460-21turbostat23:45:003
289710799432424,4cyclictest2939393-21turbostat21:14:553
289709299432424,5cyclictest2907945-21turbostat19:44:550
289710799429421,5cyclictest2993460-21turbostat23:49:553
289710799428416,9cyclictest2916913-21turbostat20:09:553
289710799427420,4cyclictest2909828-21turbostat19:49:553
289710299427419,5cyclictest3009238-21turbostat00:34:542
289710799426417,6cyclictest3010953-21turbostat00:35:003
289710799426417,6cyclictest2969264-21turbostat22:35:003
289710799425416,6cyclictest2935933-21turbostat21:04:553
289710799423414,6cyclictest2962356-21turbostat22:19:553
289710799418410,5cyclictest2916913-21turbostat20:05:003
289710799418410,5cyclictest2916913-21turbostat20:05:003
289709299417410,4cyclictest2979638-21turbostat23:09:560
289709299417410,4cyclictest2979638-21turbostat23:09:560
289710799416408,5cyclictest2977913-21turbostat23:00:013
289710799414407,4cyclictest2930757-21turbostat20:45:003
289710799412403,6cyclictest2957014-21turbostat22:00:003
289710799411404,4cyclictest3007503-21turbostat00:29:553
289710799411404,4cyclictest3007503-21turbostat00:29:553
289710799408388,17cyclictest2995201-21turbostat23:54:553
289710299405398,4cyclictest2955290-21turbostat21:59:552
289710799404383,18cyclictest3012690-21turbostat00:40:003
289710799402395,4cyclictest2998651-21turbostat00:00:003
289710799400393,4cyclictest2902759-21turbostat19:25:003
289710799399392,4cyclictest2981364-21turbostat23:10:003
289710799399392,4cyclictest2981364-21turbostat23:10:003
289710799394387,4cyclictest2896901-21turbostat19:14:553
289710799393386,4cyclictest2934208-21turbostat20:59:563
289710299393384,6cyclictest2902759-21turbostat19:29:552
289710799392385,4cyclictest2969264-21turbostat22:39:563
289710799380373,4cyclictest2923830-21turbostat20:25:003
289710799375367,5cyclictest2977913-21turbostat23:04:553
289710799374368,3cyclictest2904500-21turbostat19:30:003
289710799374365,6cyclictest3002180-21turbostat00:10:003
289710799368362,4cyclictest2983090-21turbostat23:15:003
289710799362354,5cyclictest3005770-21turbostat00:24:563
289710299362354,5cyclictest2951490-21turbostat21:49:562
289710299362354,5cyclictest2951490-21turbostat21:49:552
289710799355349,3cyclictest2991733-21turbostat23:40:003
289710299351345,3cyclictest2953420-21turbostat21:54:552
289710799350331,16cyclictest2955290-21turbostat21:55:003
289710799348342,3cyclictest2974455-21turbostat22:50:003
289710299336329,4cyclictest2918651-21turbostat20:14:552
289710799335329,4cyclictest3005770-21turbostat00:20:013
289710299326320,3cyclictest2911722-21turbostat19:54:552
289710799321315,3cyclictest2909828-21turbostat19:45:003
289710799314307,4cyclictest2990006-21turbostat23:35:003
289710799205187,14cyclictest2944581-21turbostat21:25:003
289710799205187,14cyclictest2944581-21turbostat21:25:003
289710799140133,4cyclictest2922102-21turbostat20:20:003
2896554210461,14sleep00-21swapper/019:05:540
296626721000,7sleep12897087-21cyclictest22:25:161
28971029910090,7cyclictest2988288-21turbostat23:34:562
2897102999891,4cyclictest3003902-21turbostat00:19:552
2897107999689,4cyclictest2929027-21turbostat20:40:003
2897102999487,4cyclictest2915191-21turbostat20:04:562
2897102999487,4cyclictest2915191-21turbostat20:04:552
289665029368,18sleep30-21swapper/319:07:103
289679329266,18sleep20-21swapper/219:09:022
289659729166,17sleep10-21swapper/119:06:271
30080252720,4sleep10-21swapper/100:25:131
30080252720,4sleep10-21swapper/100:25:131
29990552700,13sleep2341ksoftirqd/200:00:152
162399660,10rtkit-daemon1622-21rtkit-daemon20:31:262
162399660,10rtkit-daemon1622-21rtkit-daemon19:58:512
29440702650,4sleep20-21swapper/221:20:292
29440702650,4sleep20-21swapper/221:20:282
162399650,9rtkit-daemon1622-21rtkit-daemon00:35:522
162399650,17rtkit-daemon1622-21rtkit-daemon19:47:582
162399650,10rtkit-daemon1622-21rtkit-daemon21:15:582
162399640,9rtkit-daemon1622-21rtkit-daemon22:08:522
162399640,9rtkit-daemon1622-21rtkit-daemon21:38:442
162399630,9rtkit-daemon1622-21rtkit-daemon21:30:002
162399580,9rtkit-daemon1622-21rtkit-daemon00:25:252
162399580,9rtkit-daemon1622-21rtkit-daemon00:25:252
30051882560,5sleep0161rcuc/000:15:300
162399550,8rtkit-daemon1622-21rtkit-daemon22:56:561
29804432540,7sleep2331rcuc/223:05:202
29804432540,7sleep2331rcuc/223:05:202
29758072530,3chrt0-21swapper/222:50:322
2897102995030,10cyclictest2955150-21kworker/u8:122:04:072
289709499501,11cyclictest26-21ksoftirqd/120:35:001
2897094994912,20cyclictest26-21ksoftirqd/100:05:211
30075362480,5chrt0-21swapper/200:25:022
29600052480,6chrt0-21swapper/122:05:361
289709499488,18cyclictest26-21ksoftirqd/119:25:011
2897094994824,19cyclictest2901032-21/usr/sbin/munin19:20:001
2897092994814,13cyclictest141rcu_preempt20:10:130
29304142470,14sleep0131ksoftirqd/020:40:500
289709499479,20cyclictest26-21ksoftirqd/123:05:011
2897094994717,10cyclictest438-20systemd-journal22:35:011
2897094994711,20cyclictest26-21ksoftirqd/119:45:011
162399470,39rtkit-daemon1506-21systemd19:40:392
2897102994623,20cyclictest2935952-21awk20:59:592
289710299460,43cyclictest1622-21rtkit-daemon21:14:112
289709499465,8cyclictest26-21ksoftirqd/123:15:001
2897094994619,10cyclictest26-21ksoftirqd/100:00:021
2897094994610,20cyclictest26-21ksoftirqd/120:30:011
2897094994610,20cyclictest26-21ksoftirqd/120:30:011
2897092994620,22cyclictest2991733-21turbostat.cron23:40:000
2897094994534,8cyclictest26-21ksoftirqd/121:30:481
289709499452,16cyclictest26-21ksoftirqd/120:50:301
2897094994512,10cyclictest26-21ksoftirqd/100:34:331
2897094994510,19cyclictest26-21ksoftirqd/122:20:011
2897094994510,18cyclictest26-21ksoftirqd/123:19:591
2897092994515,6cyclictest1622-21rtkit-daemon23:28:560
162399450,32rtkit-daemon0-21swapper/121:07:551
2897102994421,19cyclictest969-21sshd21:44:192
289709499443,10cyclictest26-21ksoftirqd/119:15:011
289709499436,20cyclictest26-21ksoftirqd/122:54:591
289709499436,10cyclictest26-21ksoftirqd/122:40:121
289709499435,19cyclictest26-21ksoftirqd/120:39:591
289709499434,22cyclictest2962348-21rs:main1
289709499432,21cyclictest26-21ksoftirqd/123:05:111
289709499432,21cyclictest26-21ksoftirqd/123:05:111
2897094994319,9cyclictest26-21ksoftirqd/119:40:011
2897094994313,18cyclictest26-21ksoftirqd/119:50:001
162399430,37rtkit-daemon2287-21gmain22:57:160
2897102994216,6cyclictest141rcu_preempt22:55:152
2897102994215,8cyclictest30027242expr00:10:152
2897094994224,13cyclictest438-20systemd-journal23:40:001
2897094994215,10cyclictest26-21ksoftirqd/120:10:001
2897092994215,13cyclictest141rcu_preempt22:00:280
2897092994213,6cyclictest141rcu_preempt00:09:170
2897092994213,14cyclictest141rcu_preempt20:30:320
29537472410,6chrt26-21ksoftirqd/121:50:021
2897102994122,6cyclictest34-21ksoftirqd/223:55:022
2897102994122,14cyclictest34-21ksoftirqd/220:25:012
2897102994122,14cyclictest34-21ksoftirqd/219:40:002
289709499414,21cyclictest2945357-21cut21:25:351
289709499414,19cyclictest26-21ksoftirqd/121:45:011
289709499412,9cyclictest26-21ksoftirqd/121:50:021
289709499412,9cyclictest26-21ksoftirqd/121:50:021
2897094994123,12cyclictest438-20systemd-journal22:21:171
2897094994114,10cyclictest26-21ksoftirqd/121:15:001
289709499411,21cyclictest2913688-21seq19:55:031
289709499411,18cyclictest2988065-21seq23:27:291
2897092994114,6cyclictest141rcu_preempt22:25:120
2897092994113,7cyclictest141rcu_preempt22:35:340
162399410,27rtkit-daemon34-21ksoftirqd/223:10:312
2897102994023,6cyclictest34-21ksoftirqd/220:50:022
2897102994022,13cyclictest34-21ksoftirqd/220:05:142
2897102994021,14cyclictest34-21ksoftirqd/222:50:022
2897102994014,21cyclictest34-21ksoftirqd/200:00:032
289710299401,34cyclictest2967524-21cron22:30:002
289709499408,18cyclictest26-21ksoftirqd/123:44:591
289709499408,17cyclictest26-21ksoftirqd/120:05:001
289709499408,17cyclictest26-21ksoftirqd/120:05:001
289709499406,20cyclictest26-21ksoftirqd/100:25:001
289709499406,19cyclictest26-21ksoftirqd/119:35:011
289709499405,19cyclictest26-21ksoftirqd/100:15:221
289709499403,17cyclictest26-21ksoftirqd/121:15:151
2897094994031,6cyclictest251rcuc/119:25:221
289709499403,12cyclictest26-21ksoftirqd/100:35:191
289709499402,21cyclictest26-21ksoftirqd/122:45:111
2897094994022,13cyclictest2988909-21df23:30:161
289709499402,14cyclictest0-21swapper/123:21:561
289709499402,10cyclictest26-21ksoftirqd/123:50:291
2897094994012,8cyclictest26-21ksoftirqd/121:35:231
289709499401,12cyclictest26-21ksoftirqd/121:00:011
289709299401,5cyclictest141rcu_preempt20:20:310
2897092994015,6cyclictest141rcu_preempt20:45:100
2897092994015,6cyclictest141rcu_preempt20:00:150
2897092994015,6cyclictest141rcu_preempt20:00:150
2897092994014,6cyclictest141rcu_preempt23:55:280
2897092994014,6cyclictest141rcu_preempt23:50:180
2897092994013,6cyclictest141rcu_preempt21:10:180
2897102993922,13cyclictest34-21ksoftirqd/219:10:242
2897102993922,12cyclictest34-21ksoftirqd/220:30:022
2897102993922,12cyclictest34-21ksoftirqd/220:30:012
2897102993921,6cyclictest34-21ksoftirqd/223:25:092
289710299391,18cyclictest2992673-21needreboot23:40:212
289709499393,19cyclictest26-21ksoftirqd/119:55:021
289709499393,17cyclictest26-21ksoftirqd/100:10:171
289709499393,14cyclictest26-21ksoftirqd/120:20:131
2897094993912,7cyclictest26-21ksoftirqd/122:10:091
2897094993912,7cyclictest26-21ksoftirqd/122:10:091
289709499391,21cyclictest2932142-21taskset20:45:491
289709299395,11cyclictest13-21ksoftirqd/021:10:010
2897092993928,7cyclictest881395-21snap22:06:130
2897092993921,13cyclictest2954148-21df21:50:160
2897092993914,6cyclictest141rcu_preempt23:15:320
2897092993914,13cyclictest141rcu_preempt19:45:330
2897092993913,6cyclictest141rcu_preempt22:15:560
2897092993912,6cyclictest141rcu_preempt21:25:130
28977262380,5sleep00-21swapper/019:10:190
2897102993822,12cyclictest2978825-21needreboot23:00:262
289709499388,11cyclictest26-21ksoftirqd/120:15:211
289709499385,10cyclictest26-21ksoftirqd/100:00:041
289709499384,17cyclictest26-21ksoftirqd/123:45:291
289709499384,17cyclictest26-21ksoftirqd/120:10:261
289709299382,32cyclictest2999353-21hddtemp_smartct00:00:220
2897092993822,12cyclictest2952636-21python321:45:290
2897092993822,12cyclictest2952636-21python321:45:290
2897092993822,12cyclictest2942269-21smartctl21:15:290
2897092993822,12cyclictest2935556-21uptime20:55:320
2897092993821,13cyclictest2972023-21awk22:40:270
2897092993820,13cyclictest2899164-21df_inode19:15:030
2897092993814,6cyclictest141rcu_preempt00:40:010
2897092993813,6cyclictest141rcu_preempt23:45:190
2897092993813,6cyclictest141rcu_preempt20:25:320
2897092993813,6cyclictest141rcu_preempt20:25:320
2897092993811,6cyclictest141rcu_preempt22:22:290
2897092993810,10cyclictest881143-21gmain23:00:240
2897102993721,12cyclictest2933137-21forks20:50:182
2897102993720,5cyclictest34-21ksoftirqd/221:10:022
2897102993720,12cyclictest34-21ksoftirqd/223:45:182
2897102993718,15cyclictest34-21ksoftirqd/222:40:202
289710299371,15cyclictest1069-21runrttasks23:20:022
289709499372,20cyclictest26-21ksoftirqd/120:43:351
2897094993712,9cyclictest26-21ksoftirqd/122:35:171
2897094993710,5cyclictest141rcu_preempt21:55:341
2897092993723,11cyclictest3765663-21gnome-terminal-23:20:040
2897092993723,10cyclictest13-21ksoftirqd/022:45:120
2897092993722,11cyclictest2907202-21ntp_kernel_err19:35:240
2897092993721,12cyclictest2968014-21cat22:30:120
2897092993714,6cyclictest141rcu_preempt21:30:510
2897092993714,6cyclictest141rcu_preempt19:50:140
2897092993714,6cyclictest141rcu_preempt00:15:010
2897092993713,6cyclictest141rcu_preempt22:10:130
2897092993713,6cyclictest141rcu_preempt22:10:130
2897092993713,6cyclictest141rcu_preempt21:35:210
2897092993713,6cyclictest141rcu_preempt20:50:290
289709299371,32cyclictest3765717-21top23:43:100
2897092993712,6cyclictest141rcu_preempt23:35:020
2897092993712,6cyclictest141rcu_preempt20:05:150
2897092993712,6cyclictest141rcu_preempt00:25:230
2897092993712,6cyclictest141rcu_preempt00:25:220
2897092993711,6cyclictest141rcu_preempt21:00:160
2897092993711,5cyclictest141rcu_preempt19:25:020
2897102993625,6cyclictest34-21ksoftirqd/222:10:102
2897102993625,6cyclictest34-21ksoftirqd/222:10:092
2897102993621,11cyclictest2962922-21ls22:15:122
2897102993620,11cyclictest34-21ksoftirqd/219:30:132
2897102993618,13cyclictest34-21ksoftirqd/222:35:162
2897102993617,14cyclictest34-21ksoftirqd/219:15:202
2897102993617,13cyclictest34-21ksoftirqd/223:20:142
289709499361,11cyclictest26-21ksoftirqd/122:03:331
2897092993621,11cyclictest13-21ksoftirqd/019:55:280
2897092993620,12cyclictest3009765-21cat00:30:120
2897092993620,11cyclictest2903981-21threads19:25:290
2897092993616,14cyclictest2922316-20journal-offline20:20:020
289709299361,15cyclictest2927870-21cat20:35:130
2897102993520,11cyclictest2964327-21cat22:20:092
2897102993518,13cyclictest34-21ksoftirqd/221:00:222
289709299359,6cyclictest141rcu_preempt21:43:170
2897092993519,12cyclictest3006308-21expr00:20:130
2897092993513,5cyclictest141rcu_preempt21:55:130
2897092993513,5cyclictest141rcu_preempt21:20:110
2897092993513,5cyclictest141rcu_preempt21:20:110
2897102993410,6cyclictest141rcu_preempt20:40:142
2897092993412,6cyclictest141rcu_preempt22:50:290
2897092993319,10cyclictest2904992-21expr19:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional