You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 04:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Wed Apr 02, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
94980940,88rtkit-daemon0-21swapper/1019:06:152
275991870,77ptp4l0-21swapper/119:06:331
1766026654,10sleep50-21swapper/519:06:107
1785426554,9sleep110-21swapper/1119:09:053
1697426553,9sleep70-21swapper/719:05:249
1782326352,9sleep60-21swapper/619:08:368
1761926352,9sleep40-21swapper/419:05:336
1579526250,9sleep00-21swapper/019:05:180
48222550,0sleep70-21swapper/721:05:009
18335993426,8cyclictest0-21swapper/1022:20:242
1830699243,16cyclictest0-21swapper/422:00:026
1829799213,11cyclictest0-21swapper/222:55:004
1830699192,13cyclictest0-21swapper/400:35:006
18335991814,1cyclictest0-21swapper/1022:50:172
18335991810,8cyclictest0-21swapper/1023:15:162
1828899172,10cyclictest0-21swapper/021:35:010
141022177,7sleep80-21swapper/819:05:0110
53072160,1sleep8621rcuc/819:30:2710
53072160,1sleep8621rcuc/819:30:2710
47332160,0sleep70-21swapper/722:44:599
1828899163,7cyclictest0-21swapper/023:25:000
1828899162,9cyclictest0-21swapper/019:25:010
1828899162,9cyclictest0-21swapper/019:25:000
7862150,0sleep90-21swapper/900:20:2411
275991150,1ptp4l21-21ksoftirqd/121:10:151
1829799153,8cyclictest0-21swapper/200:15:004
1829799152,7cyclictest0-21swapper/220:20:004
1828899153,7cyclictest0-21swapper/000:20:010
1828899152,9cyclictest0-21swapper/019:50:010
1828899152,9cyclictest0-21swapper/019:50:000
1828899152,8cyclictest0-21swapper/023:50:000
275991140,1ptp4l21-21ksoftirqd/120:04:561
275991140,0ptp4l0-21swapper/123:45:001
275991140,0ptp4l0-21swapper/123:35:241
275991140,0ptp4l0-21swapper/121:20:221
1833599141,8cyclictest0-21swapper/1021:30:012
1833099140,13cyclictest0-21swapper/921:04:5611
1832599143,7cyclictest0-21swapper/821:40:0110
1832599143,6cyclictest0-21swapper/820:55:0110
1832599142,6cyclictest0-21swapper/820:30:1910
1829799143,7cyclictest0-21swapper/221:42:174
1829799142,8cyclictest0-21swapper/221:15:014
1829299140,13cyclictest31956-21if_err_enp1s023:30:121
1828899140,2cyclictest2853-21df_inode23:35:150
1828899140,13cyclictest0-21swapper/019:15:170
275991130,7ptp4l21-21ksoftirqd/122:20:231
275991130,1ptp4l21-21ksoftirqd/123:50:091
275991130,1ptp4l21-21ksoftirqd/122:13:251
275991130,1ptp4l21-21ksoftirqd/121:50:081
275991130,0ptp4l0-21swapper/119:50:211
275991130,0ptp4l0-21swapper/119:40:171
275991130,0ptp4l0-21swapper/119:40:171
18341991311,1cyclictest0-21swapper/1122:39:593
1832599133,6cyclictest0-21swapper/820:45:0110
1832099130,12cyclictest30101-21chrt00:19:359
1831699131,7cyclictest0-21swapper/623:15:018
18310991310,2cyclictest475-21turbostat22:39:597
1828899138,4cyclictest0-21swapper/020:20:140
1828899133,8cyclictest0-21swapper/021:05:010
1828899132,7cyclictest0-21swapper/021:45:020
1828899132,6cyclictest0-21swapper/020:35:230
18288991310,2cyclictest16849-21turbostat22:15:000
1828899130,1cyclictest0-21swapper/022:40:000
275991120,1ptp4l21-21ksoftirqd/123:06:461
275991120,1ptp4l21-21ksoftirqd/121:44:491
275991120,0ptp4l0-21swapper/123:49:391
275991120,0ptp4l0-21swapper/123:10:221
275991120,0ptp4l0-21swapper/122:55:191
275991120,0ptp4l0-21swapper/122:50:081
275991120,0ptp4l0-21swapper/122:40:541
275991120,0ptp4l0-21swapper/122:15:141
275991120,0ptp4l0-21swapper/121:55:251
275991120,0ptp4l0-21swapper/121:16:331
275991120,0ptp4l0-21swapper/120:57:311
275991120,0ptp4l0-21swapper/119:49:131
275991120,0ptp4l0-21swapper/119:49:121
275991120,0ptp4l0-21swapper/119:30:151
275991120,0ptp4l0-21swapper/119:30:151
275991120,0ptp4l0-21swapper/119:25:181
275991120,0ptp4l0-21swapper/119:25:171
275991120,0ptp4l0-21swapper/119:20:131
275991120,0ptp4l0-21swapper/119:20:131
18341991210,1cyclictest0-21swapper/1119:25:013
18341991210,1cyclictest0-21swapper/1119:25:003
1832599122,6cyclictest0-21swapper/819:30:0110
1832599122,6cyclictest0-21swapper/819:30:0010
1832599120,11cyclictest0-21swapper/822:10:0110
1832099120,11cyclictest0-21swapper/723:30:139
1831699122,6cyclictest0-21swapper/620:15:008
1830199120,11cyclictest32150-21irqrtprio23:30:125
1829799122,6cyclictest0-21swapper/222:45:014
1828899123,3cyclictest13737-21latency_hist19:45:000
1828899123,3cyclictest13737-21latency_hist19:45:000
1828899122,6cyclictest0-21swapper/023:55:010
1828899120,6cyclictest0-21swapper/022:50:000
275991110,2ptp4l853-21cat22:40:001
275991110,1ptp4l21-21ksoftirqd/119:35:591
275991110,1ptp4l21-21ksoftirqd/119:35:581
275991110,1ptp4l0-21swapper/122:30:091
275991110,1ptp4l0-21swapper/120:35:161
275991110,1ptp4l0-21swapper/119:55:161
275991110,1ptp4l0-21swapper/119:15:011
275991110,1ptp4l0-21swapper/119:10:181
275991110,0ptp4l0-21swapper/120:40:011
275991110,0ptp4l0-21swapper/120:24:391
18320991110,1cyclictest0-21swapper/722:53:449
18320991110,1cyclictest0-21swapper/721:35:009
18320991110,1cyclictest0-21swapper/721:05:229
1832099110,1cyclictest8299-21diskmemload21:57:209
1832099110,1cyclictest0-21swapper/723:54:019
1832099110,1cyclictest0-21swapper/723:41:349
1832099110,1cyclictest0-21swapper/723:07:459
1832099110,1cyclictest0-21swapper/722:45:269
1832099110,1cyclictest0-21swapper/722:17:509
1832099110,1cyclictest0-21swapper/722:08:459
1832099110,1cyclictest0-21swapper/721:26:479
1832099110,1cyclictest0-21swapper/721:16:239
1832099110,11cyclictest0-21swapper/723:55:159
1832099110,11cyclictest0-21swapper/721:10:249
1831699112,6cyclictest0-21swapper/619:25:008
1831699112,6cyclictest0-21swapper/619:25:008
1831099118,2cyclictest30594-21turbostat19:25:017
1831099118,2cyclictest30594-21turbostat19:25:007
1830199110,7cyclictest0-21swapper/323:55:005
1830199110,7cyclictest0-21swapper/319:15:005
1829799110,2cyclictest4152-21memory23:35:204
1829799110,10cyclictest8299-21diskmemload22:10:144
1829299110,1cyclictest0-21swapper/123:55:141
1829299110,1cyclictest0-21swapper/121:30:261
1829299110,1cyclictest0-21swapper/100:10:321
1829299110,1cyclictest0-21swapper/100:01:491
1829299110,10cyclictest0-21swapper/123:15:591
1829299110,10cyclictest0-21swapper/100:17:131
1828899118,2cyclictest29132-21turbostat22:34:590
1828899116,4cyclictest21332-21turbostat19:55:000
1828899112,4cyclictest12846-21chrt22:56:300
1828899111,2cyclictest0-21swapper/023:30:090
1828899111,2cyclictest0-21swapper/023:00:150
1828899111,2cyclictest0-21swapper/021:20:190
1828899111,1cyclictest0-21swapper/020:55:240
1828899111,1cyclictest0-21swapper/000:00:260
1828899110,3cyclictest26056-21fschecks_time21:35:150
1828899110,3cyclictest12106-21latency_hist21:15:000
1828899110,2cyclictest13705-21chrt20:34:260
1828899110,2cyclictest0-21swapper/023:55:140
1828899110,2cyclictest0-21swapper/023:40:130
1828899110,2cyclictest0-21swapper/022:40:140
1828899110,2cyclictest0-21swapper/022:00:110
1828899110,2cyclictest0-21swapper/021:40:140
1828899110,2cyclictest0-21swapper/021:05:170
1828899110,2cyclictest0-21swapper/020:45:250
1828899110,2cyclictest0-21swapper/000:35:200
275991100,1ptp4l21-21ksoftirqd/120:50:351
275991100,1ptp4l1174-21awk00:20:251
275991100,1ptp4l0-21swapper/123:20:171
275991100,1ptp4l0-21swapper/100:35:151
275991100,0ptp4l0-21swapper/123:00:151
275991100,0ptp4l0-21swapper/121:25:011
275991100,0ptp4l0-21swapper/121:00:151
275991100,0ptp4l0-21swapper/120:45:251
275991100,0ptp4l0-21swapper/120:12:251
1834199100,9cyclictest0-21swapper/1121:30:003
1833599107,2cyclictest21834-21turbostat19:15:002
1833599102,5cyclictest0-21swapper/1022:05:022
1833599100,1cyclictest0-21swapper/1019:45:152
1833599100,1cyclictest0-21swapper/1019:45:142
1833099100,9cyclictest1009-21snmpd21:55:2611
1833099100,2cyclictest0-21swapper/923:50:1311
1832099109,1cyclictest18412-21aten2_r3power_e00:00:129
1832099109,1cyclictest0-21swapper/723:35:499
1832099109,1cyclictest0-21swapper/723:20:059
1832099109,1cyclictest0-21swapper/723:15:299
1832099109,1cyclictest0-21swapper/722:35:359
1832099109,1cyclictest0-21swapper/722:20:229
1832099109,1cyclictest0-21swapper/722:02:179
1832099109,1cyclictest0-21swapper/721:40:119
1832099109,1cyclictest0-21swapper/721:35:329
1832099109,1cyclictest0-21swapper/700:28:119
1832099100,1cyclictest0-21swapper/723:10:449
1832099100,1cyclictest0-21swapper/723:00:069
1832099100,1cyclictest0-21swapper/722:30:089
1832099100,1cyclictest0-21swapper/721:45:099
1832099100,1cyclictest0-21swapper/700:36:489
1832099100,1cyclictest0-21swapper/700:06:179
1832099100,10cyclictest0-21swapper/723:46:339
1832099100,10cyclictest0-21swapper/723:26:129
1832099100,10cyclictest0-21swapper/722:55:069
1832099100,10cyclictest0-21swapper/722:26:149
1832099100,10cyclictest0-21swapper/722:13:169
1832099100,10cyclictest0-21swapper/721:50:209
1832099100,10cyclictest0-21swapper/721:20:049
1832099100,10cyclictest0-21swapper/700:31:539
1832099100,10cyclictest0-21swapper/700:21:049
1832099100,10cyclictest0-21swapper/700:10:319
1831699101,6cyclictest0-21swapper/620:45:018
1831699100,1cyclictest1527-21timerwakeupswit19:25:268
1831699100,1cyclictest1527-21timerwakeupswit19:25:258
1831099107,2cyclictest22802-21diskstats19:15:027
1831099100,9cyclictest3207-21fw_forwarded_lo23:35:187
1830699102,5cyclictest0-21swapper/423:35:006
1830699100,5cyclictest0-21swapper/422:20:026
1830199100,6cyclictest0-21swapper/323:40:005
1830199100,6cyclictest0-21swapper/323:10:005
1830199100,6cyclictest0-21swapper/320:00:005
1829799103,6cyclictest0-21swapper/220:00:254
1829799102,6cyclictest0-21swapper/200:15:004
1829799101,6cyclictest0-21swapper/219:20:264
1829799101,6cyclictest0-21swapper/219:20:254
1829799100,9cyclictest15070-21cstates20:35:134
1829299109,1cyclictest0-21swapper/123:25:431
1829299109,1cyclictest0-21swapper/122:45:331
1829299109,1cyclictest0-21swapper/122:25:361
1829299109,1cyclictest0-21swapper/122:01:091
1829299109,1cyclictest0-21swapper/121:45:161
1829299109,1cyclictest0-21swapper/100:31:241
1829299109,1cyclictest0-21swapper/100:25:071
1829299109,1cyclictest0-21swapper/100:05:401
1829299108,1cyclictest0-21swapper/121:35:151
1829299100,1cyclictest0-21swapper/122:05:501
1829299100,1cyclictest0-21swapper/121:05:361
1828899107,2cyclictest0-21swapper/019:35:140
1828899107,2cyclictest0-21swapper/019:35:130
1828899102,2cyclictest12729-21basename21:15:120
1828899100,3cyclictest31279-21grep20:10:120
1828899100,3cyclictest21722-21cstates22:20:130
1828899100,3cyclictest20496-21taskset21:28:330
1828899100,3cyclictest17540-21chrt23:06:440
1828899100,2cyclictest9825-21chrt20:28:250
1828899100,2cyclictest5222-21chrt22:46:300
1828899100,2cyclictest300442sleep023:29:570
1828899100,2cyclictest25772-21grep20:00:140
1828899100,2cyclictest2296-21chrt00:27:400
1828899100,2cyclictest18800-21cpuspeed_turbos19:10:150
1828899100,2cyclictest0-21swapper/023:15:140
1828899100,2cyclictest0-21swapper/023:10:090
1828899100,2cyclictest0-21swapper/022:15:120
1828899100,2cyclictest0-21swapper/021:55:170
1828899100,2cyclictest0-21swapper/021:50:090
1828899100,2cyclictest0-21swapper/020:50:210
1828899100,2cyclictest0-21swapper/020:40:140
1828899100,2cyclictest0-21swapper/020:15:200
1828899100,2cyclictest0-21swapper/020:10:000
1828899100,2cyclictest0-21swapper/019:55:130
1828899100,2cyclictest0-21swapper/019:30:160
1828899100,2cyclictest0-21swapper/019:30:160
1828899100,2cyclictest0-21swapper/019:20:140
1828899100,2cyclictest0-21swapper/019:20:140
1828899100,2cyclictest0-21swapper/000:30:160
1828899100,2cyclictest0-21swapper/000:20:130
1828899100,2cyclictest0-21swapper/000:10:120
1828899100,1cyclictest0-21swapper/022:09:050
9498090,4rtkit-daemon0-21swapper/1000:27:282
27599190,0ptp4l0-21swapper/120:30:081
27599190,0ptp4l0-21swapper/120:25:011
27599190,0ptp4l0-21swapper/120:15:021
27599190,0ptp4l0-21swapper/120:05:011
183419990,9cyclictest0-21swapper/1120:40:263
183419990,9cyclictest0-21swapper/1120:00:193
183359998,1cyclictest0-21swapper/1019:25:202
183359998,1cyclictest0-21swapper/1019:25:192
183359997,1cyclictest0-21swapper/1023:00:142
183359991,5cyclictest0-21swapper/1022:30:002
183359990,1cyclictest0-21swapper/1020:40:232
183309990,1cyclictest0-21swapper/919:20:2211
183309990,1cyclictest0-21swapper/919:20:2211
183259991,5cyclictest0-21swapper/821:15:1510
183259990,9cyclictest0-21swapper/821:00:5910
183169993,5cyclictest0-21swapper/622:34:558
183169992,4cyclictest0-21swapper/623:00:008
183169992,4cyclictest0-21swapper/620:50:018
183109990,8cyclictest3938-21diskstats00:30:147
183109990,1cyclictest0-21swapper/520:10:017
183069997,1cyclictest0-21swapper/423:35:166
183069994,3cyclictest94980rtkit-daemon00:17:286
183069992,5cyclictest11246-21cat22:55:206
183069990,1cyclictest0-21swapper/422:50:136
183019990,9cyclictest0-21swapper/323:00:145
183019990,9cyclictest0-21swapper/300:35:205
183019990,1cyclictest0-21swapper/322:40:195
183019990,1cyclictest0-21swapper/320:15:215
182979998,1cyclictest0-21swapper/223:55:194
182979998,1cyclictest0-21swapper/219:40:214
182979998,1cyclictest0-21swapper/219:40:204
182979992,5cyclictest0-21swapper/200:10:014
182979992,4cyclictest0-21swapper/200:25:004
182979991,6cyclictest0-21swapper/222:10:004
182979990,8cyclictest21189-21/usr/sbin/munin22:20:234
182979990,8cyclictest14070-21cpuspeed_turbos23:00:134
182889990,3cyclictest22226-21chrt00:05:370
182889990,2cyclictest250852sleep022:26:270
27609183,3phc2sys16581-21missed_timers19:05:204
183419987,1cyclictest0-21swapper/1100:30:143
183419985,2cyclictest1586-21munin-run00:25:003
183419980,8cyclictest0-21swapper/1119:30:213
183419980,8cyclictest0-21swapper/1119:30:203
183419980,1cyclictest0-21swapper/1123:11:173
183419980,1cyclictest0-21swapper/1122:40:163
183419980,1cyclictest0-21swapper/1121:05:143
183359987,1cyclictest0-21swapper/1020:55:252
183359987,1cyclictest0-21swapper/1019:40:182
183359987,1cyclictest0-21swapper/1019:40:172
183359982,5cyclictest0-21swapper/1023:55:002
183359980,8cyclictest0-21swapper/1021:10:212
183359980,8cyclictest0-21swapper/1020:20:182
183309987,1cyclictest0-21swapper/922:40:1911
183309987,1cyclictest0-21swapper/921:10:2811
183309987,1cyclictest0-21swapper/900:14:5611
183309986,1cyclictest8299-21diskmemload23:20:5211
183309984,3cyclictest0-21swapper/919:45:0011
183309984,3cyclictest0-21swapper/919:45:0011
183309980,8cyclictest0-21swapper/900:35:2211
183309980,1cyclictest0-21swapper/922:55:1911
183309980,1cyclictest0-21swapper/919:35:1711
183309980,1cyclictest0-21swapper/919:35:1611
183309980,1cyclictest0-21swapper/919:30:2111
183309980,1cyclictest0-21swapper/919:30:2111
183259987,1cyclictest0-21swapper/820:27:5910
183259987,1cyclictest0-21swapper/820:20:1310
183259981,6cyclictest0-21swapper/819:55:0010
183259980,8cyclictest0-21swapper/823:40:1710
183259980,7cyclictest0-21swapper/821:39:4110
183259980,7cyclictest0-21swapper/821:14:3910
183209987,1cyclictest0-21swapper/719:15:159
183209980,8cyclictest0-21swapper/719:41:259
183209980,8cyclictest0-21swapper/719:41:259
183209980,1cyclictest0-21swapper/720:30:589
183209980,1cyclictest0-21swapper/719:20:249
183209980,1cyclictest0-21swapper/719:20:249
183169987,1cyclictest0-21swapper/623:00:178
183169987,1cyclictest0-21swapper/622:49:598
183169987,1cyclictest0-21swapper/621:50:158
183169981,4cyclictest0-21swapper/623:30:018
183169981,4cyclictest0-21swapper/600:00:018
183169980,7cyclictest25749-21forks21:35:158
183169980,1cyclictest0-21swapper/622:40:008
183169980,1cyclictest0-21swapper/621:15:188
183169980,1cyclictest0-21swapper/619:50:278
183169980,1cyclictest0-21swapper/619:30:208
183169980,1cyclictest0-21swapper/619:30:208
183109987,1cyclictest0-21swapper/520:55:197
183109980,8cyclictest0-21swapper/522:40:207
183109980,8cyclictest0-21swapper/521:30:197
183109980,1cyclictest0-21swapper/522:29:507
183109980,1cyclictest0-21swapper/520:35:257
183109980,1cyclictest0-21swapper/500:15:277
183069987,1cyclictest0-21swapper/423:20:156
183069980,8cyclictest0-21swapper/420:50:316
183069980,8cyclictest0-21swapper/420:00:196
183069980,7cyclictest16948-21sed23:04:596
183069980,1cyclictest0-21swapper/419:15:186
183069980,1cyclictest0-21swapper/400:30:006
183019987,1cyclictest0-21swapper/321:10:205
183019987,1cyclictest0-21swapper/320:20:255
183019987,1cyclictest0-21swapper/300:10:105
183019980,5cyclictest0-21swapper/323:55:005
183019980,5cyclictest0-21swapper/323:50:005
183019980,5cyclictest0-21swapper/322:50:005
183019980,5cyclictest0-21swapper/322:40:015
183019980,5cyclictest0-21swapper/322:10:015
183019980,5cyclictest0-21swapper/321:40:015
183019980,1cyclictest0-21swapper/323:15:195
183019980,1cyclictest0-21swapper/323:10:095
182979987,1cyclictest0-21swapper/219:50:314
182979987,1cyclictest0-21swapper/219:35:254
182979987,1cyclictest0-21swapper/219:35:254
182979980,8cyclictest0-21swapper/223:30:124
182979980,8cyclictest0-21swapper/221:55:234
182979980,8cyclictest0-21swapper/220:40:174
182979980,8cyclictest0-21swapper/219:55:204
182979980,8cyclictest0-21swapper/219:45:144
182979980,8cyclictest0-21swapper/219:45:144
182979980,7cyclictest3873-21sed21:00:244
182979980,7cyclictest32009-21/usr/sbin/munin19:25:154
182979980,7cyclictest32009-21/usr/sbin/munin19:25:154
182979980,7cyclictest27680-21sed21:35:254
182979980,7cyclictest0-21swapper/222:15:254
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional