You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 04:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Apr 02, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4801210591,9sleep30-21swapper/319:09:463
459829986,9sleep10-21swapper/119:07:061
457728963,8sleep00-21swapper/019:06:500
450128868,16sleep20-21swapper/219:05:502
4914993027,2cyclictest32-21ksoftirqd/200:18:162
4909993024,4cyclictest13-21ksoftirqd/023:44:500
4914992924,4cyclictest6131-21hostname22:55:152
4918992826,1cyclictest39-21ksoftirqd/321:28:133
4912992825,2cyclictest33652sleep100:05:521
4909992823,4cyclictest13-21ksoftirqd/023:23:070
4914992724,2cyclictest119202sleep221:21:442
4912992724,2cyclictest25-21ksoftirqd/123:19:351
4918992624,1cyclictest39-21ksoftirqd/323:30:443
4918992623,2cyclictest39-21ksoftirqd/323:59:223
4918992623,2cyclictest39-21ksoftirqd/323:43:033
4918992623,2cyclictest39-21ksoftirqd/322:46:593
4918992623,2cyclictest39-21ksoftirqd/322:35:583
4918992623,2cyclictest39-21ksoftirqd/300:38:133
4918992623,2cyclictest39-21ksoftirqd/300:27:363
4914992624,2cyclictest32-21ksoftirqd/223:09:572
4912992623,2cyclictest25-21ksoftirqd/123:27:551
4912992623,2cyclictest25-21ksoftirqd/123:07:151
4912992623,2cyclictest25-21ksoftirqd/122:27:121
4912992623,2cyclictest25-21ksoftirqd/121:32:231
4912992623,2cyclictest25-21ksoftirqd/121:19:181
4912992623,2cyclictest25-21ksoftirqd/100:34:181
4912992622,3cyclictest25-21ksoftirqd/100:29:151
4909992624,1cyclictest13-21ksoftirqd/000:19:380
4909992623,2cyclictest13-21ksoftirqd/023:57:380
4909992623,2cyclictest13-21ksoftirqd/023:26:090
4909992623,2cyclictest13-21ksoftirqd/022:31:040
491899254,12cyclictest0-21swapper/322:20:133
4918992524,1cyclictest39-21ksoftirqd/322:04:053
4918992523,2cyclictest39-21ksoftirqd/323:52:223
4918992523,2cyclictest39-21ksoftirqd/323:09:203
4918992522,2cyclictest39-21ksoftirqd/322:50:143
4918992522,2cyclictest39-21ksoftirqd/300:21:243
4918992522,2cyclictest39-21ksoftirqd/300:02:083
4918992521,3cyclictest6174-21sed22:30:223
4914992523,2cyclictest32-21ksoftirqd/223:48:512
4914992523,2cyclictest32-21ksoftirqd/222:10:582
4914992523,1cyclictest1291-21runrttasks22:02:242
4914992522,2cyclictest37850irq/131-snd_hda22:50:142
4914992522,2cyclictest32-21ksoftirqd/222:05:452
4914992522,2cyclictest32-21ksoftirqd/221:16:072
4914992521,3cyclictest17345-21timerandwakeup21:50:212
4914992521,3cyclictest17345-21timerandwakeup21:50:202
4912992523,2cyclictest25-21ksoftirqd/122:55:481
4912992523,2cyclictest25-21ksoftirqd/122:43:241
4912992523,1cyclictest22251-21tail23:55:181
4912992522,2cyclictest25-21ksoftirqd/123:03:251
4912992522,2cyclictest25-21ksoftirqd/122:22:281
4912992522,2cyclictest25-21ksoftirqd/122:17:461
4912992522,2cyclictest25-21ksoftirqd/122:10:431
4912992522,2cyclictest25-21ksoftirqd/121:53:471
4912992522,2cyclictest25-21ksoftirqd/121:53:471
4912992522,2cyclictest25-21ksoftirqd/121:42:591
4912992522,2cyclictest25-21ksoftirqd/100:12:031
4912992521,3cyclictest24101-21missed_timers21:55:211
491299250,1cyclictest0-21swapper/121:21:141
4909992523,2cyclictest13-21ksoftirqd/022:48:260
4909992523,2cyclictest13-21ksoftirqd/022:38:330
4909992523,2cyclictest13-21ksoftirqd/022:04:070
4909992523,2cyclictest13-21ksoftirqd/021:15:580
4909992523,2cyclictest13-21ksoftirqd/000:32:010
4909992523,1cyclictest4551-21grep21:40:230
4909992523,1cyclictest13-21ksoftirqd/023:47:530
4909992523,1cyclictest1291-21runrttasks21:46:020
4909992523,1cyclictest1291-21runrttasks21:21:200
4909992522,2cyclictest13-21ksoftirqd/022:24:250
4909992522,2cyclictest13-21ksoftirqd/021:27:160
4909992515,6cyclictest13-21ksoftirqd/022:43:480
4918992422,2cyclictest39-21ksoftirqd/323:20:173
4918992422,2cyclictest39-21ksoftirqd/323:12:123
4918992422,2cyclictest39-21ksoftirqd/322:14:303
4918992422,2cyclictest39-21ksoftirqd/321:54:173
4918992422,2cyclictest39-21ksoftirqd/321:54:163
4918992422,2cyclictest39-21ksoftirqd/321:31:373
4918992422,1cyclictest39-21ksoftirqd/322:05:243
4918992422,1cyclictest39-21ksoftirqd/300:30:073
4918992421,2cyclictest39-21ksoftirqd/322:17:563
4918992421,2cyclictest39-21ksoftirqd/321:58:223
4918992421,2cyclictest39-21ksoftirqd/321:45:213
4918992421,2cyclictest39-21ksoftirqd/321:36:063
4918992421,2cyclictest39-21ksoftirqd/321:14:383
4918992421,2cyclictest19186-21cstates22:40:163
4918992420,3cyclictest3435-21cstates21:40:153
4914992422,2cyclictest32-21ksoftirqd/223:41:062
4914992422,2cyclictest32-21ksoftirqd/222:47:092
4914992422,2cyclictest32-21ksoftirqd/222:20:152
4914992422,2cyclictest32-21ksoftirqd/221:38:182
4914992422,2cyclictest32-21ksoftirqd/221:30:502
4914992422,2cyclictest32-21ksoftirqd/200:30:332
4914992421,2cyclictest32-21ksoftirqd/222:25:202
491499240,22cyclictest0-21swapper/222:34:222
4912992423,1cyclictest19905-21apache223:12:361
4912992421,2cyclictest25-21ksoftirqd/123:46:051
4912992421,2cyclictest25-21ksoftirqd/122:37:021
4912992421,2cyclictest25-21ksoftirqd/121:47:051
4912992421,2cyclictest25-21ksoftirqd/100:20:541
4912992421,2cyclictest25-21ksoftirqd/100:17:501
4912992421,2cyclictest25-21ksoftirqd/100:03:431
4909992422,2cyclictest13-21ksoftirqd/022:15:570
4909992422,2cyclictest13-21ksoftirqd/022:10:460
4909992422,2cyclictest13-21ksoftirqd/021:32:420
4909992422,2cyclictest13-21ksoftirqd/000:24:550
4909992422,2cyclictest13-21ksoftirqd/000:05:120
4909992421,2cyclictest13-21ksoftirqd/023:53:150
4909992421,2cyclictest13-21ksoftirqd/022:57:340
4909992421,2cyclictest13-21ksoftirqd/021:55:020
4909992421,2cyclictest13-21ksoftirqd/000:01:270
4909992420,3cyclictest31319-21apt-get22:50:090
4918992321,2cyclictest39-21ksoftirqd/321:17:423
4918992321,1cyclictest6308-21cpuspeed_turbos22:55:163
4918992320,2cyclictest39-21ksoftirqd/323:47:443
4914992321,2cyclictest32-21ksoftirqd/223:21:042
4914992321,2cyclictest32-21ksoftirqd/222:19:272
4914992321,2cyclictest32-21ksoftirqd/221:46:442
4914992320,2cyclictest32-21ksoftirqd/200:37:292
4914992319,3cyclictest27210-21unixbench_singl23:10:222
491499230,23cyclictest0-21swapper/223:39:082
491499230,23cyclictest0-21swapper/223:39:072
491499230,23cyclictest0-21swapper/200:05:372
491499230,22cyclictest0-21swapper/223:00:182
4912992321,2cyclictest25-21ksoftirqd/123:52:071
491299230,22cyclictest0-21swapper/123:22:531
491299230,22cyclictest0-21swapper/122:47:031
491299230,22cyclictest0-21swapper/122:31:191
4909992321,2cyclictest13-21ksoftirqd/021:11:200
4909992321,2cyclictest13-21ksoftirqd/000:14:520
4909992320,2cyclictest13-21ksoftirqd/023:31:230
4909992320,2cyclictest13-21ksoftirqd/022:29:060
4909992320,2cyclictest13-21ksoftirqd/000:27:320
4909992319,3cyclictest16797-21missed_timers21:50:160
4909992319,3cyclictest16797-21missed_timers21:50:160
4909992319,3cyclictest13-21ksoftirqd/023:35:080
4909992319,3cyclictest13-21ksoftirqd/023:35:080
490999230,22cyclictest0-21swapper/022:09:340
4918992221,1cyclictest39-21ksoftirqd/323:35:473
4918992221,1cyclictest39-21ksoftirqd/323:35:463
4918992220,2cyclictest39-21ksoftirqd/300:19:183
491899220,21cyclictest0-21swapper/300:07:313
4914992220,2cyclictest32-21ksoftirqd/223:30:332
491499220,22cyclictest0-21swapper/223:17:192
491499220,19cyclictest0-21swapper/223:27:462
4912992220,2cyclictest25-21ksoftirqd/123:39:451
4912992220,2cyclictest25-21ksoftirqd/123:39:441
4912992220,2cyclictest25-21ksoftirqd/121:26:011
4912992220,2cyclictest25-21ksoftirqd/100:36:231
4912992220,1cyclictest3415-21sshd23:40:241
4912992219,2cyclictest25-21ksoftirqd/123:33:521
4912992219,2cyclictest25-21ksoftirqd/122:01:251
4909992220,1cyclictest1291-21runrttasks23:05:200
4918992118,2cyclictest39-21ksoftirqd/322:29:473
491899210,21cyclictest0-21swapper/323:03:133
4914992119,2cyclictest32-21ksoftirqd/200:10:222
4914992119,1cyclictest16260-21kworker/2:223:58:082
4912992118,2cyclictest25-21ksoftirqd/122:54:141
4912992118,2cyclictest25-21ksoftirqd/121:37:191
4909992119,2cyclictest13-21ksoftirqd/021:35:380
4909992119,2cyclictest13-21ksoftirqd/000:38:570
4909992118,2cyclictest13-21ksoftirqd/023:15:390
4909992116,4cyclictest13-21ksoftirqd/023:10:080
4914992018,2cyclictest32-21ksoftirqd/221:41:072
4914992017,2cyclictest32-21ksoftirqd/222:35:442
4909992017,2cyclictest13-21ksoftirqd/023:00:100
4918991917,2cyclictest39-21ksoftirqd/321:24:393
4918991917,2cyclictest39-21ksoftirqd/300:12:413
4914991917,2cyclictest32-21ksoftirqd/200:20:202
4912991916,2cyclictest26840-21/usr/sbin/munin21:10:201
4912991916,2cyclictest25-21ksoftirqd/122:07:131
4914991817,1cyclictest32-21ksoftirqd/222:41:002
4914991816,2cyclictest32-21ksoftirqd/221:28:402
4914991815,3cyclictest32-21ksoftirqd/221:56:062
491499180,18cyclictest26077-21kworker/2:021:10:482
4918991714,2cyclictest39-21ksoftirqd/323:18:473
4914991715,2cyclictest32-21ksoftirqd/223:52:302
4909991715,1cyclictest161rcuc/019:55:270
4909991715,1cyclictest161rcuc/019:55:260
4914991614,2cyclictest32-21ksoftirqd/200:27:312
490999164,6cyclictest23965-21munin-run19:50:000
491899154,8cyclictest11726-21packagekitd19:55:213
491899154,8cyclictest11726-21packagekitd19:55:213
4914991513,1cyclictest12721-21expr19:25:162
4918991413,1cyclictest0-21swapper/320:52:183
4914991412,2cyclictest32-21ksoftirqd/200:00:442
491499140,14cyclictest14253-21kworker/2:219:36:022
491899139,3cyclictest13956-21grep19:25:273
491899138,4cyclictest14022-21sed20:40:283
491899136,5cyclictest26352-21diskmemload23:26:283
4918991310,2cyclictest29635-21cut20:00:183
491499138,3cyclictest880-21tr20:10:182
4914991310,3cyclictest0-21swapper/220:30:012
4914991310,2cyclictest10688-21grep19:20:182
491499130,12cyclictest27817-21head19:55:222
491499130,12cyclictest27817-21head19:55:212
491499130,12cyclictest0-21swapper/220:39:002
4912991310,2cyclictest17782-21grep20:50:231
4909991310,2cyclictest24989-21grep19:50:240
4909991310,2cyclictest22633-21grep21:00:180
4909991310,2cyclictest13998-21grep20:40:280
491899129,2cyclictest4768-21tr20:20:143
491899129,2cyclictest432-21grep20:10:153
491899129,2cyclictest23477-21grep21:00:263
491899129,2cyclictest21150-21grep19:40:283
491899128,3cyclictest3094-21cut20:15:213
491899128,3cyclictest24460-21grep19:50:223
491899128,3cyclictest16772-21sh20:49:593
491899122,5cyclictest224-21systemd-journal19:19:593
491499126,2cyclictest224-21systemd-journal21:10:002
491499121,0cyclictest0-21swapper/221:00:212
491499120,11cyclictest5322-21grep20:20:172
491299128,3cyclictest1015-21wc20:10:191
491299128,2cyclictest15162-21sed20:45:161
490999129,2cyclictest6804-21grep19:10:280
490999129,2cyclictest20676-21cut19:40:230
490999129,2cyclictest10690-21cut19:20:170
490999121,7cyclictest224-21systemd-journal20:25:010
491899118,2cyclictest9649-21apt20:35:003
491899118,2cyclictest31708-21chrt20:07:183
491899118,2cyclictest24505-21cut21:05:163
491899118,2cyclictest22100-21tr19:45:143
491899118,2cyclictest19961-21df20:55:163
491899118,2cyclictest19042-21chrt19:39:023
491899118,2cyclictest14865-21grep19:30:173
491499118,2cyclictest28998-21cut20:00:152
491499118,2cyclictest25912-21grep19:50:332
491499118,2cyclictest2414-21expr20:15:162
491499118,2cyclictest22125-21tr19:45:142
491499118,2cyclictest15378-21grep20:45:182
491499117,3cyclictest21535-21taskset19:44:332
491499111,4cyclictest31924-21cron20:10:012
491499110,10cyclictest5768-21uniq19:10:182
491299118,2cyclictest9211-21grep20:30:221
491299118,2cyclictest31717-21taskset20:07:241
491299118,2cyclictest29604-21grep20:00:181
491299118,2cyclictest25946-21grep19:50:331
491299118,2cyclictest20611-21tr20:55:211
491299118,2cyclictest15424-21head19:30:211
491299118,2cyclictest10672-21expr19:20:171
490999118,2cyclictest7418-21wc20:30:010
490999118,2cyclictest30603-21sed20:00:280
490999118,2cyclictest11572-21sed20:35:250
491899108,1cyclictest98742sleep319:20:073
491899108,1cyclictest69712sleep320:26:183
491899108,1cyclictest10157-21expr20:35:153
491899107,2cyclictest5736-21fschecks_count19:10:183
491499108,1cyclictest12920-21grep20:40:182
491499108,1cyclictest0-21swapper/219:15:132
491499107,2cyclictest20959-21sort20:55:262
491499107,2cyclictest14623-21chrt19:30:112
491499100,9cyclictest188852sleep220:51:462
491299108,1cyclictest7507-21basename19:15:121
491299108,1cyclictest21364-21chrt19:42:211
491299108,1cyclictest17314-21grep19:35:151
491299108,1cyclictest12282-21seq19:25:131
491299108,1cyclictest10054-21egrep20:35:141
491299107,2cyclictest5145-21grep20:20:161
491299107,2cyclictest2841-21cut20:15:191
491299107,2cyclictest27108-21sed19:55:161
491299107,2cyclictest27108-21sed19:55:151
491299107,2cyclictest23062-21ls19:45:211
491299107,2cyclictest22153-21tr21:00:161
491299107,2cyclictest13536-21awk20:40:231
491299100,9cyclictest6671-21cat19:10:271
491299100,6cyclictest7132-21sshd20:28:241
491299100,6cyclictest24784-21df21:05:161
490999108,1cyclictest32429-21expr20:10:120
490999108,1cyclictest315432sleep020:05:120
490999108,1cyclictest3081-21sed20:15:210
490999108,1cyclictest262752sleep021:09:140
490999108,1cyclictest19438-21cut20:55:140
490999108,1cyclictest15002-21expr19:30:190
490999107,2cyclictest7988-21expr19:15:160
490999104,5cyclictest17621-21df_inode20:50:220
490999104,4cyclictest13987-21grep19:25:270
490999100,9cyclictest18214-21pluginstate19:35:210
49149995,3cyclictest7523-21apache_accesses20:30:082
49099997,1cyclictest14712-21grep20:45:150
49099986,1cyclictest7726-21tr20:30:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional