You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-16 - 08:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Mon Jul 15, 2024 12:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7403210211,53sleep07479-21unixbench_multi07:04:010
782829915,51sleep37858-21munin-run07:07:173
7990999543,45cyclictest0-21swapper/208:57:172
65802949,51sleep26601-21ls07:02:362
799099902,54cyclictest25204-21taskset07:55:222
7982999057,11cyclictest141rcu_preempt07:32:171
799099883,57cyclictest6917-21taskset09:50:502
799099883,57cyclictest6917-21taskset09:50:502
799099882,59cyclictest9394-21taskset07:11:002
7990998759,16cyclictest2137-21sh09:42:172
799099862,56cyclictest7998-21ssh11:56:162
799099842,55cyclictest10551-21ssh10:58:472
799099842,53cyclictest15536-21memory07:27:432
797399842,59cyclictest32344-21apt-get09:37:360
7973998262,13cyclictest16387-21unixbench_multi10:08:010
7990998139,35cyclictest1-21systemd09:25:012
7982998114,33cyclictest32344-21apt-get09:37:301
257022810,9sleep325686-21munin-run10:27:173
7990998060,13cyclictest30069-21sendmail11:37:162
799099803,50cyclictest17673-21ssh10:11:572
7973998046,9cyclictest141rcu_preempt08:57:350
799099793,48cyclictest26009-21irqstats07:57:432
7990997910,44cyclictest747-21apt-get11:42:342
799099782,47cyclictest5333-21ntp_states08:32:492
7982997843,13cyclictest25-21ksoftirqd/107:35:111
7982997830,22cyclictest25-21ksoftirqd/107:52:461
799099773,47cyclictest0-21swapper/209:20:022
7982997752,16cyclictest20455-21munin-plugin-st10:17:171
7996997648,9cyclictest141rcu_preempt10:54:483
7996997631,21cyclictest363-21in:imklog10:13:473
799099763,65cyclictest30562-21ssh10:36:112
7982997650,15cyclictest0-21swapper/111:32:171
7973997647,7cyclictest141rcu_preempt11:57:430
799099752,63cyclictest29065-21entropy08:07:402
799099752,63cyclictest24688-21cat12:27:192
799099752,63cyclictest24688-21cat12:27:192
7982997543,8cyclictest25-21ksoftirqd/110:24:031
7973997548,9cyclictest141rcu_preempt11:43:140
7973997538,14cyclictest141rcu_preempt07:47:510
7973997527,11cyclictest141rcu_preempt10:12:380
7996997459,9cyclictest3146-21munin-plugin-st11:47:163
799099744,62cyclictest8470-21wc08:42:422
799099743,63cyclictest4507-21grep11:47:582
799099742,62cyclictest6843-21/usr/sbin/munin08:37:432
7982997438,13cyclictest25-21ksoftirqd/109:42:441
7982997434,23cyclictest25-21ksoftirqd/112:16:301
7982997434,23cyclictest25-21ksoftirqd/112:16:301
7982997432,16cyclictest25-21ksoftirqd/111:27:061
7973997432,25cyclictest141rcu_preempt12:03:220
7973997432,25cyclictest141rcu_preempt12:03:220
7973997415,12cyclictest12-21ksoftirqd/010:28:070
7990997349,12cyclictest0-21swapper/209:33:062
7990997349,12cyclictest0-21swapper/209:33:062
799099733,61cyclictest9775-21munin-node07:12:232
799099732,43cyclictest10143-21apt-get09:57:302
799099732,40cyclictest13077-21df_inode08:57:382
7982997328,24cyclictest141rcu_preempt11:57:191
7973997322,30cyclictest363-21in:imklog08:52:470
7996997248,14cyclictest28251-21idleruntime-cro10:32:173
7990997251,14cyclictest18911-21modprobe07:37:172
7990997231,33cyclictest0-21swapper/207:37:262
7990997231,33cyclictest0-21swapper/207:37:262
799099722,63cyclictest18908-21sensors_temp10:13:012
7982997237,13cyclictest25-21ksoftirqd/110:32:541
7982997230,20cyclictest141rcu_preempt08:47:171
7982997211,44cyclictest857-21sh11:42:491
7973997238,12cyclictest141rcu_preempt07:54:580
7973997236,16cyclictest141rcu_preempt07:29:290
7990997127,37cyclictest0-21swapper/211:42:212
799099712,61cyclictest25411-21ntp_states12:27:492
799099712,61cyclictest0-21swapper/210:42:352
799099712,60cyclictest11862-21needreboot12:02:482
799099712,60cyclictest11862-21needreboot12:02:482
799099712,59cyclictest31315-21/usr/sbin/munin10:37:342
799099712,44cyclictest27446-21diskstats08:02:392
799099712,44cyclictest27446-21diskstats08:02:392
7982997135,11cyclictest141rcu_preempt10:08:211
7996997054,10cyclictest1-21systemd11:16:523
7996997051,11cyclictest0-21swapper/312:17:173
7996997051,11cyclictest0-21swapper/312:17:173
799099703,60cyclictest21570-21sh09:17:172
799099703,59cyclictest26270-21fschecks_time10:27:412
799099702,38cyclictest716-21users08:18:082
7982997046,8cyclictest25-21ksoftirqd/110:21:501
7982997032,13cyclictest25-21ksoftirqd/111:58:031
7973997042,13cyclictest141rcu_preempt11:27:260
7973997037,19cyclictest141rcu_preempt12:13:270
7973997037,19cyclictest141rcu_preempt12:13:270
7973997026,10cyclictest141rcu_preempt09:25:440
66712709,32sleep10-21swapper/107:02:441
799099693,58cyclictest11147-21latency_hist12:02:182
799099693,57cyclictest14003-21chrt12:07:222
799099693,42cyclictest12965-21unixbench_singl11:02:582
799099692,42cyclictest0-21swapper/209:09:462
7982996929,22cyclictest141rcu_preempt07:07:451
7973996948,8cyclictest141rcu_preempt09:43:080
7973996938,7cyclictest141rcu_preempt08:37:460
7973996932,15cyclictest141rcu_preempt09:59:550
797399692,25cyclictest27584-21apt-get12:32:270
799699683,47cyclictest1065-21apt-get08:22:353
7996996810,29cyclictest41-21ksoftirqd/308:31:103
799099683,56cyclictest21297-21/usr/sbin/munin07:42:502
799099683,55cyclictest400-21runrttasks08:30:512
799099682,60cyclictest17758-21sh11:12:542
799099682,57cyclictest12810-21grep10:02:352
799099682,44cyclictest0-21swapper/210:52:132
799099682,41cyclictest0-21swapper/212:32:432
799099682,40cyclictest0-21swapper/208:13:422
798299686,20cyclictest183-21pr/ttyS009:48:161
798299686,20cyclictest183-21pr/ttyS009:48:161
7982996851,11cyclictest14487-21apt-key09:02:201
7982996834,12cyclictest25-21ksoftirqd/109:54:101
798299683,35cyclictest16811-21/usr/sbin/munin09:07:371
7982996833,18cyclictest25-21ksoftirqd/111:06:211
7982996828,17cyclictest141rcu_preempt09:57:491
7973996844,8cyclictest141rcu_preempt07:42:490
7973996842,8cyclictest141rcu_preempt10:53:410
7973996842,7cyclictest141rcu_preempt11:08:580
7973996841,8cyclictest141rcu_preempt07:12:540
7973996838,9cyclictest141rcu_preempt11:42:170
7973996838,9cyclictest141rcu_preempt10:04:410
7973996834,7cyclictest141rcu_preempt11:48:470
7973996825,9cyclictest141rcu_preempt11:20:300
7973996823,21cyclictest52-21kcompactd007:26:500
7973996814,22cyclictest0-21swapper/007:36:330
7996996741,8cyclictest141rcu_preempt09:34:213
7996996741,8cyclictest141rcu_preempt09:34:213
7996996721,14cyclictest16144-21diskmemload12:12:193
799099676,35cyclictest14785-21apt-get09:02:322
799099674,54cyclictest24928-21latency_hist11:27:192
799099673,53cyclictest1781-21fschecks_count08:23:032
799099672,58cyclictest28415-21ssh09:29:282
799099672,58cyclictest28415-21ssh09:29:282
799099672,57cyclictest23581-21ssh10:22:352
799099672,56cyclictest21971-21latency_hist12:22:192
799099671,59cyclictest8152-21grep09:52:432
7982996745,15cyclictest22128-21cut09:17:391
7973996736,17cyclictest141rcu_preempt10:17:500
7973996735,16cyclictest141rcu_preempt11:06:330
7973996733,9cyclictest141rcu_preempt10:37:070
7973996726,8cyclictest141rcu_preempt09:20:540
7973996724,29cyclictest141rcu_preempt11:26:070
799699668,14cyclictest41-21ksoftirqd/307:12:423
7996996639,10cyclictest141rcu_preempt09:24:363
7996996629,18cyclictest363-21in:imklog09:31:073
7996996629,18cyclictest363-21in:imklog09:31:073
799099662,57cyclictest9900-21grep08:47:392
799099662,56cyclictest7533-21sed10:53:002
799099662,56cyclictest20603-21/usr/sbin/munin11:17:532
7982996649,11cyclictest31879-21cron08:17:171
7982996647,11cyclictest0-21swapper/109:07:181
7982996645,13cyclictest0-21swapper/108:37:171
7982996643,9cyclictest25-21ksoftirqd/111:07:351
7982996640,7cyclictest141rcu_preempt09:14:081
7982996632,12cyclictest141rcu_preempt07:43:331
7982996630,23cyclictest141rcu_preempt07:22:561
7973996644,9cyclictest141rcu_preempt11:34:190
7973996627,16cyclictest141rcu_preempt10:23:120
7973996627,12cyclictest0-21swapper/007:57:550
7973996624,10cyclictest141rcu_preempt08:37:190
7973996614,20cyclictest363-21in:imklog08:21:340
7973996612,29cyclictest12-21ksoftirqd/008:28:020
7996996532,16cyclictest183-21pr/ttyS011:20:343
799699652,34cyclictest32353-21ssh09:37:303
799099653,55cyclictest26230-21unixbench_multi11:27:582
799099653,51cyclictest22613-21fschecks_count07:47:392
799099652,54cyclictest21632-21sed10:18:022
7982996539,17cyclictest0-21swapper/110:58:051
7982996536,8cyclictest25-21ksoftirqd/112:33:061
7982996526,22cyclictest141rcu_preempt11:12:351
797399659,25cyclictest141rcu_preempt09:07:520
7973996544,7cyclictest141rcu_preempt09:02:300
7973996543,12cyclictest24596-21rm12:27:190
7973996543,12cyclictest24596-21rm12:27:190
7973996541,7cyclictest141rcu_preempt07:07:390
7973996527,10cyclictest141rcu_preempt08:47:200
799699644,28cyclictest24211-21fschecks_count07:52:373
7996996423,19cyclictest363-21in:imklog12:36:223
799099642,55cyclictest17664-21cat12:12:552
799099642,55cyclictest17664-21cat12:12:552
798299645,15cyclictest26161-21ntp_kernel_pll_07:57:481
7982996443,13cyclictest0-21swapper/111:42:171
7982996441,15cyclictest26234-21fschecks_count10:27:401
7982996436,8cyclictest141rcu_preempt08:21:341
7982996423,16cyclictest141rcu_preempt07:17:201
7982996422,22cyclictest141rcu_preempt12:10:401
7973996433,8cyclictest141rcu_preempt09:51:020
7973996433,8cyclictest141rcu_preempt09:51:020
7973996429,15cyclictest141rcu_preempt08:25:220
7973996419,23cyclictest141rcu_preempt09:32:420
7973996419,23cyclictest141rcu_preempt09:32:420
7973996412,22cyclictest0-21swapper/011:57:070
7973996411,32cyclictest12-21ksoftirqd/012:12:180
7996996348,8cyclictest22516-21seq11:22:203
7996996339,14cyclictest14583-21apt-key09:02:213
7996996331,8cyclictest141rcu_preempt11:48:283
7996996327,17cyclictest363-21in:imklog12:24:243
7996996327,17cyclictest363-21in:imklog12:24:243
799699632,42cyclictest14793-21modprobe09:02:313
799099633,53cyclictest15582-21unixbench_singl11:08:002
799099633,53cyclictest12538-21systemd-journal07:27:172
799099632,41cyclictest0-21swapper/207:21:262
7982996341,8cyclictest25-21ksoftirqd/110:37:391
7982996341,14cyclictest0-21swapper/109:27:431
7982996341,14cyclictest0-21swapper/109:27:431
7982996334,11cyclictest141rcu_preempt09:33:521
7982996334,11cyclictest141rcu_preempt09:33:521
7982996334,10cyclictest25-21ksoftirqd/109:23:021
7973996337,6cyclictest141rcu_preempt11:00:450
7973996336,7cyclictest141rcu_preempt12:27:370
7973996330,14cyclictest141rcu_preempt09:14:080
7973996311,30cyclictest12-21ksoftirqd/010:43:020
7973996311,29cyclictest12-21ksoftirqd/008:02:400
7973996311,29cyclictest12-21ksoftirqd/008:02:400
7996996246,8cyclictest0-21swapper/309:07:313
7996996239,13cyclictest0-21swapper/307:57:583
799099622,48cyclictest4962-21latency_hist09:47:202
7982996241,8cyclictest25-21ksoftirqd/108:07:421
7982996232,7cyclictest0-21swapper/112:25:131
7982996232,7cyclictest0-21swapper/112:25:131
7973996244,12cyclictest412-21ntpd09:27:590
7973996244,12cyclictest412-21ntpd09:27:590
7973996244,12cyclictest0-21swapper/009:57:160
7973996235,7cyclictest141rcu_preempt07:37:370
7973996235,7cyclictest141rcu_preempt07:37:370
7973996233,8cyclictest141rcu_preempt10:52:140
7973996233,8cyclictest141rcu_preempt10:38:330
7973996229,8cyclictest141rcu_preempt08:07:550
7973996225,10cyclictest141rcu_preempt11:14:500
7973996224,15cyclictest141rcu_preempt12:19:090
7996996134,7cyclictest141rcu_preempt12:20:453
7996996126,20cyclictest363-21in:imklog12:06:513
7996996126,20cyclictest363-21in:imklog12:06:513
7996996126,20cyclictest363-21in:imklog11:31:163
7996996126,20cyclictest363-21in:imklog10:02:473
7996996126,20cyclictest363-21in:imklog09:20:413
7996996126,18cyclictest141rcu_preempt09:16:163
7996996124,7cyclictest141rcu_preempt08:37:503
7982996127,12cyclictest25-21ksoftirqd/110:43:381
7996996039,12cyclictest0-21swapper/308:12:573
7996996028,19cyclictest120-21pr/ttymxc007:36:263
798299606,27cyclictest25-21ksoftirqd/112:27:431
7982996039,7cyclictest25-21ksoftirqd/112:03:401
7982996039,7cyclictest25-21ksoftirqd/112:03:401
7982996027,19cyclictest141rcu_preempt07:17:461
7982996024,18cyclictest141rcu_preempt11:17:331
7973996025,7cyclictest0-21swapper/008:49:530
7973996022,11cyclictest141rcu_preempt07:21:060
7996995943,7cyclictest0-21swapper/307:22:173
7996995938,13cyclictest0-21swapper/308:37:173
7996995931,7cyclictest141rcu_preempt11:01:483
7996995926,18cyclictest183-21pr/ttyS012:31:503
7996995913,21cyclictest401ktimers/309:50:443
7996995913,21cyclictest401ktimers/309:50:443
798299598,22cyclictest25-21ksoftirqd/107:52:191
7982995940,11cyclictest52-21kcompactd008:25:341
7982995934,7cyclictest25-21ksoftirqd/107:37:261
7982995934,7cyclictest25-21ksoftirqd/107:37:261
798299593,22cyclictest4023-21cstates10:47:331
7982995922,16cyclictest141rcu_preempt08:49:581
7973995926,7cyclictest141rcu_preempt08:12:550
799699588,13cyclictest141rcu_preempt11:02:333
7996995843,10cyclictest363-21in:imklog12:00:433
799699584,22cyclictest12744-21latency_hist08:57:193
7996995841,10cyclictest30953-21/usr/sbin/munin11:37:473
7996995832,14cyclictest141rcu_preempt10:37:043
7996995825,6cyclictest141rcu_preempt11:32:403
7996995823,11cyclictest183-21pr/ttyS010:11:363
799699582,22cyclictest41-21ksoftirqd/310:19:293
7996995821,12cyclictest363-21in:imklog10:48:543
7982995843,8cyclictest25-21ksoftirqd/111:32:441
798299583,30cyclictest141rcu_preempt10:56:251
7982995819,14cyclictest141rcu_preempt12:17:381
7982995818,21cyclictest141rcu_preempt08:54:271
799699574,24cyclictest41-21ksoftirqd/307:42:523
7996995730,15cyclictest0-21swapper/308:07:423
7996995726,22cyclictest19168-21/usr/sbin/munin07:37:453
7996995726,22cyclictest19168-21/usr/sbin/munin07:37:453
7996995725,10cyclictest183-21pr/ttyS009:43:253
7996995717,20cyclictest363-21in:imklog11:53:273
7996995711,23cyclictest41-21ksoftirqd/309:56:583
7982995735,7cyclictest25-21ksoftirqd/110:07:181
7982995722,8cyclictest141rcu_preempt08:02:451
7982995722,8cyclictest141rcu_preempt08:02:451
7996995636,6cyclictest141rcu_preempt09:58:203
7996995622,18cyclictest363-21in:imklog08:02:243
7996995622,18cyclictest363-21in:imklog08:02:243
798299564,29cyclictest400-21runrttasks11:52:031
7996995539,9cyclictest8977-21unixbench_multi08:42:563
7996995532,7cyclictest141rcu_preempt07:22:403
7996995531,10cyclictest363-21in:imklog10:39:563
7996995516,21cyclictest41-21ksoftirqd/307:09:183
7996995514,19cyclictest0-21swapper/307:27:363
7996995512,14cyclictest16144-21diskmemload10:46:273
7996995511,20cyclictest401ktimers/311:07:553
7996995430,7cyclictest141rcu_preempt08:47:353
7982995432,7cyclictest25-21ksoftirqd/108:27:461
7982995322,17cyclictest0-21swapper/108:37:391
799699512,22cyclictest22565-21diskstats07:47:383
7996995113,12cyclictest0-21swapper/308:18:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional