You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 06:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Wed Apr 02, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
232049912757,58cyclictest0-21swapper/019:42:180
232339911053,50cyclictest28585-21latency_hist23:02:203
2321299952,47cyclictest6980-21apt-get23:22:331
2322499943,56cyclictest15257-21df_inode22:38:232
216982926,14sleep30-21swapper/319:02:283
23224999017,41cyclictest26293-21munin-run23:57:192
2322499893,58cyclictest16055-21apt-get20:27:332
2322499893,58cyclictest16055-21apt-get20:27:332
23212998729,10cyclictest23042-21fschecks_count22:52:401
23212998322,14cyclictest15768-21fschecks_count23:37:381
23224998121,30cyclictest402-21runrttasks00:07:172
23212998121,12cyclictest0-21swapper/119:12:421
23212998022,10cyclictest0-21swapper/119:37:371
23212998017,11cyclictest0-21swapper/122:02:221
23224997925,24cyclictest0-21swapper/223:39:502
23212997923,50cyclictest0-21swapper/120:53:321
23212997914,19cyclictest5079-21latency_hist00:17:221
23212997914,13cyclictest0-21swapper/121:17:421
23212997816,11cyclictest0-21swapper/123:07:211
23212997816,11cyclictest0-21swapper/123:07:211
23204997857,13cyclictest5837-21latency_hist21:22:210
23212997753,14cyclictest15385-21ssh22:38:291
2321299763,31cyclictest11007-21grep20:08:211
23212997628,40cyclictest0-21swapper/121:52:501
23212997626,45cyclictest8066-1kworker/1:1H22:07:221
23204997640,8cyclictest0-21swapper/023:57:200
42282750,5sleep20-21swapper/200:15:342
23233997555,12cyclictest0-21swapper/323:27:183
23224997524,27cyclictest21941-21cut21:52:232
23224997524,27cyclictest21941-21cut21:52:232
23212997521,11cyclictest0-21swapper/123:17:431
23224997454,8cyclictest9552-21kworker/u8:220:22:332
23224997428,18cyclictest1897-21sh00:12:082
23212997429,8cyclictest0-21swapper/123:52:461
23224997346,17cyclictest25702-21latency_hist22:57:212
23224997342,21cyclictest20105-21/usr/sbin/munin22:47:372
23212997352,13cyclictest29340-21sh21:07:221
23212997316,12cyclictest0-21swapper/121:24:541
23212997316,12cyclictest0-21swapper/121:24:541
2195027326,23sleep20-21swapper/219:02:472
23233997226,14cyclictest41-21ksoftirqd/322:07:533
23233997226,14cyclictest41-21ksoftirqd/322:07:533
23224997247,17cyclictest8574-21ssh23:25:302
23212997222,7cyclictest0-21swapper/123:15:181
23212997217,11cyclictest0-21swapper/123:44:331
23212997216,11cyclictest0-21swapper/100:07:371
23233997154,10cyclictest14660-21gpgv22:37:223
23233997125,13cyclictest41-21ksoftirqd/300:12:423
23224997146,18cyclictest15128-21date00:37:332
23224997140,23cyclictest28951-21fschecks_time19:22:412
23224997140,23cyclictest28951-21fschecks_time19:22:412
23212997138,18cyclictest420-21snmpd22:21:011
23212997138,18cyclictest420-21snmpd22:21:011
23212997120,12cyclictest0-21swapper/121:12:401
2321299702,14cyclictest14391-21apt-config22:37:201
23212997017,12cyclictest0-21swapper/119:52:581
2320499703,33cyclictest2815-21idleruntime-cro21:17:180
23224996940,21cyclictest27922-21/usr/sbin/munin23:58:252
23224996937,25cyclictest22816-21sort20:47:532
23224996937,25cyclictest22816-21sort20:47:532
23212996925,10cyclictest3300-21awk22:17:191
23212996925,10cyclictest3300-21awk22:17:191
23212996919,11cyclictest0-21swapper/122:50:401
23212996917,11cyclictest0-21swapper/119:28:271
23204996951,12cyclictest7262-21latency_hist23:22:210
23224996841,16cyclictest12274-21munin-plugin-st00:32:182
23224996840,18cyclictest30027-21/usr/sbin/munin22:07:502
23224996840,18cyclictest30027-21/usr/sbin/munin22:07:502
23224996821,19cyclictest29251-21diskmemload00:19:432
23224996821,17cyclictest29251-21diskmemload23:21:192
23224996819,20cyclictest30965-21diskstats23:07:352
23212996848,12cyclictest15211-21sendmail23:37:181
2321299683,26cyclictest16354-21fschecks_time20:28:111
2321299683,26cyclictest16354-21fschecks_time20:28:111
23212996822,8cyclictest0-21swapper/122:27:281
23212996820,11cyclictest0-21swapper/123:08:451
23212996812,13cyclictest0-21swapper/122:12:211
23212996812,13cyclictest0-21swapper/122:12:211
23204996844,14cyclictest26768-21cron19:17:190
23233996752,9cyclictest27642-21munin-plugin-st21:02:183
23233996752,7cyclictest0-21swapper/323:57:563
23233996729,8cyclictest41-21ksoftirqd/323:52:203
23224996747,10cyclictest3139-21ssh21:17:312
23224996746,13cyclictest0-21swapper/220:07:212
23224996742,17cyclictest17822-21/usr/sbin/munin20:32:372
23224996734,24cyclictest4595-21sort19:47:472
23224996733,23cyclictest32557-21apt-get21:12:332
23224996732,23cyclictest11689-21apt-get20:12:362
23224996729,26cyclictest29684-21ssh23:05:062
23224996729,26cyclictest29684-21ssh23:05:062
23224996727,30cyclictest30100-21sed21:07:472
23224996727,30cyclictest30100-21sed21:07:472
23212996715,12cyclictest0-21swapper/119:09:011
23212996714,25cyclictest25-21ksoftirqd/100:37:171
23204996752,8cyclictest8250irq/69-2188000.00:02:470
47512660,13sleep0161rcuc/000:17:100
23233996649,10cyclictest0-21swapper/323:27:363
23224996643,15cyclictest16197-21ssh21:40:462
23224996643,15cyclictest16197-21ssh21:40:462
23224996639,18cyclictest9627-21/usr/sbin/munin23:27:312
23224996636,22cyclictest24182-21timerandwakeup19:07:552
23224996633,25cyclictest5989-21cpu19:52:362
23224996620,19cyclictest29251-21diskmemload21:56:472
2321299662,12cyclictest32446-21ssh21:12:221
2321299662,12cyclictest32446-21ssh21:12:221
23212996618,13cyclictest0-21swapper/120:59:171
23204996643,16cyclictest2381-21unixbench_singl23:12:550
23204996628,19cyclictest12-21ksoftirqd/023:58:170
23204996611,20cyclictest12-21ksoftirqd/023:37:310
23233996521,9cyclictest141rcu_preempt20:12:433
23224996534,22cyclictest1298-21sort23:12:302
23224996533,23cyclictest13062-21/usr/sbin/munin23:32:472
23224996530,25cyclictest15040-21latency_hist20:22:222
23224996520,16cyclictest20157-21ssh23:46:592
23212996544,12cyclictest8274-21kworker/u8:122:27:181
23212996540,10cyclictest141rcu_preempt20:42:581
23212996540,10cyclictest141rcu_preempt20:42:581
23233996438,16cyclictest6273-21idleruntime-cro22:22:193
23233996438,16cyclictest6273-21idleruntime-cro22:22:193
23224996440,13cyclictest0-21swapper/220:53:092
23224996433,20cyclictest25529-21/usr/sbin/munin19:12:372
23224996432,23cyclictest8574-21apt22:27:262
23224996428,27cyclictest27424-21/usr/sbin/munin19:17:442
23224996428,26cyclictest30649-21/usr/sbin/munin19:27:462
23224996428,26cyclictest10004-21cmp20:07:232
23224996420,17cyclictest29251-21diskmemload22:04:252
2321299642,31cyclictest2941-21uniq19:42:401
23204996443,13cyclictest25926-21ssh22:57:360
230992649,33sleep0339-21dbus-daemon19:07:210
2323399639,20cyclictest141rcu_preempt20:22:223
23233996332,11cyclictest122-21pr/ttymxc023:42:533
23233996326,8cyclictest41-21ksoftirqd/319:32:423
23233996326,8cyclictest41-21ksoftirqd/319:32:423
23224996343,9cyclictest0-21swapper/221:58:092
23224996341,14cyclictest21274-21/usr/sbin/munin23:47:462
23224996340,16cyclictest3597-21apt22:17:302
23224996340,15cyclictest27495-21rm22:59:362
23224996332,23cyclictest9616-21unin-run00:27:182
23224996331,23cyclictest26588-21taskset20:58:012
23224996319,17cyclictest10247-21ssh21:30:532
23233996244,10cyclictest13763-21munin-plugin-st21:37:193
23233996239,12cyclictest0-21swapper/321:32:213
23233996218,14cyclictest41-21ksoftirqd/322:26:333
23224996243,13cyclictest0-21swapper/219:32:322
23224996243,13cyclictest0-21swapper/219:32:322
23224996232,21cyclictest1840-21cat19:37:532
23224996230,25cyclictest4114-21/usr/sbin/munin22:17:502
23224996230,25cyclictest4114-21/usr/sbin/munin22:17:502
23224996230,24cyclictest11680-21ssh21:32:482
23224996230,22cyclictest8140-21/usr/sbin/munin19:57:512
23224996229,25cyclictest11942-21basename22:32:442
23212996244,10cyclictest19506-21latency_hist22:47:181
2321299624,20cyclictest24982-21munin-node20:37:301
23233996146,11cyclictest22278-21kworker/u8:322:47:183
23233996128,9cyclictest41-21ksoftirqd/319:37:513
23233996125,9cyclictest41-21ksoftirqd/319:27:223
23233996125,9cyclictest41-21ksoftirqd/319:27:223
23233996124,12cyclictest41-21ksoftirqd/321:58:063
23224996136,17cyclictest7928-21cat21:27:212
23224996136,17cyclictest7928-21cat21:27:212
23224996133,17cyclictest0-21swapper/222:47:212
23224996131,20cyclictest21392-21/usr/sbin/munin20:42:572
23224996131,20cyclictest21392-21/usr/sbin/munin20:42:572
23224996129,23cyclictest28462-21/usr/sbin/munin21:02:502
23224996126,26cyclictest635-21grep22:12:352
23212996135,17cyclictest29251-21diskmemload23:58:121
23204996123,9cyclictest12-21ksoftirqd/020:28:280
23204996123,9cyclictest12-21ksoftirqd/020:28:280
2323399603,38cyclictest30071-21ssh00:02:473
2323399603,15cyclictest402-21runrttasks21:56:083
23233996026,8cyclictest41-21ksoftirqd/300:12:213
23233996017,13cyclictest41-21ksoftirqd/300:27:203
23224996032,19cyclictest365-21rs:main2
23224996032,18cyclictest19764-21cat20:37:512
23224996029,20cyclictest6672-21ssh22:22:462
23212996036,15cyclictest0-21swapper/123:27:481
23204996047,8cyclictest20058-21ssh23:46:420
23204996046,8cyclictest29828-21ssh22:07:210
23233995947,7cyclictest0-21swapper/321:17:563
23233995926,8cyclictest41-21ksoftirqd/321:02:473
23233995923,13cyclictest41-21ksoftirqd/322:47:373
23224995927,23cyclictest17083-21ssh21:42:332
23212995941,11cyclictest280-21systemd-journal00:17:271
23204995944,10cyclictest12-21ksoftirqd/022:38:080
23204995941,11cyclictest339-21dbus-daemon20:07:200
23204995937,15cyclictest30091-21latency21:07:470
23204995937,15cyclictest30091-21latency21:07:470
23204995915,26cyclictest0-21swapper/022:23:330
217272599,29sleep121722-21apt-get19:02:341
23233995840,11cyclictest364-21in:imklog19:57:383
23233995839,12cyclictest364-21in:imklog21:12:403
23233995832,19cyclictest14359-21grep21:37:413
23233995832,19cyclictest14359-21grep21:37:413
2321299588,34cyclictest0-21swapper/122:57:221
23212995843,7cyclictest0-21swapper/119:47:501
23212995838,13cyclictest12089-21unixbench_multi21:32:541
23212995836,15cyclictest363-21in:imuxsock19:22:201
23212995825,12cyclictest141rcu_preempt21:44:221
23204995841,11cyclictest0-21swapper/023:37:180
23204995838,13cyclictest8250irq/69-2188000.23:47:480
23204995837,11cyclictest23549-21ssh22:52:520
23204995829,11cyclictest8250irq/69-2188000.21:47:330
2320499581,31cyclictest28934-21fschecks_time19:22:410
2320499581,31cyclictest28934-21fschecks_time19:22:410
23233995734,12cyclictest0-21swapper/323:07:213
23233995734,12cyclictest0-21swapper/323:07:213
23233995718,8cyclictest41-21ksoftirqd/320:28:103
23233995718,8cyclictest41-21ksoftirqd/320:28:103
2321299573,30cyclictest14968-21ssh21:37:531
2321299573,30cyclictest14968-21ssh21:37:531
2320499577,18cyclictest10756-21munin-plugin-st21:32:180
23204995741,10cyclictest0-21swapper/020:22:220
2323399565,8cyclictest141rcu_preempt00:37:173
23233995639,11cyclictest5422-21unixbench_singl23:17:573
23233995639,11cyclictest0-21swapper/323:17:183
23233995637,12cyclictest280-21systemd-journal21:46:213
23233995636,13cyclictest280-21systemd-journal20:08:093
23233995621,9cyclictest141rcu_preempt22:38:403
23212995641,10cyclictest0-21swapper/120:48:061
23212995641,10cyclictest0-21swapper/120:48:061
23204995640,10cyclictest29251-21diskmemload00:36:020
23204995639,10cyclictest22423-21fschecks_time21:52:500
23204995638,9cyclictest0-21swapper/021:37:190
23204995628,14cyclictest12-21ksoftirqd/022:16:210
23204995628,14cyclictest12-21ksoftirqd/022:16:210
23204995622,14cyclictest12-21ksoftirqd/022:20:330
23204995622,14cyclictest12-21ksoftirqd/022:20:330
2323399555,9cyclictest41-21ksoftirqd/319:17:193
23233995541,9cyclictest185-21pr/ttyS022:27:323
23233995537,9cyclictest21780-21runrttasks21:52:203
23233995537,9cyclictest21780-21runrttasks21:52:203
23233995537,12cyclictest0-21swapper/321:27:213
23233995537,12cyclictest0-21swapper/321:27:213
23233995526,8cyclictest41-21ksoftirqd/323:52:323
2321299557,33cyclictest0-21swapper/100:07:201
2321299555,27cyclictest25-21ksoftirqd/119:27:221
23212995539,10cyclictest20150-21unixbench_multi21:48:001
23212995539,10cyclictest20150-21unixbench_multi21:48:001
23212995539,10cyclictest0-21swapper/119:57:551
23212995535,13cyclictest9089-21sed21:27:541
23212995534,11cyclictest0-21swapper/119:37:191
23212995534,11cyclictest0-21swapper/119:37:191
2321299553,34cyclictest0-21swapper/120:27:221
23212995531,15cyclictest280-21systemd-journal20:12:551
23204995534,12cyclictest0-21swapper/020:53:350
23204995532,14cyclictest8858-21ssh00:25:090
2320499553,19cyclictest29249-21sh21:07:190
23233995439,8cyclictest0-21swapper/323:07:493
23233995438,9cyclictest5698-21kthreadd19:52:203
23233995436,11cyclictest364-21in:imklog19:07:423
23233995434,13cyclictest122-21pr/ttymxc023:40:263
23233995425,14cyclictest141rcu_preempt21:07:523
23233995425,14cyclictest141rcu_preempt21:07:523
23233995423,13cyclictest41-21ksoftirqd/319:17:563
23233995418,14cyclictest41-21ksoftirqd/323:37:213
23204995437,11cyclictest363-21in:imuxsock20:22:230
23204995436,12cyclictest9666-21unixbench_multi22:28:030
23204995435,9cyclictest0-21swapper/000:07:310
2320499543,32cyclictest141rcu_preempt22:32:460
23204995433,12cyclictest0-21swapper/023:32:180
2320499543,16cyclictest30337-21fschecks_time22:07:400
2320499543,16cyclictest30337-21fschecks_time22:07:400
23204995429,12cyclictest0-21swapper/000:22:180
23233995333,12cyclictest364-21in:imklog19:57:193
23233995329,15cyclictest122-21pr/ttymxc022:53:083
23233995319,8cyclictest41-21ksoftirqd/322:12:503
23233995319,8cyclictest41-21ksoftirqd/322:12:503
2321299533,20cyclictest9668-21latency_hist00:27:201
23204995342,7cyclictest12-21ksoftirqd/022:50:040
23204995337,11cyclictest0-21swapper/023:27:180
23204995332,11cyclictest0-21swapper/020:42:520
23204995332,11cyclictest0-21swapper/020:42:520
23233995230,9cyclictest0-21swapper/320:42:213
23233995228,13cyclictest0-21swapper/320:42:523
23233995228,13cyclictest0-21swapper/320:42:523
23233995213,17cyclictest0-21swapper/300:17:533
2321299523,35cyclictest0-21swapper/100:27:311
2321299523,29cyclictest0-21swapper/123:52:191
23212995231,13cyclictest0-21swapper/120:17:411
23204995236,11cyclictest25408-21tune2fs21:57:540
2320499523,17cyclictest7993-21latency19:57:450
2320499522,18cyclictest10362-21fschecks_time20:08:080
23204995213,11cyclictest0-21swapper/023:02:280
23204995213,11cyclictest0-21swapper/023:02:280
23233995137,8cyclictest122-21pr/ttymxc022:05:563
23233995136,10cyclictest0-21swapper/320:37:183
23233995133,11cyclictest364-21in:imklog00:27:413
2323399513,19cyclictest15907-21apt-config20:27:223
23233995129,10cyclictest0-21swapper/320:57:213
23233995122,16cyclictest41-21ksoftirqd/320:47:533
23233995122,16cyclictest41-21ksoftirqd/320:47:533
2320499518,33cyclictest0-21swapper/023:07:480
2320499514,39cyclictest12404-21wc00:32:220
2320499513,14cyclictest19336-21fschecks_count21:47:430
2320499513,14cyclictest19336-21fschecks_count21:47:430
2323399507,20cyclictest9739-21cat20:07:213
23212995029,11cyclictest0-21swapper/120:38:011
23212995025,14cyclictest0-21swapper/119:22:401
23204995035,9cyclictest5706-21systemd19:52:210
23204995034,11cyclictest0-21swapper/021:42:180
23204995034,11cyclictest0-21swapper/021:42:180
23204995032,10cyclictest0-21swapper/021:02:200
23204995031,13cyclictest6267-21kworker/u8:022:43:290
23233994911,8cyclictest41-21ksoftirqd/319:32:213
23212994928,14cyclictest9608-21systemd20:07:201
2320499493,13cyclictest31892-21/usr/sbin/munin19:32:430
2320499493,13cyclictest31892-21/usr/sbin/munin19:32:430
23204994833,8cyclictest0-21swapper/020:37:480
2320499483,36cyclictest19155-21apt-get20:37:350
23204994832,9cyclictest0-21swapper/020:17:190
23204994831,11cyclictest27830-21threads19:17:530
23204994826,9cyclictest12-21ksoftirqd/019:07:480
23233994729,11cyclictest41-21ksoftirqd/319:42:363
23204994726,12cyclictest12-21ksoftirqd/019:27:530
23204994524,11cyclictest0-21swapper/020:52:210
23204994524,11cyclictest0-21swapper/020:52:210
23204994522,10cyclictest12-21ksoftirqd/021:27:210
23204994522,10cyclictest12-21ksoftirqd/021:27:210
23204994419,15cyclictest0-21swapper/019:42:540
2320499433,13cyclictest5918-21/usr/sbin/munin19:52:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional