You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:17
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Thu Apr 03, 2025 00:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1732221220,6sleep317314-21ssh23:27:303
1732221220,6sleep317314-21ssh23:27:303
2933121118,73sleep30-21swapper/319:02:473
30403210711,59sleep030395-21munin-run19:07:160
3043428313,27sleep20-21swapper/219:07:182
30539998141,15cyclictest141rcu_preempt19:33:101
30539998141,15cyclictest141rcu_preempt19:33:101
30547997761,9cyclictest16478-21modprobe20:02:152
30539997651,17cyclictest22128-21latency_hist21:37:171
133302760,6sleep113318-21ssh23:18:271
2970527410,42sleep129744-21/usr/sbin/munin19:03:021
30539997344,15cyclictest141rcu_preempt00:07:451
3052999732,38cyclictest5070-21diskmemload23:17:480
3102720,9chrt0-21swapper/219:12:342
30539997235,15cyclictest280-21systemd-journal23:23:531
3052999723,33cyclictest22757-21df_inode21:37:560
30529997017,21cyclictest280-21systemd-journal21:27:360
30529997017,21cyclictest280-21systemd-journal21:27:360
30559996941,7cyclictest141rcu_preempt00:17:143
30559996941,7cyclictest141rcu_preempt00:17:143
30559996937,6cyclictest141rcu_preempt21:12:143
30559996937,11cyclictest364-21in:imklog00:17:263
30559996926,16cyclictest141rcu_preempt21:43:193
30539996936,16cyclictest141rcu_preempt00:19:591
30559996838,7cyclictest141rcu_preempt20:32:423
30559996821,9cyclictest0-21swapper/323:00:103
30547996853,9cyclictest22319-21latency_hist21:37:182
30539996832,12cyclictest141rcu_preempt22:33:081
30559996719,18cyclictest32287-21kworker/3:223:10:393
30539996746,15cyclictest32268-21latency_hist19:12:181
30529996724,24cyclictest141rcu_preempt21:47:460
30529996648,12cyclictest13997-21true21:22:150
30529996648,12cyclictest13997-21true21:22:150
30529996642,15cyclictest5070-21diskmemload22:37:330
30529996642,15cyclictest5070-21diskmemload22:37:330
30529996633,19cyclictest141rcu_preempt22:12:170
30529996616,30cyclictest12-21ksoftirqd/022:12:370
30559996545,13cyclictest32039-21modprobe19:12:153
30559996538,7cyclictest141rcu_preempt20:57:173
30559996537,8cyclictest141rcu_preempt23:15:213
30559996537,7cyclictest141rcu_preempt23:37:193
30559996535,8cyclictest141rcu_preempt00:07:163
30559996524,13cyclictest141rcu_preempt00:00:513
30547996550,9cyclictest22098-21munin-plugin-st00:37:142
30547996547,12cyclictest6998-21wc23:07:442
30539996527,14cyclictest141rcu_preempt19:47:511
30539996519,20cyclictest0-21swapper/120:37:181
30529996537,12cyclictest12-21ksoftirqd/022:17:540
30529996535,20cyclictest6384-21latency_hist00:07:160
3052999653,12cyclictest16836-21cstates20:02:330
30529996514,26cyclictest12-21ksoftirqd/021:32:420
30559996435,7cyclictest141rcu_preempt21:02:373
30559996435,7cyclictest141rcu_preempt21:02:373
30559996434,7cyclictest141rcu_preempt19:27:153
30539996443,13cyclictest32199-21unixbench_multi23:52:511
30539996439,8cyclictest141rcu_preempt21:16:481
30529996446,11cyclictest1647-21modprobe19:17:150
30529996442,16cyclictest5075-21idleruntime-cro21:07:140
30529996442,16cyclictest5075-21idleruntime-cro21:07:140
30559996320,19cyclictest364-21in:imklog22:27:163
30547996347,9cyclictest4299-21ssh00:02:142
30547996344,13cyclictest22457-21idleruntime-cro22:37:152
30529996337,10cyclictest1962-21latency_hist20:57:170
30529996332,12cyclictest280-21systemd-journal21:09:530
30529996322,19cyclictest280-21systemd-journal00:36:240
30559996244,11cyclictest20815-21unixbench_multi22:32:493
30559996235,17cyclictest11015-21threads19:42:493
30559996229,10cyclictest141rcu_preempt21:12:433
30559996226,12cyclictest280-21systemd-journal20:22:523
30529996247,10cyclictest20862-21unixbench_singl22:32:500
3052999624,39cyclictest12-21ksoftirqd/021:17:180
30529996242,12cyclictest12069-21latency_hist23:17:180
30529996221,11cyclictest141rcu_preempt20:12:170
30529996220,21cyclictest141rcu_preempt23:07:170
30529996220,21cyclictest141rcu_preempt23:07:170
30529996218,12cyclictest5070-21diskmemload23:37:370
30529996218,12cyclictest5070-21diskmemload23:37:370
30559996129,13cyclictest141rcu_preempt20:32:153
30559996125,16cyclictest185-21pr/ttyS022:21:583
30559996117,18cyclictest364-21in:imklog00:09:063
30539996142,13cyclictest8815-21munin-plugin-st23:12:141
30539996140,14cyclictest16597-21latency_hist00:27:161
30539996136,12cyclictest0-21swapper/119:27:521
30539996133,8cyclictest122-21pr/ttymxc020:12:271
30559996031,7cyclictest141rcu_preempt22:27:433
30559996031,7cyclictest141rcu_preempt21:38:133
30547996045,9cyclictest31591-21ssh21:53:122
30547996042,11cyclictest27925-21systemd-run23:47:152
30547996040,12cyclictest16504-21cron00:27:142
30539996022,8cyclictest0-21swapper/122:12:271
30539996019,11cyclictest141rcu_preempt21:27:151
30529996044,10cyclictest17862-21runrttasks22:27:460
30529996033,12cyclictest141rcu_preempt21:47:170
30529996025,21cyclictest141rcu_preempt00:13:510
30529996025,21cyclictest141rcu_preempt00:13:510
30559995943,9cyclictest359-21systemd-logind21:52:153
30559995919,20cyclictest364-21in:imklog21:36:043
30539995940,13cyclictest19261-21unixbench_multi20:07:521
30539995940,11cyclictest20673-21unixbench_singl00:32:511
30539995935,14cyclictest5070-21diskmemload23:32:371
30539995932,7cyclictest141rcu_preempt23:41:471
30539995932,7cyclictest141rcu_preempt23:41:471
30539995930,8cyclictest5070-21diskmemload23:57:481
30539995923,13cyclictest0-21swapper/120:48:581
30529995942,10cyclictest0-21swapper/022:02:140
30529995941,12cyclictest17805-21unixbench_multi00:27:450
30529995929,7cyclictest141rcu_preempt00:26:220
30529995911,27cyclictest141rcu_preempt23:12:160
30559995835,12cyclictest0-21swapper/322:52:143
30559995834,7cyclictest141rcu_preempt21:58:443
30559995831,8cyclictest141rcu_preempt22:43:443
30559995831,7cyclictest141rcu_preempt00:33:273
30559995829,7cyclictest141rcu_preempt00:23:033
30547995842,9cyclictest28447-21ssh22:47:422
30547995842,9cyclictest21312-21grep23:32:502
30539995831,14cyclictest141rcu_preempt00:30:521
3052999588,12cyclictest364-21in:imklog00:22:160
30529995844,8cyclictest1469-21latency_hist23:57:160
30529995830,14cyclictest12-21ksoftirqd/019:47:430
30529995819,17cyclictest12-21ksoftirqd/021:52:310
30559995741,10cyclictest30708-21idleruntime21:52:373
30539995731,7cyclictest141rcu_preempt19:37:501
30539995719,13cyclictest0-21swapper/122:07:441
30529995732,7cyclictest141rcu_preempt23:25:390
30529995720,23cyclictest141rcu_preempt20:36:350
30559995623,16cyclictest141rcu_preempt22:37:323
30559995623,16cyclictest141rcu_preempt22:37:323
30559995621,14cyclictest141rcu_preempt23:54:333
30547995639,9cyclictest321ktimers/223:27:272
30547995639,9cyclictest321ktimers/223:27:272
30547995639,11cyclictest280-21systemd-journal22:46:032
30547995638,10cyclictest0-21swapper/219:57:152
3053999563,6cyclictest141rcu_preempt21:49:141
30539995630,7cyclictest141rcu_preempt23:52:171
30539995630,7cyclictest141rcu_preempt22:30:351
30539995625,8cyclictest185-21pr/ttyS022:27:121
30539995622,17cyclictest141rcu_preempt21:40:521
30539995621,23cyclictest0-21swapper/121:57:171
30539995615,18cyclictest141rcu_preempt00:12:361
30539995615,18cyclictest141rcu_preempt00:12:361
30539995613,17cyclictest141rcu_preempt23:02:171
30539995613,17cyclictest141rcu_preempt23:02:171
30529995639,10cyclictest364-21in:imklog19:27:170
30529995629,14cyclictest141rcu_preempt22:47:020
3052999562,15cyclictest141rcu_preempt23:45:070
30529995617,15cyclictest12-21ksoftirqd/019:52:320
30529995612,19cyclictest0-21swapper/020:17:160
30559995540,9cyclictest6806-21latency_hist19:32:183
30559995540,9cyclictest1902-21latency_hist19:17:183
30559995540,11cyclictest16960-21kworker/u8:023:52:143
30559995531,8cyclictest141rcu_preempt23:23:533
30559995530,7cyclictest141rcu_preempt21:27:583
30559995530,7cyclictest141rcu_preempt21:27:583
30559995520,10cyclictest364-21in:imklog19:57:163
30547995540,10cyclictest364-21in:imklog20:11:152
30547995539,8cyclictest0-21swapper/200:07:462
30547995539,10cyclictest17437-21proc_pri00:27:412
30547995538,11cyclictest5070-21diskmemload22:12:332
30547995537,12cyclictest1431-40mandb23:57:172
30539995533,16cyclictest11790-21sendmail23:17:151
30539995531,6cyclictest141rcu_preempt20:26:261
30539995526,14cyclictest141rcu_preempt20:42:411
30539995517,12cyclictest141rcu_preempt22:58:311
30529995536,13cyclictest3064-21unixbench_singl20:57:490
30529995529,13cyclictest141rcu_preempt00:10:100
3052999552,39cyclictest26180-21kworker/0:023:48:030
30529995522,14cyclictest364-21in:imklog22:52:580
30529995516,14cyclictest12-21ksoftirqd/023:32:360
30529995512,21cyclictest2386-21/usr/sbin/munin23:57:480
30529995512,20cyclictest15902-21kworker/0:022:57:480
30559995440,7cyclictest0-21swapper/320:12:143
30559995437,11cyclictest364-21in:imklog20:02:353
30559995427,7cyclictest141rcu_preempt20:20:373
30559995426,7cyclictest141rcu_preempt22:09:413
30559995424,8cyclictest141rcu_preempt19:42:153
30559995414,7cyclictest41-21ksoftirqd/323:42:123
30559995414,7cyclictest41-21ksoftirqd/323:42:123
30547995439,9cyclictest23004-21kernelversion21:38:042
30547995439,8cyclictest0-21swapper/220:22:532
30547995439,11cyclictest30519-21cyclictest23:27:142
30547995439,11cyclictest30519-21cyclictest21:17:172
30547995438,10cyclictest364-21in:imklog21:26:072
30547995437,9cyclictest0-21swapper/221:32:152
30547995437,9cyclictest0-21swapper/221:32:152
30547995437,10cyclictest364-21in:imklog20:32:422
30547995435,13cyclictest873-21ssh22:57:142
30539995438,10cyclictest280-21systemd-journal19:57:371
30539995423,7cyclictest141rcu_preempt21:59:221
30539995423,13cyclictest0-21swapper/121:18:101
30539995423,13cyclictest0-21swapper/121:18:101
30529995432,13cyclictest26173-21idleruntime-cro20:32:140
3052999542,11cyclictest21775-21fschecks_count20:17:360
30529995415,17cyclictest185-21pr/ttyS021:22:430
30529995413,24cyclictest12-21ksoftirqd/019:32:160
263442540,4sleep10-21swapper/120:32:171
30559995336,11cyclictest578-21cron21:22:153
30559995336,11cyclictest578-21cron21:22:153
30559995335,11cyclictest364-21in:imklog22:03:443
30559995329,8cyclictest141rcu_preempt20:57:413
3054799534,23cyclictest1884-21/usr/sbin/munin22:57:482
30547995337,10cyclictest364-21in:imklog22:31:262
30547995337,10cyclictest0-21swapper/200:17:142
30547995337,10cyclictest0-21swapper/200:17:142
30547995336,11cyclictest0-21swapper/221:47:452
30547995335,11cyclictest364-21in:imklog22:19:482
30547995334,12cyclictest13319-21grep00:17:502
30539995336,11cyclictest364-21in:imklog00:07:141
30539995335,12cyclictest28567-21ntp_kernel_pll_20:37:431
3053999532,9cyclictest27968-21fschecks_count22:47:311
30539995322,13cyclictest141rcu_preempt22:37:501
30539995322,13cyclictest141rcu_preempt22:37:501
30539995321,12cyclictest141rcu_preempt22:43:451
30539995320,7cyclictest0-21swapper/121:43:581
30529995337,10cyclictest364-21in:imklog19:20:170
30529995332,8cyclictest141rcu_preempt22:26:460
30559995237,8cyclictest0-21swapper/320:17:143
30559995234,11cyclictest0-21swapper/322:52:413
30559995212,7cyclictest141rcu_preempt00:27:223
3054799525,8cyclictest141rcu_preempt22:07:232
30547995234,12cyclictest364-21in:imklog21:21:182
30547995234,12cyclictest364-21in:imklog21:21:182
30547995232,13cyclictest17691-21sed20:02:512
3053999527,9cyclictest185-21pr/ttyS019:12:511
30539995237,10cyclictest5070-21diskmemload22:03:591
30539995236,8cyclictest0-21swapper/120:17:501
30539995236,8cyclictest0-21swapper/119:27:151
3053999522,10cyclictest31259-21ssh22:52:411
30529995227,15cyclictest12-21ksoftirqd/020:37:370
30529995227,13cyclictest141rcu_preempt19:37:430
30529995218,7cyclictest0-21swapper/019:12:170
30529995212,28cyclictest0-21swapper/023:29:070
30529995212,28cyclictest0-21swapper/023:29:070
150432522,39sleep315047-21sh21:24:413
30559995137,7cyclictest0-21swapper/320:42:513
30559995133,11cyclictest364-21in:imklog22:12:533
30559995128,7cyclictest141rcu_preempt23:04:033
30559995128,7cyclictest141rcu_preempt23:04:033
30559995114,9cyclictest141rcu_preempt19:47:303
30547995134,11cyclictest364-21in:imklog22:00:212
30547995134,10cyclictest280-21systemd-journal21:10:122
3054799513,10cyclictest0-21swapper/223:52:172
30539995137,8cyclictest185-21pr/ttyS022:20:501
30539995134,10cyclictest364-21in:imklog21:09:271
30539995126,6cyclictest141rcu_preempt20:57:461
30539995117,8cyclictest141rcu_preempt21:32:151
30539995117,8cyclictest141rcu_preempt21:32:151
3052999517,15cyclictest280-21systemd-journal20:22:440
30529995134,10cyclictest5814-21latency_hist22:07:170
30529995133,10cyclictest12-21ksoftirqd/019:37:170
30529995133,10cyclictest12-21ksoftirqd/019:37:170
30529995132,12cyclictest0-21swapper/019:57:490
3052999513,20cyclictest0-21swapper/020:42:400
30529995114,14cyclictest0-21swapper/022:47:260
30559995025,7cyclictest141rcu_preempt20:37:433
30559995012,11cyclictest364-21in:imklog20:47:523
3054799508,13cyclictest33-21ksoftirqd/223:12:242
30547995036,9cyclictest122-21pr/ttymxc022:06:442
30547995035,9cyclictest280-21systemd-journal23:18:382
30547995035,8cyclictest0-21swapper/220:12:512
30547995033,10cyclictest26167-21sh20:32:142
30547995030,10cyclictest0-21swapper/220:42:522
3054799502,11cyclictest141rcu_preempt00:07:162
30547995012,17cyclictest33-21ksoftirqd/221:47:172
30547995010,8cyclictest141rcu_preempt20:47:432
3055999497,7cyclictest141rcu_preempt23:42:253
3055999493,19cyclictest0-21swapper/323:17:293
30547994934,9cyclictest122-21pr/ttymxc023:37:372
30547994934,9cyclictest122-21pr/ttymxc023:37:372
30547994933,8cyclictest0-21swapper/219:27:392
30547994933,10cyclictest280-21systemd-journal21:00:432
3054799493,12cyclictest8651-21apt-get19:37:312
30547994931,11cyclictest280-21systemd-journal20:52:232
3054799492,15cyclictest6502-21apt-get19:32:282
3054799492,15cyclictest6502-21apt-get19:32:282
30539994933,10cyclictest17622-21unixbench_multi20:02:511
30539994930,12cyclictest11211-21unixbench_singl19:42:511
30539994923,12cyclictest141rcu_preempt23:28:061
30539994923,12cyclictest141rcu_preempt23:28:061
30547994829,11cyclictest141rcu_preempt20:17:482
30539994834,8cyclictest185-21pr/ttyS020:52:331
30539994822,8cyclictest141rcu_preempt19:17:501
30529994831,10cyclictest0-21swapper/019:42:370
30529994813,14cyclictest280-21systemd-journal20:47:300
3054799473,22cyclictest12683-21sed19:47:522
30547994731,10cyclictest280-21systemd-journal23:02:262
30547994731,10cyclictest280-21systemd-journal23:02:262
30547994729,11cyclictest280-21systemd-journal22:27:042
30539994732,9cyclictest0-21swapper/119:57:151
30559994627,13cyclictest364-21in:imklog19:19:063
30547994632,8cyclictest0-21swapper/222:40:442
30547994632,8cyclictest0-21swapper/222:40:442
30547994629,11cyclictest364-21in:imklog21:02:402
30547994629,11cyclictest364-21in:imklog21:02:402
30547994629,10cyclictest280-21systemd-journal19:42:412
30539994620,7cyclictest141rcu_preempt23:42:561
30559994530,10cyclictest280-21systemd-journal19:33:103
30559994530,10cyclictest280-21systemd-journal19:33:103
30547994529,9cyclictest122-21pr/ttymxc020:41:002
30547994528,11cyclictest280-21systemd-journal19:17:502
30539994527,12cyclictest4644-21unixbench_multi21:02:501
30539994527,12cyclictest4644-21unixbench_multi21:02:501
30547994431,8cyclictest23602-21kworker/u8:219:22:372
30547994428,10cyclictest280-21systemd-journal19:11:042
30559994229,6cyclictest0-21swapper/320:02:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional