You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 02:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Thu Nov 21, 2024 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31843999270,14cyclictest9876-21latency_hist09:14:392
31823999040,21cyclictest1274-21proc_pri08:50:070
3098029022,26sleep30-21swapper/307:05:143
3098029022,26sleep30-21swapper/307:05:143
314302897,62sleep00-21swapper/007:06:130
314302897,62sleep00-21swapper/007:06:130
3184399846,40cyclictest15768-21ssh11:24:502
31843998457,18cyclictest21273-21latency_hist11:34:392
31823998349,26cyclictest17551-21fschecks_time09:29:560
31823998349,26cyclictest17551-21fschecks_time09:29:560
31823998329,19cyclictest25807-21/usr/sbin/munin10:45:050
31823998138,15cyclictest1121-21ssh10:59:520
3183299803,69cyclictest29026-21grep08:35:251
3183299803,69cyclictest29026-21grep08:35:251
3183299793,15cyclictest275-21systemd-journal10:16:361
31823997932,38cyclictest0-21swapper/011:25:140
31823997926,17cyclictest1-21systemd10:44:380
31843997758,12cyclictest4627-21basename07:24:372
3183299774,16cyclictest339-21timerandwakeup07:10:101
3183299773,15cyclictest394-21runrttasks08:55:101
3183299763,8cyclictest0-21swapper/109:34:511
3183299763,21cyclictest23557-21tail10:40:121
31823997628,23cyclictest6942-21diskmemload10:00:330
31823997628,23cyclictest6942-21diskmemload10:00:330
31823997626,23cyclictest6942-21diskmemload11:52:540
31823997623,29cyclictest14538-21sh11:22:370
31823997622,26cyclictest14378-21sh12:16:310
31823997617,42cyclictest0-21swapper/012:22:040
3183299754,17cyclictest27369-21ssh11:44:501
3183299754,17cyclictest27369-21ssh11:44:501
3183299753,18cyclictest11583-21/usr/sbin/munin10:20:021
3183299753,18cyclictest11583-21/usr/sbin/munin10:20:021
3183299753,18cyclictest10078-21fschecks_count12:10:011
3183299753,11cyclictest0-21swapper/111:54:581
3183299753,11cyclictest0-21swapper/111:54:581
3183299752,16cyclictest20928-21unixbench_singl08:10:271
3183299752,16cyclictest13844-21tail12:15:121
31823997524,28cyclictest25045-21ssh09:44:130
31823997523,22cyclictest28140-21ssh10:49:560
3168727512,38sleep231735-21idleruntime-cro07:09:372
3168727512,38sleep231735-21idleruntime-cro07:09:372
31852997427,18cyclictest2847-21runrttasks11:02:113
31852997412,34cyclictest13003-21ssh09:19:503
31852997412,34cyclictest13003-21ssh09:19:503
3183299744,58cyclictest16330-21latency_hist07:59:401
3183299744,58cyclictest16330-21latency_hist07:59:401
3183299744,16cyclictest18964-21/usr/sbin/munin12:24:471
3183299744,16cyclictest18964-21/usr/sbin/munin12:24:471
3183299743,64cyclictest15345-21latency_hist11:24:381
3183299743,13cyclictest16935-21grep11:25:371
3183299742,17cyclictest26161-21apt08:29:361
3183299742,17cyclictest26161-21apt08:29:361
247362740,2sleep30-21swapper/311:39:593
31852997321,22cyclictest18653-21chrt10:32:023
31843997352,14cyclictest487-21idleruntime-cro11:54:362
3183299734,60cyclictest31715-21latency_hist07:09:401
3183299734,60cyclictest31715-21latency_hist07:09:401
3183299734,15cyclictest19661-21cp08:09:401
3183299734,12cyclictest24084-21sendmail-msp11:39:371
3183299733,17cyclictest7626-21switchtime12:05:091
3183299733,17cyclictest7626-21switchtime12:05:091
3183299733,17cyclictest2083-21tail11:00:111
3183299733,16cyclictest13822-21wc09:20:331
3183299733,16cyclictest13822-21wc09:20:331
3183299733,13cyclictest4901-21grep09:00:101
3183299733,13cyclictest10224-21fschecks_count09:14:591
3183299733,13cyclictest10224-21fschecks_count09:14:591
3183299732,17cyclictest30581-21sort11:50:051
3183299732,13cyclictest25173-21ssh09:44:371
31823997326,16cyclictest12466-21ssh12:14:310
31823997324,25cyclictest6942-21diskmemload10:37:570
31823997324,19cyclictest32151-21ssh10:56:300
31823997321,28cyclictest6942-21diskmemload11:10:380
31823997321,28cyclictest6942-21diskmemload09:16:160
31823997321,28cyclictest6942-21diskmemload09:16:160
31852997228,18cyclictest10824-21chrt07:41:173
31843997250,13cyclictest0-21swapper/210:19:392
31843997212,31cyclictest0-21swapper/209:54:382
31843997212,31cyclictest0-21swapper/209:54:382
3183299724,60cyclictest6669-21fschecks_count07:29:571
3183299724,59cyclictest16887-21latency_hist10:29:391
3183299724,13cyclictest20042-21ssh10:34:571
3183299723,62cyclictest28032-21grep09:49:381
3183299723,22cyclictest10702-21unixbench_singl07:40:121
3183299723,17cyclictest27044-21munin-run11:44:361
3183299723,17cyclictest16906-21grep12:20:131
3183299723,15cyclictest2971-21modprobe07:19:371
3183299722,10cyclictest17026-21ssh10:29:481
31823997224,18cyclictest6942-21diskmemload10:31:380
31823997223,18cyclictest11069-21ssh10:18:580
31823997222,29cyclictest6942-21diskmemload10:13:100
31823997222,18cyclictest10435-21/usr/sbin/munin07:40:100
31823997219,19cyclictest14506-21sh09:21:290
31823997219,19cyclictest14506-21sh09:21:290
31852997124,14cyclictest0-21swapper/307:44:493
31852997122,21cyclictest6942-21diskmemload10:29:133
31843997152,7cyclictest141rcu_preempt10:05:112
3183299714,16cyclictest5776-21tail07:25:111
3183299714,15cyclictest31192-21cut08:44:391
3183299714,15cyclictest31192-21cut08:44:391
3183299714,13cyclictest8287-21unixbench_singl09:10:101
3183299714,13cyclictest3536-21cat10:04:391
3183299714,13cyclictest3536-21cat10:04:391
3183299713,61cyclictest3954-21/usr/sbin/munin11:05:111
3183299713,61cyclictest3954-21/usr/sbin/munin11:05:111
3183299713,59cyclictest25295-21ssh12:34:521
3183299713,57cyclictest22498-21tail08:15:271
3183299713,57cyclictest22498-21tail08:15:271
3183299713,21cyclictest28567-21fschecks_time09:50:021
3183299713,21cyclictest28567-21fschecks_time09:50:021
3183299713,19cyclictest8296-21fschecks_count07:34:561
3183299713,19cyclictest23139-21grep12:30:171
3183299713,17cyclictest32086-21netstat08:45:591
3183299713,16cyclictest4076-21tune2fs10:04:571
3183299713,16cyclictest10723-21grep11:15:121
3183299713,15cyclictest28989-21grep10:50:111
3183299713,12cyclictest6882-21taskset11:09:581
3183299712,19cyclictest31546-21find09:55:081
3183299712,14cyclictest23352-21/usr/sbin/munin08:19:581
3183299712,10cyclictest18602-21/usr/sbin/munin11:30:111
31832997117,46cyclictest0-21swapper/107:44:521
31823997127,13cyclictest0-21swapper/011:46:320
31823997127,13cyclictest0-21swapper/011:46:320
31823997122,20cyclictest1228-21taskset07:14:420
31823997121,27cyclictest6942-21diskmemload12:31:150
31852997028,17cyclictest20814-21sh11:33:393
31852997025,16cyclictest6308-21ssh11:09:033
31852997025,16cyclictest6308-21ssh11:09:033
3183299704,59cyclictest6631-21latency_hist12:04:381
3183299704,59cyclictest0-21swapper/109:29:491
3183299704,59cyclictest0-21swapper/109:29:491
3183299704,11cyclictest3533-21cut07:19:581
3183299703,16cyclictest25521-21grep10:44:581
3183299703,15cyclictest17010-21awk09:29:361
3183299703,15cyclictest14402-21latency_hist07:54:381
3183299703,14cyclictest6742-21ntpq10:10:031
3183299703,13cyclictest26882-21/usr/sbin/munin08:30:071
3183299702,20cyclictest6172-21cut09:05:141
3183299702,17cyclictest16563-21ls07:59:571
31823997042,19cyclictest28056-21apt-get08:35:050
31823997042,19cyclictest28056-21apt-get08:35:050
31823997024,15cyclictest20277-21runrttasks11:31:580
31823997022,19cyclictest25409-21fschecks_count12:34:560
31823997019,19cyclictest29493-21ssh09:50:510
31823997019,19cyclictest29493-21ssh09:50:510
31852996925,16cyclictest6135-21taskset07:28:353
31843996939,12cyclictest141rcu_preempt09:05:502
31843996933,17cyclictest0-21swapper/212:28:132
31843996933,17cyclictest0-21swapper/212:28:132
3183299694,55cyclictest30901-21fschecks_time10:54:571
3183299694,15cyclictest1611-21sed08:50:111
31823996951,8cyclictest5579-21apt-get09:04:490
31823996924,15cyclictest7942-21ssh12:05:160
31823996924,15cyclictest7942-21ssh12:05:160
31823996920,19cyclictest6206-21ssh12:03:460
31823996917,16cyclictest0-21swapper/011:38:150
31823996915,17cyclictest0-21swapper/012:29:060
31823996915,17cyclictest0-21swapper/012:29:060
3185299683,42cyclictest15742-21apt-get11:25:153
31852996822,18cyclictest31863-21chrt10:55:393
31852996821,17cyclictest4551-21seq07:23:573
3184399689,33cyclictest33-21ksoftirqd/210:51:032
3184399687,40cyclictest321ktimers/209:19:572
3184399687,40cyclictest321ktimers/209:19:572
31843996845,17cyclictest6239-21munin-plugin-st07:29:372
31843996830,18cyclictest141rcu_preempt10:39:382
31843996825,20cyclictest141rcu_preempt08:35:232
31843996825,20cyclictest141rcu_preempt08:35:232
31843996821,22cyclictest0-21swapper/208:50:082
31843996821,18cyclictest0-21swapper/212:19:032
31823996821,21cyclictest15303-21sed07:55:080
31823996821,21cyclictest15303-21sed07:55:080
31823996820,20cyclictest394-21runrttasks07:10:470
31823996819,19cyclictest32162-21ssh09:55:590
31823996819,19cyclictest22078-21ssh09:38:260
31823996819,18cyclictest21709-21/usr/sbin/munin08:14:580
31823996819,18cyclictest21709-21/usr/sbin/munin08:14:580
31823996814,17cyclictest394-21runrttasks07:20:530
31852996752,8cyclictest29400-21idleruntime-cro08:39:363
31852996752,8cyclictest29400-21idleruntime-cro08:39:363
31852996747,13cyclictest30628-21fschecks_time11:50:053
31852996720,20cyclictest17332-21chrt12:21:083
31843996733,18cyclictest141rcu_preempt12:34:382
31823996748,9cyclictest6495-21apt-get07:29:510
31823996728,31cyclictest3968-21ssh10:04:550
31823996724,34cyclictest30369-21/usr/sbin/munin08:40:120
31823996724,34cyclictest30369-21/usr/sbin/munin08:40:120
31823996720,19cyclictest3546-21systemd-run11:59:370
31823996720,19cyclictest3546-21systemd-run11:59:370
31823996719,18cyclictest26244-21taskset11:42:230
31823996719,18cyclictest12027-21ssh11:18:510
31852996612,19cyclictest394-21runrttasks07:19:313
31843996651,9cyclictest16121-21fschecks_count12:19:592
31843996647,12cyclictest14565-21fschecks_time10:24:562
31843996644,14cyclictest22927-21idleruntime-cro08:19:362
31843996644,14cyclictest22927-21idleruntime-cro08:19:362
31843996640,17cyclictest4221-21fschecks_time08:59:572
31843996633,11cyclictest0-21swapper/208:13:242
31843996610,25cyclictest0-21swapper/211:09:092
31843996610,25cyclictest0-21swapper/211:09:092
31823996624,14cyclictest6942-21diskmemload10:21:210
31823996624,14cyclictest6942-21diskmemload10:21:210
31823996621,24cyclictest6942-21diskmemload09:13:490
31823996616,21cyclictest26344-21/usr/sbin/munin09:45:100
31852996538,17cyclictest25501-21apt-get09:44:473
31852996521,18cyclictest19175-21chrt08:06:273
31852996521,17cyclictest28764-21taskset11:46:063
31852996521,17cyclictest28764-21taskset11:46:063
31852996519,20cyclictest23441-21sh12:30:503
31852996519,18cyclictest6942-21diskmemload12:13:073
31843996545,13cyclictest29948-21fschecks_time08:40:002
31843996545,13cyclictest29948-21fschecks_time08:40:002
31843996543,12cyclictest1814-21tail11:55:112
31843996543,12cyclictest1814-21tail11:55:112
31843996537,8cyclictest141rcu_preempt08:04:392
31843996533,8cyclictest141rcu_preempt11:44:382
31843996529,8cyclictest141rcu_preempt07:15:102
31843996526,18cyclictest141rcu_preempt09:56:282
31843996521,11cyclictest0-21swapper/210:59:402
31823996542,15cyclictest0-21swapper/007:50:030
31823996529,25cyclictest8279-21seq07:34:560
31823996520,17cyclictest24210-21seq08:20:270
31823996520,17cyclictest19050-21awk08:05:130
31823996519,18cyclictest16643-21ssh09:28:230
31823996518,17cyclictest16497-21seq10:28:560
31823996513,24cyclictest4360-21sed09:00:000
31852996416,20cyclictest24336-21runrttasks08:22:053
3184399649,29cyclictest33-21ksoftirqd/212:37:442
3184399649,23cyclictest321ktimers/207:59:372
3184399649,23cyclictest321ktimers/207:59:372
3184399647,30cyclictest33-21ksoftirqd/210:02:592
3184399647,30cyclictest33-21ksoftirqd/210:02:592
31843996431,7cyclictest141rcu_preempt09:45:092
31843996413,28cyclictest0-21swapper/209:15:032
31843996413,28cyclictest0-21swapper/209:15:032
31843996412,23cyclictest321ktimers/209:29:392
31823996418,16cyclictest5301-21sh11:05:570
31823996418,16cyclictest5301-21sh11:05:570
31852996336,18cyclictest23817-21/usr/sbin/munin09:40:113
31852996323,22cyclictest394-21runrttasks08:29:113
31852996323,22cyclictest394-21runrttasks08:29:113
31852996314,20cyclictest10304-21ssh11:15:063
31843996334,10cyclictest141rcu_preempt11:19:592
31843996332,7cyclictest141rcu_preempt11:15:102
31843996329,8cyclictest141rcu_preempt08:29:412
31843996329,8cyclictest141rcu_preempt08:29:412
31843996328,18cyclictest0-21swapper/210:23:312
31843996328,18cyclictest0-21swapper/210:23:312
31843996321,19cyclictest141rcu_preempt10:10:162
31843996313,24cyclictest0-21swapper/212:00:192
31852996244,12cyclictest1814-21unixbench_multi11:55:113
31852996244,12cyclictest1814-21unixbench_multi11:55:113
31852996222,14cyclictest6942-21diskmemload12:26:583
31852996222,14cyclictest6942-21diskmemload12:26:583
31852996216,16cyclictest394-21runrttasks09:12:293
31852996213,21cyclictest15490-21ssh12:19:303
31843996224,21cyclictest0-21swapper/210:56:142
31843996215,14cyclictest0-21swapper/207:30:212
31843996212,27cyclictest0-21swapper/207:39:542
31843996211,26cyclictest0-21swapper/209:35:442
3182399626,35cyclictest19937-21apt-get08:09:520
31823996226,29cyclictest0-21swapper/008:04:390
31852996141,12cyclictest0-21swapper/312:39:363
31852996137,16cyclictest20085-21ls10:34:583
31852996134,17cyclictest6061-21latency_hist10:09:393
31852996122,14cyclictest394-21runrttasks09:00:273
31852996119,18cyclictest0-21swapper/311:09:553
31852996115,20cyclictest30952-21taskset08:43:513
31852996115,20cyclictest30952-21taskset08:43:513
31843996128,17cyclictest141rcu_preempt11:11:332
31843996116,16cyclictest311rcuc/212:06:232
31843996116,16cyclictest311rcuc/212:06:232
31843996112,22cyclictest321ktimers/210:30:022
31823996124,11cyclictest12374-21taskset07:45:300
31852996043,10cyclictest7825-21unixbench_multi12:05:103
31852996043,10cyclictest7825-21unixbench_multi12:05:103
31852996020,18cyclictest770-21chrt09:59:363
31852996018,20cyclictest9141-21taskset07:35:483
31852996014,16cyclictest6616-21chrt09:05:503
31852996011,22cyclictest1174-21munin-node08:49:473
31843996038,14cyclictest0-21swapper/207:39:372
31843996034,8cyclictest141rcu_preempt07:11:422
31843996030,8cyclictest141rcu_preempt09:31:112
31843996030,8cyclictest141rcu_preempt09:31:112
31843996026,8cyclictest141rcu_preempt10:49:392
31843996024,21cyclictest0-21swapper/207:45:472
31843996023,22cyclictest0-21swapper/208:48:392
31843996012,21cyclictest0-21swapper/210:39:562
31823996036,16cyclictest3663-21modprobe08:59:370
31823996033,17cyclictest31710-21cstates08:45:490
31823996020,32cyclictest26285-21ls08:29:390
31823996020,32cyclictest26285-21ls08:29:390
31852995942,11cyclictest8640-21sh10:14:393
31852995942,11cyclictest21701-21fschecks_time11:34:593
31852995939,13cyclictest12578-21sed09:19:383
31852995939,13cyclictest12578-21sed09:19:383
3185299593,36cyclictest19937-21apt-get08:09:503
31852995917,15cyclictest6942-21diskmemload10:46:303
31843995933,7cyclictest141rcu_preempt08:32:372
31843995929,15cyclictest141rcu_preempt11:47:032
31843995929,15cyclictest141rcu_preempt11:47:032
31843995925,9cyclictest141rcu_preempt11:34:562
31852995825,16cyclictest0-21swapper/312:03:363
31852995820,18cyclictest23734-21taskset10:40:473
31852995812,18cyclictest394-21runrttasks09:38:293
31852995812,18cyclictest21537-21/usr/sbin/munin08:15:093
31852995812,18cyclictest21537-21/usr/sbin/munin08:15:093
31843995832,7cyclictest141rcu_preempt12:14:192
31843995823,7cyclictest0-21swapper/209:44:372
31843995814,23cyclictest321ktimers/207:54:372
31843995812,28cyclictest33-21ksoftirqd/208:19:572
31823995839,11cyclictest26426-21apt-get08:29:480
31852995732,15cyclictest27207-21/usr/sbin/munin08:30:253
31852995723,27cyclictest1204-21sort09:59:523
31852995723,27cyclictest1204-21sort09:59:523
31852995713,17cyclictest14920-21ssh11:23:463
31852995712,16cyclictest394-21runrttasks07:11:233
31852995710,20cyclictest6942-21diskmemload09:25:293
31852995710,18cyclictest28112-21ssh10:49:543
31843995730,8cyclictest141rcu_preempt08:54:572
31852995635,11cyclictest2317-21apt-get08:54:473
31843995630,7cyclictest141rcu_preempt08:04:592
31823995632,16cyclictest5144-21fschecks_time07:24:590
31852995541,7cyclictest0-21swapper/309:29:553
31852995541,7cyclictest0-21swapper/309:29:553
3185299549,18cyclictest15791-21taskset07:57:583
3185299549,18cyclictest15791-21taskset07:57:583
31852995339,9cyclictest0-21swapper/310:15:083
31852995328,17cyclictest633-21latency_hist08:49:393
31852995325,19cyclictest8074-21latency_hist07:34:403
3185299523,19cyclictest12473-21timerandwakeup10:20:113
3185299523,19cyclictest12473-21timerandwakeup10:20:113
31852995231,14cyclictest16389-21date07:59:403
31852995230,15cyclictest14322-21latency_hist07:54:363
31852995228,15cyclictest28516-21fschecks_time09:50:013
31852995228,15cyclictest28516-21fschecks_time09:50:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional