You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-27 - 01:09
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Dec 26, 2024 12:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2508621410,20sleep20-21swapper/207:02:502
26225991274,94cyclictest24744-21apt-get10:52:532
2837721180,4chrt28374-21ntp_states10:57:461
2541529819,47sleep0412-21snmpd07:03:010
2617499874,41cyclictest14230-21ssh10:33:000
2617499874,41cyclictest14230-21ssh10:33:000
257992876,14sleep10-21swapper/107:05:391
26250998661,15cyclictest1031-21unixbench_multi10:07:473
26193998665,8cyclictest0-21swapper/110:53:021
2625099852,14cyclictest16453-21sed08:12:513
26193998424,25cyclictest16926-21ssh11:36:581
26174998232,12cyclictest0-21swapper/009:27:480
26174998232,12cyclictest0-21swapper/009:27:480
26174998127,33cyclictest9250irq/72-mmc207:07:300
26174998122,31cyclictest0-21swapper/012:02:440
26250998038,34cyclictest0-21swapper/309:32:503
26193998056,17cyclictest13737-21grep12:28:191
26250997930,40cyclictest0-21swapper/311:57:433
26250997914,52cyclictest394-21runrttasks10:52:413
26174997955,14cyclictest8358-21unixbench_singl12:17:520
26250997859,12cyclictest12515-21ssh12:27:263
26250997838,33cyclictest0-21swapper/308:37:163
2625099783,20cyclictest27747-21/usr/sbin/munin10:57:403
2625099783,18cyclictest14692-21cut11:32:403
2625099782,19cyclictest28244-21fschecks_time09:57:363
26174997810,60cyclictest12502-21apt-get12:27:310
2535027812,33sleep30-21swapper/307:03:003
2625099773,12cyclictest15819-21ssh12:32:373
2625099764,17cyclictest13387-21ssh10:32:173
2625099763,23cyclictest27640-21tune2fs08:47:383
2625099763,16cyclictest3898-21wc10:12:513
2625099762,14cyclictest8167-21grep09:17:523
2622599763,35cyclictest12585-21ssh12:27:372
2625099753,19cyclictest6099-21grep07:42:393
2625099753,18cyclictest9119-21sort07:52:333
2625099753,16cyclictest3534-21unixbench_singl07:32:503
2625099753,16cyclictest3534-21unixbench_singl07:32:503
2625099753,16cyclictest23429-21grep11:47:483
2625099753,16cyclictest18568-21unixbench_singl09:37:523
2625099752,20cyclictest1150-21latency_hist12:07:193
2625099752,18cyclictest21861-21grep08:28:213
2625099752,16cyclictest17688-21fschecks_time11:37:403
26193997558,11cyclictest20073-21idleruntime-cro09:42:171
26174997548,16cyclictest31535-21apt-key12:02:200
2625099744,18cyclictest5177-21unixbench_singl07:37:523
2625099744,17cyclictest1922-21chrt07:27:503
2625099744,14cyclictest7639-21fschecks_count07:47:363
2625099743,22cyclictest25104-21grep08:37:513
2625099743,21cyclictest275-21systemd-journal10:47:173
2625099743,18cyclictest711-21grep09:02:513
2625099743,18cyclictest3938-21latency_hist12:12:163
2625099743,18cyclictest3938-21latency_hist12:12:163
2625099743,16cyclictest11674-21fschecks_time11:27:343
26250997421,42cyclictest0-21swapper/307:17:373
2625099734,18cyclictest22481-21cut09:47:333
2625099734,18cyclictest22481-21cut09:47:333
2625099734,17cyclictest26392-21find08:42:483
2625099734,16cyclictest28112-21munin-run11:57:163
2625099734,16cyclictest21837-21grep10:47:343
2625099733,18cyclictest29615-21ps08:52:483
2625099733,18cyclictest29615-21ps08:52:483
2625099733,17cyclictest3643-21fschecks_time11:12:413
2625099733,14cyclictest0-21swapper/311:07:363
2625099732,22cyclictest28537-21find07:12:293
2625099732,21cyclictest6255-21grep10:17:383
2625099732,20cyclictest5178-21grep12:12:503
2625099732,17cyclictest25341-21tune2fs09:52:353
26250997319,45cyclictest0-21swapper/309:32:193
26250997319,45cyclictest0-21swapper/309:32:193
26250997311,10cyclictest0-21swapper/312:27:173
26174997342,11cyclictest32359-21kworker/u8:208:02:190
2625099723,20cyclictest32242-21latency_hist10:07:183
2625099723,19cyclictest13263-21grep08:02:503
2625099723,18cyclictest4877-21ssh09:12:453
2625099723,18cyclictest18804-21cmp08:22:203
2625099723,17cyclictest26770-21fschecks_time07:08:083
2625099723,14cyclictest2328-21threads09:07:483
2625099722,22cyclictest9234-21latency_hist11:22:193
2625099722,19cyclictest32597-21cat07:22:493
2625099722,19cyclictest32597-21cat07:22:493
2625099722,18cyclictest314002sleep308:57:483
2625099722,17cyclictest9989-21latency_hist12:22:183
2625099722,17cyclictest11652-21unixbench_singl07:57:533
26174997254,11cyclictest11073-21munin-run11:27:160
2625099713,20cyclictest10573-21latency_hist10:27:193
2625099713,17cyclictest14852-21unixbench_multi08:07:523
2625099713,16cyclictest22111-21ssh09:47:193
2625099713,13cyclictest17552-21find10:37:483
2625099713,13cyclictest17552-21find10:37:483
26250997123,40cyclictest0-21swapper/308:27:173
26250997123,40cyclictest0-21swapper/308:27:173
26225997147,16cyclictest4001-21modprobe09:12:162
26174997150,8cyclictest141rcu_preempt12:37:200
2625099703,19cyclictest14316-21fschecks_count10:33:043
2625099703,19cyclictest14316-21fschecks_count10:33:043
26250997010,11cyclictest0-21swapper/311:22:203
2617499706,43cyclictest412-21snmpd10:52:300
26174997027,32cyclictest0-21swapper/007:12:210
26174997025,17cyclictest9837-21latency_hist09:22:190
2625099693,18cyclictest32523-21latency_hist11:07:183
26250996917,10cyclictest0-21swapper/311:47:183
2622599693,41cyclictest26498-21apt-get07:07:302
26174996933,30cyclictest0-21swapper/011:52:160
2625099682,21cyclictest10418-21runrttasks09:23:583
26193996843,15cyclictest1185-21idleruntime-cro09:07:171
26174996821,10cyclictest0-21swapper/010:17:330
26225996743,15cyclictest657-21cron09:42:212
26193996752,9cyclictest23270-21tail09:47:471
26193996752,9cyclictest23270-21tail09:47:471
26174996743,18cyclictest6054-21idleruntime-cro11:17:160
26174996743,15cyclictest25767-21ssh11:52:440
26174996733,11cyclictest141rcu_preempt08:06:400
26225996647,13cyclictest31433-21latency_hist12:02:192
26225996645,11cyclictest4225-21apt-get07:37:312
2619399669,28cyclictest19049-21apt-get08:22:321
2619399669,28cyclictest19049-21apt-get08:22:321
26174996644,15cyclictest3913-21cron07:37:170
26174996644,15cyclictest3913-21cron07:37:170
26174996644,14cyclictest5265-21ssh09:12:530
2617499663,43cyclictest141rcu_preempt08:32:200
26225996510,37cyclictest0-21swapper/208:22:322
26225996510,37cyclictest0-21swapper/208:22:322
26174996532,10cyclictest141rcu_preempt09:51:180
26174996532,10cyclictest141rcu_preempt09:51:180
26225996446,12cyclictest179-21pr/ttyS012:07:432
26225996446,12cyclictest179-21pr/ttyS012:07:432
26225996443,10cyclictest0-21swapper/209:17:162
26225996412,20cyclictest28569-21ssh09:57:442
26193996447,10cyclictest15951-21runrttasks12:32:391
26193996423,15cyclictest1175-21diskmemload10:31:041
26174996443,14cyclictest19859-21kworker/u8:010:43:270
26174996438,17cyclictest28625-21threads10:57:510
26174996433,9cyclictest141rcu_preempt12:26:070
26225996320,18cyclictest10778-21sh12:23:582
26193996341,13cyclictest24260-21apt-key10:52:171
26193996340,14cyclictest3938-21idleruntime-cro07:37:171
26193996340,14cyclictest3938-21idleruntime-cro07:37:171
26193996316,19cyclictest1175-21diskmemload12:26:521
26174996332,8cyclictest141rcu_preempt12:07:200
26174996332,8cyclictest141rcu_preempt12:07:200
26174996325,19cyclictest0-21swapper/009:37:380
26225996232,19cyclictest3114-21/usr/sbin/munin07:32:432
26225996232,19cyclictest3114-21/usr/sbin/munin07:32:432
26193996234,17cyclictest275-21systemd-journal08:02:371
26193996225,9cyclictest0-21swapper/108:32:511
2617499626,35cyclictest0-21swapper/007:52:320
26174996225,21cyclictest0-21swapper/010:42:190
26174996225,21cyclictest0-21swapper/010:42:190
2617499622,35cyclictest4497-21sort12:12:370
26225996133,20cyclictest0-21swapper/208:07:172
26193996119,17cyclictest0-21swapper/109:01:441
26174996139,11cyclictest0-21swapper/011:27:470
26174996129,10cyclictest0-21swapper/010:24:150
26174996127,9cyclictest141rcu_preempt10:07:180
26174996126,8cyclictest0-21swapper/011:47:180
26174996126,28cyclictest0-21swapper/008:22:210
26225996043,9cyclictest0-21swapper/212:37:162
26193996017,16cyclictest19167-21ssh11:39:471
26193996011,21cyclictest1175-21diskmemload09:55:271
26174996040,7cyclictest141rcu_preempt09:24:000
26174996040,13cyclictest27130-21awk08:47:160
2617499603,20cyclictest16919-21munin-run08:17:170
26225995937,12cyclictest0-21swapper/207:17:502
26193995942,10cyclictest0-21swapper/111:47:481
26193995939,10cyclictest0-21swapper/109:37:161
26193995938,10cyclictest6230-21sh11:17:191
26193995937,15cyclictest8215-21unixbench_singl09:17:521
26193995932,18cyclictest7677-21fschecks_time07:47:361
26193995929,20cyclictest20953-21apt-get08:27:581
26193995919,28cyclictest17233-21cpu08:17:311
26193995915,20cyclictest1175-21diskmemload10:32:431
26193995915,20cyclictest1175-21diskmemload10:32:431
26174995943,10cyclictest24332-21fschecks_count08:37:370
26174995934,13cyclictest0-21swapper/008:07:520
26174995931,22cyclictest394-21runrttasks11:41:050
26174995922,30cyclictest0-21swapper/011:17:400
26225995842,10cyclictest14069-21idleruntime-cro11:32:162
26193995840,9cyclictest0-21swapper/108:57:171
26193995840,9cyclictest0-21swapper/108:57:171
26193995830,18cyclictest6393-21apt-get11:17:321
26193995822,25cyclictest11052-21cron11:27:161
26174995838,13cyclictest2007-21ssh10:10:230
26174995835,15cyclictest1321-21unixbench_multi11:07:470
26174995835,13cyclictest0-21swapper/007:27:500
26174995824,26cyclictest30394-21sendmail-msp08:57:170
26174995824,26cyclictest30394-21sendmail-msp08:57:170
175002580,6sleep20-21swapper/208:17:372
26225995743,7cyclictest0-21swapper/209:47:342
26225995743,7cyclictest0-21swapper/209:47:342
26225995741,8cyclictest0-21swapper/210:07:472
26225995739,11cyclictest1274-21latency_hist09:07:192
26225995735,14cyclictest4437-21unixbench_singl11:12:542
26225995734,16cyclictest12022-21cut09:27:332
26225995734,16cyclictest12022-21cut09:27:332
26193995741,9cyclictest0-21swapper/110:17:191
26193995732,18cyclictest27970-21proc_pri08:47:471
26174995734,14cyclictest1599-21ssh09:07:320
26174995727,9cyclictest12-21ksoftirqd/008:47:330
26174995727,13cyclictest12-21ksoftirqd/010:31:150
26174995726,16cyclictest31539-21apt07:22:170
26174995716,25cyclictest12-21ksoftirqd/009:45:160
2617499571,51cyclictest32359-21kworker/u8:208:22:510
2617499571,51cyclictest32359-21kworker/u8:208:22:510
2622599563,29cyclictest1430-21awk11:07:492
26225995627,23cyclictest30917-21fschecks_time08:57:362
26225995624,22cyclictest5816-21/usr/sbin/munin07:42:342
26225995618,31cyclictest21524-21ps08:28:182
26225995610,17cyclictest1175-21diskmemload10:37:122
26225995610,17cyclictest1175-21diskmemload10:37:122
26193995634,14cyclictest0-21swapper/112:07:281
26193995634,14cyclictest0-21swapper/112:07:281
26193995633,15cyclictest12248-21latency09:27:371
26193995633,15cyclictest12248-21latency09:27:371
26193995630,14cyclictest0-21swapper/109:12:321
26193995623,22cyclictest5205-21/usr/sbin/munin07:37:521
26193995614,17cyclictest0-21swapper/111:30:481
2617499565,19cyclictest25898-21switchtime09:52:470
2617499564,40cyclictest703-21apt07:27:170
2617499564,40cyclictest703-21apt07:27:170
26174995633,9cyclictest12-21ksoftirqd/011:04:130
2617499563,18cyclictest30547-21latency_hist10:02:200
2617499563,17cyclictest15444-21sed11:32:530
2617499562,45cyclictest5572-21apt07:42:160
2622599555,23cyclictest141rcu_preempt11:54:162
26225995539,7cyclictest16859-21ssh10:37:282
26225995539,7cyclictest16859-21ssh10:37:282
26225995539,10cyclictest13640-21latency_hist10:32:192
26225995538,12cyclictest17216-21kworker/u8:009:37:162
26225995536,11cyclictest0-21swapper/209:27:162
26225995533,12cyclictest0-21swapper/210:57:522
26225995531,16cyclictest6407-21ssh11:17:302
26225995524,23cyclictest22691-21fschecks_count08:32:392
26225995518,14cyclictest141rcu_preempt10:17:182
26225995517,26cyclictest29969-21latency_hist07:17:192
26193995539,11cyclictest0-21swapper/111:07:161
26193995539,10cyclictest6998-21systemd-run12:17:171
26193995536,9cyclictest0-21swapper/110:26:241
26193995524,23cyclictest28546-21apt-get07:12:321
26193995510,22cyclictest141rcu_preempt09:11:311
26174995539,8cyclictest0-21swapper/010:52:170
2617499552,12cyclictest5597-21latency_hist10:17:190
26225995433,10cyclictest0-21swapper/208:47:182
26225995432,14cyclictest0-21swapper/211:47:392
26225995430,13cyclictest0-21swapper/210:27:182
26225995426,18cyclictest29009-21apt-get08:52:332
26225995426,18cyclictest29009-21apt-get08:52:332
26225995423,23cyclictest971-21taskset07:27:272
26193995436,9cyclictest13937-21apt-get08:07:291
26193995431,12cyclictest25436-21apt-get11:52:371
26193995430,15cyclictest20851-21ssh10:45:541
26193995425,22cyclictest16308-21timerandwakeup08:12:501
26193995424,22cyclictest26247-21/usr/sbin/munin08:42:431
26193995422,22cyclictest32473-21/usr/sbin/munin10:07:231
26174995432,12cyclictest0-21swapper/007:47:400
26225995337,10cyclictest10467-21systemd-run07:57:182
26225995328,15cyclictest275-21systemd-journal10:47:362
26225995328,14cyclictest0-21swapper/208:07:382
26225995327,16cyclictest31146-21ssh11:03:352
26225995325,17cyclictest7892-21/usr/sbin/munin07:47:402
26225995322,8cyclictest179-21pr/ttyS009:37:422
26225995322,21cyclictest11054-21cron11:27:162
26225995319,24cyclictest16429-21/usr/sbin/munin08:12:512
2619399539,12cyclictest0-21swapper/111:57:471
26193995337,9cyclictest0-21swapper/107:08:221
26193995332,13cyclictest0-21swapper/109:47:161
26193995330,14cyclictest0-21swapper/107:42:391
26193995330,11cyclictest0-21swapper/109:27:191
26193995324,20cyclictest6103-21/usr/sbin/munin10:17:341
26193995319,8cyclictest1110-21df07:27:331
26225995224,19cyclictest11144-21cat07:57:432
26225995220,24cyclictest18342-21ssh11:37:532
26225995218,26cyclictest8288-21grep12:17:512
26193995235,10cyclictest1-21systemd12:22:171
26193995232,11cyclictest10705-21apt-get07:57:301
26193995231,13cyclictest31520-21ssh10:05:191
26193995225,18cyclictest24141-21sshd08:37:271
26193995224,18cyclictest9567-21/usr/sbin/munin07:52:441
26174995211,24cyclictest0-21swapper/009:32:520
26225995131,11cyclictest0-21swapper/209:52:352
26225995130,15cyclictest15389-21unixbench_multi11:32:532
26193995129,13cyclictest30334-21latency_hist10:02:191
26193995129,12cyclictest0-21swapper/111:47:161
26193995116,28cyclictest394-21runrttasks10:42:111
26193995116,28cyclictest394-21runrttasks10:42:111
2617499515,18cyclictest30341-21kworker/u8:209:02:310
26225995031,11cyclictest0-21swapper/207:22:502
26225995031,11cyclictest0-21swapper/207:22:502
26225995027,16cyclictest919-21cron12:07:162
26225995023,18cyclictest24360-21fschecks_count08:37:372
26225995022,20cyclictest4535-21fschecks_time12:12:372
26193995024,17cyclictest32314-21sed07:22:431
26193995024,17cyclictest32314-21sed07:22:431
26193995023,18cyclictest514-21/usr/sbin/munin11:07:331
26193995021,21cyclictest31534-21sh07:22:171
2617499503,15cyclictest30624-21/usr/sbin/munin08:57:450
26225994929,9cyclictest0-21swapper/210:07:162
26225994924,17cyclictest22076-21cron11:47:152
26225994924,16cyclictest27479-21cpuspeed08:47:332
26193994930,10cyclictest0-21swapper/112:07:181
2617499499,12cyclictest141rcu_preempt08:32:510
26174994925,13cyclictest0-21swapper/007:42:520
26225994824,15cyclictest21477-21cat10:47:192
26225994812,26cyclictest8101-21wc09:17:512
26225994722,17cyclictest6171-21fschecks_count10:17:382
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional