You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-22 - 08:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Fri Nov 22, 2024 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2137211420,61sleep12236-21grep19:05:281
172821046,15sleep20-21swapper/219:05:162
2721998846,32cyclictest26249-21cron20:19:351
2721998846,32cyclictest26249-21cron20:19:351
272099886,59cyclictest5196-21apt-get19:15:120
2720998841,9cyclictest0-21swapper/020:44:350
2720998633,18cyclictest0-21swapper/022:25:070
136662850,6sleep30-21swapper/323:02:533
136662850,6sleep30-21swapper/323:02:533
2721998438,38cyclictest31448-21sh21:44:351
272099834,38cyclictest14273-21apt-get23:04:550
2721998142,31cyclictest14865-21kworker/u8:223:24:371
2721998142,31cyclictest14865-21kworker/u8:223:24:371
129628119,25sleep00-21swapper/019:04:540
2723998062,9cyclictest5396-21cron23:44:353
2723998048,26cyclictest23789-21unixbench_singl21:30:143
2721998038,11cyclictest8276-21latency_hist19:24:371
2722997859,12cyclictest30578-21unixbench_multi22:35:092
2723997753,15cyclictest3545-21munin-plugin-st20:49:353
2723997635,31cyclictest0-21swapper/321:50:023
35472750,5sleep20-21swapper/222:45:012
2723997436,28cyclictest0-21swapper/300:00:053
272199745,10cyclictest5393-21ssh21:54:441
272199745,10cyclictest5393-21ssh21:54:441
272199744,13cyclictest18638-21grep23:10:071
2720997417,30cyclictest18400-21apt-get19:55:050
272399733,60cyclictest10074-21apt-get19:29:473
2722997353,13cyclictest14639-21diskmemload21:15:102
272199734,61cyclictest23132-21modprobe23:19:341
272199734,60cyclictest32743-21modprobe00:34:341
272199733,16cyclictest27484-21head20:20:091
272199733,11cyclictest10248-21cut19:29:551
2723997236,30cyclictest23241-21unixbench_singl00:15:093
272399723,60cyclictest5389-21apt-get21:54:463
272399723,60cyclictest5389-21apt-get21:54:463
272199723,62cyclictest3823-21grep23:40:101
272199723,18cyclictest5571-21modprobe22:49:351
272199723,17cyclictest394-21runrttasks19:14:381
272199723,15cyclictest14549-21/usr/sbin/munin22:10:031
272199723,13cyclictest24635-21modprobe20:14:351
2722997149,14cyclictest0-21swapper/222:15:102
272199714,7cyclictest0-21swapper/119:34:471
272199714,7cyclictest0-21swapper/119:34:471
272199714,14cyclictest7257-21cut20:59:561
272199714,14cyclictest7257-21cut20:59:561
272199713,14cyclictest27764-21latency_hist00:24:371
272199713,13cyclictest29213-21grep23:29:371
272199713,12cyclictest8744-21apt-get22:54:451
272199713,12cyclictest5196-21apt-get19:15:161
272199712,15cyclictest5665-21fschecks_time20:54:541
272199712,15cyclictest5665-21fschecks_time20:54:541
272199712,14cyclictest14910-21ssh23:05:131
272199712,13cyclictest938-21fschecks_time00:34:561
272199712,13cyclictest938-21fschecks_time00:34:561
2720997147,16cyclictest0-21swapper/000:14:370
2720997147,16cyclictest0-21swapper/000:14:370
272399702,60cyclictest0-21swapper/320:04:463
272399702,57cyclictest29630-21fschecks_time23:30:003
272199704,58cyclictest15837-21grep19:45:101
272199704,17cyclictest2927-21/usr/sbin/munin21:50:041
272199704,13cyclictest32572-21/usr/sbin/munin22:40:091
272199704,10cyclictest21665-21apt-get20:04:461
272199703,16cyclictest6208-21/usr/sbin/munin22:50:001
272199703,16cyclictest22456-21cut00:14:561
272199703,14cyclictest9361-21grep19:25:091
272199703,14cyclictest32091-21latency_hist23:34:371
272199703,14cyclictest23362-21modprobe22:24:341
272199703,12cyclictest11261-21/usr/sbin/munin21:10:061
272199702,18cyclictest14236-21sendmail_mailtr00:00:061
272199702,17cyclictest32326-21cpuspeed23:34:511
2720997049,14cyclictest19012-21munin-plugin-st00:09:340
272099703,22cyclictest8914-21apt-get21:04:500
272399694,57cyclictest0-21swapper/319:44:493
272399694,56cyclictest16685-21apt-get21:19:503
2723996934,30cyclictest0-21swapper/300:30:113
2721996953,10cyclictest23952-21fschecks_time22:24:551
272199694,15cyclictest14292-21idleruntime-cro23:04:351
272199694,15cyclictest14292-21idleruntime-cro23:04:351
272199693,59cyclictest17414-21grep00:05:101
272199693,57cyclictest4047-21fschecks_time20:49:571
272199693,17cyclictest28157-21fschecks_time00:24:561
272199693,17cyclictest28157-21fschecks_time00:24:561
272199693,15cyclictest19927-21ntpq00:10:071
272199693,15cyclictest19927-21ntpq00:10:071
272199693,14cyclictest31729-21/usr/sbin/munin21:44:561
272199693,14cyclictest30542-21ssh22:35:091
272199693,13cyclictest8824-21/usr/sbin/munin21:59:571
272199693,12cyclictest10549-21/usr/sbin/munin23:54:441
272199693,12cyclictest10450-21latency_hist21:09:371
2721996930,32cyclictest0-21swapper/120:34:351
2721996930,32cyclictest0-21swapper/120:34:351
272199692,13cyclictest7832-21/usr/sbin/munin23:49:491
272199692,13cyclictest31587-21tune2fs20:34:541
2721996912,48cyclictest10271-21diskmemload22:04:581
2721996912,48cyclictest10271-21diskmemload22:04:581
2723996851,10cyclictest0-21swapper/321:40:083
2723996831,30cyclictest0-21swapper/322:14:343
272199687,12cyclictest2988-21sed20:45:071
272199686,14cyclictest18602-21sort19:55:111
272199685,10cyclictest1921-21munin-plugin-st20:44:351
272199684,12cyclictest25978-21fschecks_time21:34:541
272199684,11cyclictest17796-21/usr/sbin/munin22:14:561
272199683,55cyclictest14212-21/usr/sbin/munin21:15:031
272199683,17cyclictest21402-21apt20:04:351
272199683,14cyclictest17471-21unixbench_multi19:50:091
272199683,13cyclictest22741-21ssh21:29:501
272199683,13cyclictest16665-21/usr/sbin/munin21:19:581
272199683,11cyclictest14254-21grep19:40:121
272199682,12cyclictest20387-21/usr/sbin/munin21:25:071
2720996847,14cyclictest20319-21munin-plugin-st22:19:350
2720996841,19cyclictest6778-21munin-run20:59:350
2720996841,19cyclictest6778-21munin-run20:59:350
2720996840,19cyclictest23366-21latency_hist23:19:370
272399674,55cyclictest0-21swapper/320:34:463
272399673,57cyclictest0-21swapper/322:34:473
272399673,56cyclictest0-21swapper/319:59:493
272399673,53cyclictest20537-21apt-get23:14:483
2723996733,25cyclictest0-21swapper/321:44:483
272399672,57cyclictest13167-21munin-run23:59:343
272399672,56cyclictest6781-21latency_hist19:19:383
2723996725,34cyclictest0-21swapper/320:19:353
2723996725,34cyclictest0-21swapper/320:19:353
272199677,9cyclictest26654-21/usr/sbin/munin22:29:541
2723996629,28cyclictest0-21swapper/322:25:093
272399662,57cyclictest14737-21apt-get23:05:133
272399662,55cyclictest18647-21grep22:15:103
2723996623,37cyclictest1007-21sh00:34:593
2723996623,37cyclictest1007-21sh00:34:593
272199664,55cyclictest28340-21fschecks_count20:24:541
272399654,51cyclictest11686-21apt-get19:34:473
272399654,51cyclictest11686-21apt-get19:34:473
272399653,53cyclictest32453-21fschecks_count23:34:553
272399653,53cyclictest24202-21grep20:10:103
272399653,52cyclictest7657-21apt-get23:49:433
272399653,52cyclictest26713-21grep21:35:073
272399653,52cyclictest14253-21unixbench_multi19:40:123
272399653,51cyclictest23486-21apt-get23:19:483
272399653,51cyclictest23486-21apt-get23:19:483
272399652,56cyclictest0-21swapper/322:39:493
272399652,55cyclictest16776-21fschecks_time19:49:563
272399652,53cyclictest13613-21/usr/sbin/munin21:15:033
272199652,13cyclictest7487-21wc23:49:371
2720996538,18cyclictest8657-21latency_hist21:04:370
2720996538,18cyclictest8657-21latency_hist21:04:370
2720996517,38cyclictest131ktimers/021:59:560
272399644,51cyclictest19359-21apt-get00:09:513
272399644,51cyclictest19359-21apt-get00:09:513
272399644,51cyclictest17621-21apt-get23:09:463
272399643,54cyclictest6144-21sort22:49:593
272399643,54cyclictest0-21swapper/321:05:103
272399643,52cyclictest563-21taskset20:39:463
272399643,52cyclictest28810-21grep00:25:093
272399643,52cyclictest28810-21grep00:25:093
272399643,52cyclictest19702-21/usr/sbin/munin21:24:523
272399642,5cyclictest0-21swapper/322:44:473
272399642,55cyclictest10826-21fschecks_count21:09:573
272399642,54cyclictest4402-21cat19:14:383
272399642,54cyclictest4066-21fschecks_time20:49:573
272399642,54cyclictest11413-21latency_hist22:04:373
272399642,53cyclictest9803-21unixbench_singl22:55:073
272399642,53cyclictest2847-21latency_hist19:09:493
2722996449,9cyclictest17035-21runrttasks00:05:062
2722996439,15cyclictest394-21runrttasks00:24:562
2722996439,15cyclictest394-21runrttasks00:24:562
272299643,13cyclictest141rcu_preempt20:59:352
272299643,13cyclictest141rcu_preempt20:59:352
272399635,49cyclictest9376-21unixbench_multi19:25:093
272399634,51cyclictest11831-21fschecks_time22:04:583
272399634,51cyclictest11831-21fschecks_time22:04:583
272399634,49cyclictest29764-21apt-get20:29:473
272399634,49cyclictest29764-21apt-get20:29:473
272399633,53cyclictest0-21swapper/320:24:463
272399632,6cyclictest0-21swapper/300:19:463
272399632,54cyclictest0-21swapper/319:19:493
272399632,53cyclictest5903-21awk20:54:593
272399632,53cyclictest5903-21awk20:54:593
272399632,52cyclictest8742-21latency_hist21:04:373
272399632,52cyclictest8742-21latency_hist21:04:373
272399632,52cyclictest17392-21grep00:05:103
2720996339,13cyclictest0-21swapper/023:24:340
2720996339,13cyclictest0-21swapper/023:24:340
272399624,52cyclictest0-21swapper/320:19:483
272399623,52cyclictest7404-21latency_hist23:49:363
272399623,52cyclictest18409-21apt-get19:54:493
272399623,50cyclictest208572sleep322:19:553
272399622,52cyclictest28881-21taskset23:29:163
272399622,52cyclictest27088-21cut22:30:033
2722996248,9cyclictest0-21swapper/219:19:352
2722996240,11cyclictest0-21swapper/222:05:112
2722996240,11cyclictest0-21swapper/222:05:112
2720996241,14cyclictest21408-21kworker/u8:100:35:230
2720996241,14cyclictest21408-21kworker/u8:100:35:230
2722996131,20cyclictest25873-21/usr/sbin/munin20:15:122
2722996131,20cyclictest25873-21/usr/sbin/munin20:15:122
2722996123,21cyclictest141rcu_preempt21:34:492
2720996137,14cyclictest18336-21latency_hist19:54:390
2722996044,10cyclictest376-21systemd-logind21:34:362
2722996034,16cyclictest275-21systemd-journal00:35:012
2722996034,16cyclictest275-21systemd-journal00:35:012
2722996025,15cyclictest141rcu_preempt23:49:532
2720996042,11cyclictest29554-21diskmemload23:29:590
2720996033,15cyclictest0-21swapper/023:39:560
2720996022,27cyclictest0-21swapper/021:16:310
2720996014,12cyclictest141rcu_preempt21:40:090
2720996010,22cyclictest12-21ksoftirqd/021:39:040
2722995945,7cyclictest0-21swapper/223:59:352
2720995932,17cyclictest0-21swapper/022:39:350
2720995923,28cyclictest0-21swapper/022:19:460
272099592,27cyclictest141rcu_preempt20:24:370
2722995835,15cyclictest627-21in:imklog22:25:052
2722995833,16cyclictest10271-21diskmemload23:41:032
2722995831,15cyclictest0-21swapper/200:14:372
2722995831,15cyclictest0-21swapper/200:14:372
2722995824,17cyclictest141rcu_preempt23:15:072
2722995821,7cyclictest141rcu_preempt23:45:412
2722995740,11cyclictest31114-21munin-plugin-st20:34:352
2722995740,11cyclictest31114-21munin-plugin-st20:34:352
2722995739,12cyclictest275-21systemd-journal23:29:352
2722995728,12cyclictest275-21systemd-journal23:05:422
2720995743,8cyclictest131ktimers/019:39:590
2720995733,14cyclictest0-21swapper/020:30:090
2720995733,14cyclictest0-21swapper/020:30:090
2720995732,16cyclictest0-21swapper/022:29:590
2720995724,23cyclictest0-21swapper/019:39:350
2720995724,23cyclictest0-21swapper/019:39:350
2722995640,10cyclictest15874-21unixbench_singl19:45:112
2722995636,13cyclictest376-21systemd-logind00:04:352
2722995634,12cyclictest0-21swapper/223:34:352
2722995630,15cyclictest0-21swapper/222:04:342
2720995643,7cyclictest131ktimers/022:49:580
272099564,16cyclictest24718-21munin-run00:19:340
2720995637,10cyclictest0-21swapper/023:00:070
2720995637,10cyclictest0-21swapper/023:00:070
2720995636,12cyclictest0-21swapper/020:14:350
2720995634,14cyclictest24613-21kworker/u8:020:39:340
272099563,18cyclictest20681-21wc21:25:100
2720995613,15cyclictest26253-21sh20:19:350
2720995613,15cyclictest26253-21sh20:19:350
2720995611,34cyclictest0-21swapper/021:55:070
2720995611,34cyclictest0-21swapper/021:55:070
2722995541,8cyclictest0-21swapper/221:55:082
2722995541,8cyclictest0-21swapper/221:55:082
2722995538,11cyclictest33-21ksoftirqd/221:39:452
2722995538,11cyclictest275-21systemd-journal19:29:352
2722995536,8cyclictest0-21swapper/222:34:342
2722995529,14cyclictest0-21swapper/222:20:102
272099559,20cyclictest12-21ksoftirqd/022:08:540
272099559,20cyclictest12-21ksoftirqd/022:08:540
272099557,11cyclictest0-21swapper/023:14:360
2720995534,13cyclictest0-21swapper/021:45:080
272099553,17cyclictest11333-21munin-run22:59:350
272099553,14cyclictest23740-21grep21:30:140
2722995433,14cyclictest321ktimers/222:56:452
2722995433,14cyclictest275-21systemd-journal20:19:562
2722995418,14cyclictest141rcu_preempt21:44:542
2720995437,10cyclictest447-21fschecks_time22:39:560
2720995433,13cyclictest29051-21latency_hist23:29:360
2720995432,12cyclictest0-21swapper/023:59:350
2720995428,8cyclictest0-21swapper/022:14:340
2720995426,17cyclictest0-21swapper/023:39:340
2720995422,15cyclictest141rcu_preempt00:31:060
2720995418,20cyclictest141rcu_preempt23:47:470
2720995412,25cyclictest141rcu_preempt20:50:100
2722995335,12cyclictest275-21systemd-journal00:18:042
2722995335,12cyclictest22233-21proc_pri20:05:052
2722995333,11cyclictest0-21swapper/220:49:352
2722995323,14cyclictest141rcu_preempt23:39:372
2722995318,10cyclictest0-21swapper/222:13:052
2720995329,14cyclictest4473-21apt-key19:14:380
2720995311,12cyclictest141rcu_preempt20:09:380
2722995237,9cyclictest10880-21fschecks_time21:09:572
2722995237,7cyclictest0-21swapper/219:54:392
2722995232,13cyclictest0-21swapper/220:14:352
2722995231,14cyclictest0-21swapper/220:04:372
272299522,12cyclictest141rcu_preempt19:20:002
272099529,36cyclictest7974-21fschecks_time23:49:530
272099529,21cyclictest12-21ksoftirqd/000:02:550
2720995238,8cyclictest8150irq/69-2188000.00:24:380
2720995237,9cyclictest10271-21diskmemload22:44:560
272099522,19cyclictest10970-21grep19:30:080
2720995217,9cyclictest0-21swapper/021:20:550
2722995137,7cyclictest0-21swapper/223:19:562
2722995137,7cyclictest0-21swapper/223:19:562
2722995135,8cyclictest0-21swapper/200:29:592
2722995134,9cyclictest0-21swapper/222:59:452
2722995134,9cyclictest0-21swapper/222:59:452
2722995134,9cyclictest0-21swapper/219:39:352
2722995134,9cyclictest0-21swapper/219:39:352
2722995134,10cyclictest8640-21apt-key21:04:372
2722995134,10cyclictest8640-21apt-key21:04:372
2722995132,13cyclictest32317-21unixbench_singl20:35:082
2722995114,13cyclictest0-21swapper/219:55:172
2720995127,13cyclictest0-21swapper/000:29:360
2720995127,13cyclictest0-21swapper/000:29:360
272099512,40cyclictest2858-21sort21:50:020
2722995035,9cyclictest394-21runrttasks20:39:562
2722995032,11cyclictest275-21systemd-journal21:24:382
2722995031,14cyclictest6756-21kworker/u8:321:29:382
2722995027,17cyclictest17598-21/usr/sbin/munin23:10:042
272099503,16cyclictest3095-21unixbench_multi20:45:080
2720995018,16cyclictest141rcu_preempt21:10:410
272299493,30cyclictest8413-21apt-get21:04:482
2722994932,11cyclictest14265-21unixbench_singl19:40:122
2722994931,9cyclictest0-21swapper/200:19:522
272099494,11cyclictest0-21swapper/019:44:520
272099493,36cyclictest29503-21munin-run20:29:350
2720994931,12cyclictest9818-21munin-plugin-st19:29:350
2722994832,10cyclictest275-21systemd-journal19:30:002
2722994832,10cyclictest275-21systemd-journal19:10:062
2722994831,11cyclictest10271-21diskmemload21:49:592
2722994826,14cyclictest0-21swapper/222:54:372
2720994725,9cyclictest20049-21apt-get19:59:490
2720994720,7cyclictest141rcu_preempt19:20:000
2722994626,14cyclictest275-21systemd-journal20:25:082
2722994428,9cyclictest394-21runrttasks20:50:022
2722994427,11cyclictest275-21systemd-journal22:39:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional