You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-02-05 - 17:57
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Feb 05, 2025 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
633821140,4chrt6339-21/usr/sbin/munin19:39:561
25054210413,61sleep024874-21/usr/sbin/munin19:00:000
263819910354,34cyclictest0-21swapper/020:19:060
263819910338,41cyclictest0-21swapper/023:02:520
26381999944,27cyclictest25053-21ssh23:55:390
26381999661,27cyclictest5205-21/usr/sbin/munin22:14:530
26381999661,27cyclictest5205-21/usr/sbin/munin22:14:530
26381999238,29cyclictest1514-21diskmemload23:28:590
26381999140,25cyclictest1514-21diskmemload22:29:280
26381999140,25cyclictest1514-21diskmemload22:29:280
26381999037,16cyclictest1514-21diskmemload23:12:140
26381998935,20cyclictest0-21swapper/000:00:130
26381998839,17cyclictest1514-21diskmemload00:22:270
26381998839,16cyclictest1514-21diskmemload23:08:240
26381998839,16cyclictest1514-21diskmemload23:08:240
26381998838,27cyclictest1514-21diskmemload00:26:490
26381998838,25cyclictest1514-21diskmemload22:31:400
26381998836,17cyclictest1514-21diskmemload22:54:440
26381998836,17cyclictest1514-21diskmemload22:54:440
2638799876,49cyclictest13642-21apt-get23:35:051
26381998766,12cyclictest32008-21apt-get22:04:440
26381998740,15cyclictest1514-21diskmemload00:30:140
26381998739,23cyclictest1514-21diskmemload21:17:070
26381998739,15cyclictest1514-21diskmemload21:08:260
2567228730,29sleep30-21swapper/319:00:203
26381998645,32cyclictest19084-21ssh21:36:230
26381998643,35cyclictest4787-21ssh21:09:400
26381998637,18cyclictest4623-21ssh23:16:140
26381998637,15cyclictest1514-21diskmemload22:21:170
26381998634,18cyclictest1514-21diskmemload21:22:490
26381998541,16cyclictest23096-21tail21:45:060
26381998540,15cyclictest21434-21latency20:25:200
26381998540,15cyclictest21434-21latency20:25:200
26381998534,15cyclictest394-21runrttasks21:52:520
26387998445,22cyclictest0-21swapper/122:35:091
26381998445,17cyclictest30849-21latency_hist00:09:360
26381998445,17cyclictest30849-21latency_hist00:09:360
26381998440,17cyclictest16494-21taskset23:40:020
26381998434,17cyclictest16645-21sh21:32:290
26400998327,19cyclictest25595-21taskset20:36:403
26381998340,15cyclictest1514-21diskmemload22:40:430
26381998337,16cyclictest28967-21ssh21:56:090
26381998336,19cyclictest5758-21taskset19:37:470
26381998334,30cyclictest20172-21taskset20:20:390
26400998235,23cyclictest1514-21diskmemload00:12:113
26400998235,23cyclictest1514-21diskmemload00:12:113
26381998244,29cyclictest21687-21/usr/sbin/munin23:50:050
26381998238,18cyclictest8795-21taskset19:45:230
26381998236,18cyclictest20528-21seq21:40:250
26381998236,15cyclictest0-21swapper/000:18:180
46832810,5sleep30-21swapper/321:09:383
26381998161,10cyclictest10596-21apt-get23:29:450
26381998139,33cyclictest22847-21/usr/sbin/munin20:29:460
26381998137,17cyclictest6015-21ssh23:19:530
26381998137,14cyclictest0-21swapper/022:00:030
26381998134,18cyclictest415-21sshd23:39:090
2536828114,37sleep125396-21ntpq19:00:091
26400997927,16cyclictest394-21runrttasks22:34:003
2638799797,53cyclictest2925-21apt19:29:481
2638799797,53cyclictest2925-21apt19:29:481
26381997963,8cyclictest11046-21apt-get19:54:470
26381997963,8cyclictest11046-21apt-get19:54:470
26381997956,15cyclictest15355-21apt-get22:35:070
26381997954,17cyclictest19284-21/usr/sbin/munin23:45:080
26381997936,16cyclictest23795-21taskset22:50:380
26381997935,14cyclictest31685-21ntpq00:10:030
26381997935,14cyclictest31685-21ntpq00:10:030
26381997931,14cyclictest0-21swapper/020:37:240
26381997856,15cyclictest29073-21idleruntime-cro20:49:340
26381997852,17cyclictest29953-21modprobe19:14:350
26381997837,14cyclictest30232-21/usr/sbin/munin19:15:080
26381997837,14cyclictest30232-21/usr/sbin/munin19:15:080
26381997834,15cyclictest13781-21taskset20:01:520
26381997834,13cyclictest0-21swapper/021:27:190
26381997753,16cyclictest20398-21/usr/sbin/munin22:45:020
26381997732,18cyclictest10733-21taskset19:54:320
26387997653,15cyclictest23506-21sendmail23:54:341
26381997653,16cyclictest4711-21ls22:14:350
26381997653,16cyclictest1350-21runrttasks21:02:480
26381997652,16cyclictest32698-21wc19:20:090
26381997651,17cyclictest4504-21latency_hist19:34:380
26381997651,17cyclictest4504-21latency_hist19:34:380
26381997629,20cyclictest27156-21runrttasks20:40:560
26381997558,9cyclictest29342-21apt-get20:49:480
26400997428,17cyclictest19292-21seq23:45:093
26400997426,17cyclictest10034-21runrttasks23:28:473
26381997452,15cyclictest32339-21idleruntime-cro20:59:350
2560527412,39sleep225670-21grep19:00:192
26400997326,18cyclictest4124-21seq19:32:523
26400997326,18cyclictest4124-21seq19:32:523
26400997326,17cyclictest16188-21sh21:30:593
26381997350,15cyclictest16957-21chrt20:11:150
26381997349,17cyclictest2495-21apt-key19:29:360
26381997349,16cyclictest26804-21tune2fs19:04:550
26381997349,16cyclictest26804-21tune2fs19:04:550
26381997348,16cyclictest15181-21tail20:05:120
26381997328,16cyclictest0-21swapper/019:41:380
26400997226,22cyclictest23359-21chrt21:45:453
2639399723,41cyclictest13642-21apt-get23:34:482
26400997150,15cyclictest16369-21grep22:35:313
26400997129,16cyclictest0-21swapper/322:16:303
26400997129,16cyclictest0-21swapper/322:16:303
26387997145,17cyclictest4535-21latency_hist21:09:381
26400997055,8cyclictest0-21swapper/300:30:103
2638799703,32cyclictest4299-21apt-get21:09:491
26400996944,17cyclictest4836-21apt-get21:10:053
26393996953,10cyclictest628-21rs:main2
26393996953,10cyclictest628-21rs:main2
26400996844,15cyclictest27903-21idleruntime-cro19:09:353
26400996844,15cyclictest27903-21idleruntime-cro19:09:353
26400996822,17cyclictest394-21runrttasks19:40:343
2639399683,25cyclictest15255-21apt-get22:34:522
26400996643,16cyclictest29659-21latency_hist23:04:363
26400996642,17cyclictest20633-21latency_hist20:24:373
26400996642,15cyclictest13465-21dpkg23:34:383
26400996641,18cyclictest24899-21sh23:55:113
26400996638,18cyclictest2849-21apt-key23:14:363
26400996638,16cyclictest27786-21sort21:54:553
26400996621,22cyclictest10529-21runrttasks19:51:583
26387996632,9cyclictest1514-21diskmemload00:28:251
26400996539,19cyclictest4655-21cpuspeed19:34:523
26400996539,18cyclictest8185-21ntpq21:15:023
26400996522,19cyclictest0-21swapper/323:43:583
26393996546,12cyclictest18888-21munin-plugin-st20:19:352
26387996544,15cyclictest27953-21fschecks_time20:45:001
26387996541,15cyclictest961-21sendmail_mailqu23:10:091
26400996445,12cyclictest25123-21latency_hist22:54:373
26400996442,14cyclictest32410-21munin-plugin-st23:09:353
26400996442,14cyclictest32410-21munin-plugin-st23:09:353
26400996441,15cyclictest17257-21munin-run20:14:343
26400996441,14cyclictest8320-21/usr/sbin/munin19:45:053
26400996440,18cyclictest0-21swapper/322:54:493
26400996440,18cyclictest0-21swapper/322:54:493
26400996439,16cyclictest25603-21/usr/sbin/munin21:50:083
26400996438,18cyclictest6929-21unixbench_singl00:25:093
26393996446,11cyclictest29975-21munin-run19:14:352
26393996424,25cyclictest30677-21kworker/u8:121:10:032
26400996338,16cyclictest405-21sort21:00:003
26400996337,18cyclictest2714-21cut19:29:383
26400996337,17cyclictest31265-21ssh22:03:033
26400996337,17cyclictest22068-21unixbench_singl23:50:093
26400996337,17cyclictest17536-21apt-get20:14:473
26400996336,18cyclictest30767-21/usr/sbin/munin19:15:023
26400996336,18cyclictest30767-21/usr/sbin/munin19:15:023
26400996336,17cyclictest30984-21apt-get20:54:523
26400996336,17cyclictest10440-21grep22:25:053
26400996336,17cyclictest10440-21grep22:25:053
26393996342,15cyclictest275-21systemd-journal19:39:352
26393996338,16cyclictest15088-21latency_hist22:34:382
26400996245,10cyclictest20499-21apt-get20:24:473
26400996245,10cyclictest20499-21apt-get20:24:473
26400996241,12cyclictest7552-21systemd-run23:24:353
26400996238,16cyclictest0-21swapper/300:14:463
26400996237,17cyclictest21787-21modprobe21:44:343
26400996236,18cyclictest14712-21latency_hist21:29:383
26400996233,18cyclictest29511-21/usr/sbin/munin20:49:563
26400996228,27cyclictest15131-21timerwakeupswit20:05:123
26387996241,14cyclictest0-21swapper/119:35:101
26387996223,18cyclictest0-21swapper/120:13:051
26400996142,11cyclictest27990-21fschecks_time20:45:003
26400996141,11cyclictest19625-21latency_hist22:44:373
26400996138,15cyclictest28645-21fschecks_count00:04:513
26400996138,15cyclictest28645-21fschecks_count00:04:513
26400996138,15cyclictest22109-21taskset22:49:073
26400996137,17cyclictest26368-21fschecks_time20:39:573
26400996137,16cyclictest11596-21ssh21:23:563
26400996136,17cyclictest3280-21unixbench_multi22:10:113
26400996136,17cyclictest14774-21grep23:35:273
26400996136,16cyclictest29561-21unixbench_singl19:10:113
26400996134,16cyclictest18658-21grep21:35:103
26393996147,7cyclictest0-21swapper/222:54:352
26393996140,12cyclictest0-21swapper/222:59:362
26393996140,12cyclictest0-21swapper/222:59:362
26393996139,12cyclictest0-21swapper/223:05:062
26393996139,12cyclictest0-21swapper/223:05:062
2638799617,37cyclictest22350-21apt-get20:29:481
26387996142,11cyclictest22400-21latency_hist22:49:371
2638799612,24cyclictest20499-21apt-get20:24:491
2638799612,24cyclictest20499-21apt-get20:24:491
26400996042,10cyclictest22851-21apt-get20:30:063
26400996040,15cyclictest0-21swapper/323:15:073
26400996036,17cyclictest416-21unixbench_singl19:20:113
26400996036,16cyclictest579-21head22:05:063
26400996036,16cyclictest275-21systemd-journal19:59:353
26400996036,16cyclictest275-21systemd-journal19:59:353
26387996025,8cyclictest5046-21sshd23:17:351
26387996012,26cyclictest241ktimers/100:04:421
26387996012,26cyclictest241ktimers/100:04:421
26387996012,25cyclictest241ktimers/123:04:571
26387996012,25cyclictest241ktimers/123:04:571
26387996010,25cyclictest241ktimers/122:40:311
26400995931,20cyclictest9162-21ssh22:24:343
26400995919,15cyclictest394-21runrttasks20:02:533
26393995939,13cyclictest0-21swapper/220:04:352
26387995941,9cyclictest25024-21cron22:54:351
26387995911,24cyclictest241ktimers/123:29:481
26393995838,10cyclictest11797-21kworker/u8:322:19:382
26393995838,10cyclictest11797-21kworker/u8:322:19:382
26393995835,16cyclictest7534-21kworker/u8:119:44:352
26387995839,12cyclictest31719-21kworker/u8:022:04:351
26387995835,13cyclictest0-21swapper/100:34:351
2638799582,36cyclictest11939-21modprobe22:29:341
2638799582,36cyclictest11939-21modprobe22:29:341
26387995812,9cyclictest241ktimers/123:44:361
26387995812,22cyclictest241ktimers/121:24:571
26387995812,20cyclictest241ktimers/121:42:181
26387995811,37cyclictest241ktimers/121:14:581
26387995811,23cyclictest241ktimers/122:09:551
26400995733,16cyclictest27456-21chrt00:02:313
26400995729,18cyclictest394-21runrttasks00:19:403
2639399577,12cyclictest2523-21fschecks_count22:09:572
2639399574,18cyclictest394-21runrttasks20:41:492
26393995740,9cyclictest0-21swapper/220:14:342
26393995739,12cyclictest17493-21latency_hist21:34:372
2639399573,13cyclictest2149-21fschecks_time21:05:012
26393995730,20cyclictest9178-21latency_hist19:49:372
26387995740,11cyclictest597-21unixbench_singl22:05:061
26387995739,12cyclictest6026-21unixbench_multi22:15:111
26387995739,12cyclictest6026-21unixbench_multi22:15:111
26387995715,19cyclictest141rcu_preempt00:15:091
26387995710,10cyclictest241ktimers/121:49:571
26393995640,10cyclictest2706-21idleruntime-cro23:14:342
26393995638,12cyclictest26506-21latency_hist23:59:372
2638799568,29cyclictest241ktimers/123:55:051
26387995637,13cyclictest20238-21latency_hist21:39:381
26387995637,12cyclictest0-21swapper/121:34:341
26387995628,16cyclictest14693-21latency_hist22:34:371
26387995616,31cyclictest231rcuc/120:34:561
26387995614,13cyclictest26788-21ssh22:56:181
26387995614,13cyclictest26788-21ssh22:56:181
26387995611,21cyclictest241ktimers/121:20:131
26387995611,21cyclictest241ktimers/120:04:531
26387995610,23cyclictest241ktimers/121:54:471
26393995538,11cyclictest29482-21sshd00:05:092
26393995538,11cyclictest29482-21sshd00:05:092
26393995535,10cyclictest0-21swapper/223:24:342
26393995522,9cyclictest0-21swapper/222:25:042
26393995522,9cyclictest0-21swapper/222:25:042
26387995533,15cyclictest366-21dbus-daemon19:29:361
26387995525,6cyclictest0-21swapper/100:21:431
26387995511,22cyclictest241ktimers/121:48:501
26387995511,11cyclictest231rcuc/119:19:131
26387995511,11cyclictest231rcuc/119:19:131
26393995434,13cyclictest29078-21munin-plugin-st20:49:342
2639399542,35cyclictest22350-21apt-get20:29:482
26393995415,9cyclictest94052latency_hist22:24:372
2638799549,10cyclictest241ktimers/123:24:541
2638799546,9cyclictest241ktimers/119:07:141
2638799546,9cyclictest241ktimers/119:07:141
2638799544,22cyclictest0-21swapper/121:00:111
26387995441,8cyclictest231rcuc/119:51:131
26387995438,8cyclictest231rcuc/123:20:291
26387995435,13cyclictest28144-21systemctl19:09:391
26387995425,18cyclictest0-21swapper/120:54:341
26387995412,18cyclictest0-21swapper/120:21:411
26387995410,21cyclictest241ktimers/119:20:111
26393995335,11cyclictest13089-21apt-get23:34:342
26393995333,9cyclictest0-21swapper/220:25:282
26393995333,9cyclictest0-21swapper/220:25:282
26393995329,15cyclictest179-21pr/ttyS020:59:572
26387995335,9cyclictest231rcuc/123:01:541
26387995311,21cyclictest241ktimers/119:59:351
26387995311,21cyclictest241ktimers/119:59:351
26387995311,20cyclictest241ktimers/123:45:051
26393995238,7cyclictest0-21swapper/221:24:572
26393995237,7cyclictest0-21swapper/219:05:092
26393995237,7cyclictest0-21swapper/219:05:092
26393995236,8cyclictest0-21swapper/220:09:352
26393995236,10cyclictest1-21systemd21:21:102
26393995233,13cyclictest101272sleep219:50:132
26393995232,11cyclictest21990-21unixbench_multi23:50:082
26393995224,19cyclictest22097-21ssh21:44:442
2639399522,19cyclictest1514-21diskmemload22:00:462
2638799528,22cyclictest241ktimers/119:44:501
26387995212,11cyclictest241ktimers/100:10:051
26387995212,11cyclictest241ktimers/100:10:051
26393995131,13cyclictest0-21swapper/219:29:382
2638799519,19cyclictest241ktimers/100:01:311
26387995124,6cyclictest0-21swapper/122:19:461
26387995123,17cyclictest0-21swapper/120:40:091
26387995111,23cyclictest241ktimers/120:00:031
26387995111,22cyclictest241ktimers/120:15:051
26387995111,20cyclictest241ktimers/120:54:561
26393995035,9cyclictest394-21runrttasks21:14:562
26393995034,8cyclictest0-21swapper/223:44:342
26393995034,8cyclictest0-21swapper/221:59:342
2639399503,25cyclictest3086-21diskstats19:29:572
2639399503,25cyclictest3086-21diskstats19:29:572
26393995031,12cyclictest275-21systemd-journal20:54:352
26393995030,9cyclictest0-21swapper/223:04:342
26393995030,9cyclictest0-21swapper/200:24:552
26393995028,9cyclictest0-21swapper/220:19:562
2639399497,8cyclictest141rcu_preempt20:39:342
26393994911,9cyclictest0-21swapper/219:15:412
26393994911,9cyclictest0-21swapper/219:15:412
26393994831,11cyclictest627-21in:imklog19:54:512
26393994831,11cyclictest627-21in:imklog19:54:512
26393994828,10cyclictest0-21swapper/223:15:182
26393994814,8cyclictest1514-21diskmemload23:44:542
2639399474,8cyclictest141rcu_preempt22:44:352
2717824613,6sleep20-21swapper/200:01:382
2639399468,12cyclictest394-21runrttasks19:21:512
2639399466,10cyclictest321ktimers/221:40:082
26393994631,8cyclictest0-21swapper/222:05:042
26393994631,8cyclictest0-21swapper/200:29:562
26393994628,12cyclictest627-21in:imklog00:17:062
2639399447,9cyclictest394-21runrttasks20:58:132
26393994426,10cyclictest7833-21apt-get23:24:462
26393994421,11cyclictest0-21swapper/200:24:362
26393994415,8cyclictest1514-21diskmemload21:49:492
26393994326,8cyclictest0-21swapper/221:35:002
26393994217,18cyclictest20810-21tail22:45:072
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional