You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7.osadl.org (updated Thu Apr 03, 2025 00:43:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
125974199451,6cyclictest1413057-21/usr/sbin/munin23:36:040
125974299421,6cyclictest1412506-21ssh23:36:001
125974199425,22cyclictest0-21swapper/023:21:030
125974199411,27cyclictest1385794-21sed23:01:030
1259742993923,11cyclictest1458482-21/usr/sbin/munin00:41:051
125974199390,6cyclictest259-21jbd2/mmcblk0p2-823:22:390
1259741993622,7cyclictest1-21systemd22:42:350
125974399355,23cyclictest1458481-21ssh00:41:052
125974199343,7cyclictest171rcu_preempt00:16:020
125974299337,4cyclictest171rcu_preempt21:17:381
125974399321,19cyclictest0-21swapper/222:16:012
1259741993118,5cyclictest1361251-21cut22:30:520
125974199311,20cyclictest0-21swapper/023:03:240
125974199291,21cyclictest0-21swapper/021:20:360
1259743992811,14cyclictest1401757-21sh23:21:342
125974199282,25cyclictest0-21swapper/022:53:220
1259743992720,5cyclictest1402897-21diskmemload23:22:392
1259741992720,6cyclictest1313925-21fschecks_time21:30:550
1259741992717,2cyclictest0-21swapper/021:58:520
125974199271,2cyclictest0-21swapper/022:22:350
125974199271,21cyclictest0-21swapper/021:33:140
125974199271,18cyclictest1458285-21ssh00:41:030
125974199270,21cyclictest0-21swapper/023:29:230
1259743992610,5cyclictest11050irq/107-eth%d23:49:592
125974299263,3cyclictest0-21swapper/100:16:021
125974199260,20cyclictest0-21swapper/021:39:120
112699260,2rtkit-daemon1125-21rtkit-daemon23:40:040
125974399247,13cyclictest1006-21mta-sts-daemon22:34:362
125974399245,9cyclictest11050irq/107-eth%d22:30:552
1259743992417,4cyclictest1288626-21grep20:36:042
1259741992412,2cyclictest0-21swapper/023:50:220
125974399231,4cyclictest0-21swapper/219:41:002
125974199230,21cyclictest0-21swapper/021:42:390
112699230,3rtkit-daemon1125-21rtkit-daemon23:58:142
125974299222,4cyclictest1259740-21cyclictest19:12:351
125974199221,20cyclictest0-21swapper/022:10:540
1259741992211,1cyclictest0-21swapper/022:44:390
125974399219,7cyclictest1318249-21sed21:35:572
125974399212,6cyclictest1322036-21/usr/sbin/munin21:40:552
125974399212,13cyclictest1259700-21munin-run19:12:352
1259743992118,2cyclictest1290964-21grep20:45:572
125974399211,10cyclictest0-21swapper/220:31:032
1259742992114,6cyclictest1269936-21idleruntime-cro19:42:341
1259741992114,5cyclictest1279076-21fschecks_time20:10:580
125974199211,19cyclictest0-21swapper/000:22:350
112699210,1rtkit-daemon0-21swapper/122:52:351
125974499205,6cyclictest0-21swapper/322:41:033
1259743992014,4cyclictest1387317-21grep23:02:352
1259743992010,9cyclictest1329460-21grep21:50:512
1259742992016,3cyclictest1273240-21unin-run19:52:351
125974299200,1cyclictest1402896-21apt-get23:22:381
1259741992017,2cyclictest1433198-21latency_hist00:02:350
1259741992014,4cyclictest0-21swapper/000:10:580
125974399193,12cyclictest1451059-21if_wlan000:30:532
125974399193,12cyclictest1451059-21if_wlan000:30:532
1259743991914,3cyclictest1281664-21grep20:16:042
125974399191,12cyclictest1272176-21fschecks_time19:50:582
125974299194,5cyclictest171rcu_preempt21:12:351
125974299192,14cyclictest1300208-21diskmemload23:47:501
125974299192,14cyclictest1005-21mta-sts-daemon21:06:011
125974199194,3cyclictest171rcu_preempt23:15:570
125974199194,12cyclictest764-21rs:main0
125974199192,16cyclictest0-21swapper/000:30:580
125974199192,16cyclictest0-21swapper/000:30:580
1259741991915,3cyclictest1271584-21munin-plugin-st19:47:350
1259741991911,2cyclictest1377083-21cut22:50:530
125974199190,10cyclictest0-21swapper/023:55:590
125974499185,7cyclictest0-21swapper/323:36:003
125974399183,9cyclictest1264071-21ntpq19:26:002
1259743991810,6cyclictest111788-21systemd-journal19:32:342
125974399180,6cyclictest11050irq/107-eth%d23:07:352
125974299186,4cyclictest11050irq/107-eth%d22:10:581
125974299185,9cyclictest0-21swapper/122:50:571
125974299182,11cyclictest11050irq/107-eth%d19:31:021
1259742991813,3cyclictest1303861-21grep21:16:041
1259742991811,5cyclictest11050irq/107-eth%d22:30:501
125974199182,8cyclictest11050irq/107-eth%d21:16:010
125974199181,2cyclictest1295408-21latency_hist20:57:350
1259741991811,5cyclictest1365453-21sh22:35:550
1259741991811,1cyclictest0-21swapper/022:22:440
112699180,0rtkit-daemon0-21swapper/022:07:140
125974499177,6cyclictest0-21swapper/300:41:053
125974499173,7cyclictest0-21swapper/323:49:593
125974399179,6cyclictest1353705-21/usr/sbin/munin22:20:572
125974399176,2cyclictest11050irq/107-eth%d00:03:402
125974399175,8cyclictest1396877-21grep23:15:592
125974399175,6cyclictest1437022-21ssh00:09:592
125974399175,10cyclictest1342922-21cat22:07:342
125974399173,9cyclictest1413010-21/usr/sbin/munin23:36:042
125974399173,8cyclictest1429104-21latency_hist23:57:352
125974399173,11cyclictest1380671-21/usr/sbin/munin22:56:032
125974399172,3cyclictest1377306-21ntpq22:50:552
1259743991713,2cyclictest1274855-21apt-get19:57:372
1259743991713,2cyclictest1274855-21apt-get19:57:372
1259743991711,5cyclictest1416295-21gltestperf23:40:572
1259743991711,5cyclictest1302785-21fschecks_count21:15:552
1259743991711,3cyclictest36-21ksoftirqd/222:42:352
125974399171,12cyclictest1334450-21grep21:56:012
1259743991710,4cyclictest1345210-21cut22:10:512
125974299174,8cyclictest1384983-21aten_r2power_en23:00:571
125974299173,11cyclictest1409799-21cat23:32:351
125974299173,10cyclictest1456130-21ssh00:37:321
125974299171,11cyclictest1371760-21sendmail-msp22:42:351
1259742991710,5cyclictest0-21swapper/121:00:591
125974299170,3cyclictest1365649-21gltestperf22:35:561
125974199179,3cyclictest0-21swapper/020:35:530
125974199178,2cyclictest0-21swapper/021:53:390
125974199172,8cyclictest0-21swapper/019:15:560
1259741991716,0cyclictest0-21swapper/000:05:520
1259741991713,1cyclictest0-21swapper/021:26:520
125974199171,14cyclictest1454176-21aten_r2power_fr00:35:550
1259741991711,1cyclictest0-21swapper/022:16:040
125974399167,6cyclictest1278512-21munin-run20:07:342
125974399166,8cyclictest1446347-21grep00:21:052
125974399166,7cyclictest351ktimers/219:57:352
125974399164,9cyclictest1408016-21ssh23:30:572
125974399162,2cyclictest1284583-21ntpq20:26:022
125974399161,3cyclictest1314274-21/usr/sbin/munin21:30:572
1259743991612,3cyclictest0-21swapper/222:27:352
125974399161,11cyclictest1282103-21aten_r2power_fr20:20:542
125974399160,13cyclictest1307304-21iwlist21:20:582
125974299169,3cyclictest171rcu_preempt21:50:551
125974299166,3cyclictest171rcu_preempt22:21:001
125974299164,7cyclictest1324400-21ssh21:42:351
125974299163,9cyclictest1444974-21aten_r2power_cu00:20:541
125974299163,8cyclictest1279050-21fschecks_count20:10:581
125974299162,2cyclictest171rcu_preempt23:55:541
125974299162,2cyclictest171rcu_preempt23:43:001
125974299162,2cyclictest11050irq/107-eth%d21:55:521
1259742991613,3cyclictest1431621-21ntp_kernel_err00:01:011
1259742991613,2cyclictest0-21swapper/121:35:561
125974299160,3cyclictest171rcu_preempt20:27:351
125974299160,13cyclictest1400849-21ssh23:20:591
125974199166,6cyclictest0-21swapper/019:12:350
125974199162,1cyclictest0-21swapper/023:10:560
125974199162,11cyclictest1268888-21fschecks_time19:40:550
125974199161,2cyclictest0-21swapper/020:25:540
1259741991611,4cyclictest1277452-21fschecks_time20:05:570
1259741991611,4cyclictest1262960-21unixbench_singl19:21:040
125974199161,12cyclictest1274831-21/usr/sbin/munin19:56:020
125974199160,7cyclictest171rcu_preempt19:32:340
125974499153,7cyclictest0-21swapper/323:00:573
125974399157,5cyclictest1293633-21apt20:52:342
125974399156,7cyclictest1298229-21/usr/sbin/munin21:06:042
125974399153,8cyclictest1441871-21/usr/sbin/munin00:15:582
125974399153,8cyclictest1339111-21sed22:01:082
125974399153,11cyclictest1294169-21grep20:55:562
125974399151,10cyclictest1454499-21tune2fs00:35:582
1259743991510,4cyclictest1300314-21latency_hist21:12:352
125974299159,5cyclictest633-21gpm21:30:501
125974299155,3cyclictest171rcu_preempt20:55:561
125974299154,6cyclictest1311745-21latency_hist21:27:351
125974299152,12cyclictest1263010-21munin-run19:22:351
125974299151,3cyclictest171rcu_preempt23:10:561
125974299150,10cyclictest1435614-21idleruntime-cro00:07:351
125974299150,10cyclictest1273738-21fschecks_count19:55:531
125974499143,5cyclictest0-21swapper/321:12:353
125974499143,4cyclictest0-21swapper/321:20:563
125974499141,8cyclictest0-21swapper/300:01:013
125974499140,7cyclictest0-21swapper/323:20:573
125974399148,5cyclictest1260407-21gltestperf19:15:592
125974399146,5cyclictest0-21swapper/219:46:012
125974399143,1cyclictest171rcu_preempt21:01:002
125974399142,2cyclictest0-21swapper/220:11:022
1259743991412,1cyclictest0-21swapper/219:20:542
125974399140,11cyclictest1327182-21cat21:47:342
125974299149,4cyclictest1437777-21cut00:10:521
125974299145,6cyclictest1267422-21/usr/sbin/munin19:35:581
125974299143,10cyclictest1290877-21/usr/sbin/munin20:45:571
125974299142,8cyclictest1415812-21ssh23:40:541
125974299142,8cyclictest1350378-21/usr/sbin/munin22:16:041
125974299142,8cyclictest1263772-21if_end019:25:571
125974299142,10cyclictest1285657-21apt-get20:27:361
125974299141,9cyclictest1450477-21/usr/sbin/munin00:30:491
125974299141,9cyclictest1450477-21/usr/sbin/munin00:30:491
125974299141,10cyclictest1337858-21aten_r2power_fr22:00:581
125974299140,3cyclictest171rcu_preempt20:22:351
125974199141,8cyclictest1331480-21latency_hist21:52:350
125974199141,8cyclictest1268262-21/usr/sbin/munin19:36:050
125974199141,2cyclictest0-21swapper/020:15:590
1259741991412,1cyclictest0-21swapper/019:27:350
1259741991410,3cyclictest1291931-21unixbench_singl20:46:050
125974199140,9cyclictest1-21systemd00:27:340
125974499134,6cyclictest0-21swapper/300:30:553
125974499134,6cyclictest0-21swapper/300:30:553
125974499133,6cyclictest0-21swapper/322:30:503
125974499133,5cyclictest0-21swapper/323:22:393
125974499133,5cyclictest0-21swapper/322:49:123
125974499132,7cyclictest0-21swapper/323:10:553
125974499132,6cyclictest0-21swapper/319:16:043
125974499132,6cyclictest0-21swapper/300:10:313
125974499132,4cyclictest0-21swapper/300:33:143
125974499131,8cyclictest0-21swapper/322:15:563
125974399139,3cyclictest1289363-21grep20:40:572
125974399132,9cyclictest1267250-21tune2fs19:35:572
125974399131,9cyclictest111788-21systemd-journal23:42:512
1259743991310,2cyclictest11050irq/107-eth%d23:03:122
125974299138,2cyclictest0-21swapper/120:41:021
125974299132,8cyclictest1280677-21tune2fs20:15:561
125974299131,8cyclictest1397112-21cat23:16:011
125974199131,9cyclictest1275480-21aten2.4-expect20:00:520
125974199131,9cyclictest1275480-21aten2.4-expect20:00:520
125974199131,8cyclictest1292814-21/usr/sbin/munin20:51:010
1259741991311,1cyclictest0-21swapper/021:11:050
125974199130,9cyclictest1297342-21awk21:05:550
125974499127,3cyclictest0-21swapper/323:04:483
125974499124,4cyclictest0-21swapper/319:46:013
125974499123,6cyclictest0-21swapper/323:15:573
125974499123,6cyclictest0-21swapper/322:10:533
125974499123,6cyclictest0-21swapper/321:35:573
125974499123,5cyclictest0-21swapper/323:31:053
125974499123,5cyclictest0-21swapper/321:50:493
125974499123,5cyclictest0-21swapper/319:27:373
125974499123,3cyclictest0-21swapper/319:25:563
125974499122,6cyclictest0-21swapper/323:41:013
125974499122,5cyclictest0-21swapper/322:21:033
125974499122,5cyclictest0-21swapper/321:55:533
125974499122,5cyclictest0-21swapper/320:06:033
125974399127,3cyclictest1311702-21cron21:27:342
125974299128,4cyclictest0-21swapper/122:42:401
125974299126,4cyclictest111788-21systemd-journal22:02:361
125974299125,4cyclictest1288254-21/usr/sbin/munin20:36:021
125974299123,8cyclictest1270515-21fschecks_time19:45:561
125974299122,7cyclictest1260900-21/usr/sbin/munin19:16:041
125974299122,6cyclictest1276934-21cat20:02:351
125974299122,6cyclictest1276934-21cat20:02:351
125974299121,7cyclictest1277492-21tune2fs20:05:571
125974299120,4cyclictest171rcu_preempt20:50:591
125974199124,5cyclictest1282019-21/usr/sbin/munin20:20:570
125974199122,5cyclictest1289303-21cut20:40:570
125974199121,8cyclictest1295992-21/usr/sbin/munin21:00:560
125974199121,10cyclictest1272205-21fschecks_time19:50:590
125974199121,10cyclictest1008-21mta-sts-daemon20:27:360
125974499113,7cyclictest0-21swapper/322:54:033
125974499112,7cyclictest0-21swapper/323:56:023
125974499112,6cyclictest0-21swapper/321:30:533
125974499112,4cyclictest0-21swapper/319:57:373
125974499112,4cyclictest0-21swapper/319:57:373
125974499111,6cyclictest0-21swapper/322:35:573
125974499111,5cyclictest0-21swapper/322:00:583
125974499111,5cyclictest0-21swapper/320:36:033
125974499111,5cyclictest0-21swapper/320:21:023
125974499111,5cyclictest0-21swapper/300:15:593
125974399119,1cyclictest11050irq/107-eth%d22:44:572
125974399112,7cyclictest111788-21systemd-journal00:27:342
125974299118,2cyclictest1390043-21latency_hist23:07:351
125974299117,3cyclictest1324558-21sh21:42:381
125974299112,4cyclictest1448950-21munin-run00:27:341
125974299110,2cyclictest171rcu_preempt22:24:031
125974499103,3cyclictest0-21swapper/321:15:533
125974499103,3cyclictest0-21swapper/320:10:583
125974499102,5cyclictest0-21swapper/321:40:543
125974499102,4cyclictest0-21swapper/320:40:573
125974499102,4cyclictest0-21swapper/300:19:163
125974499101,5cyclictest0-21swapper/321:00:553
125974499101,5cyclictest0-21swapper/320:45:583
125974499101,5cyclictest0-21swapper/320:26:003
12597449996,2cyclictest0-21swapper/321:22:493
12597449995,2cyclictest0-21swapper/322:43:123
12597449993,5cyclictest0-21swapper/319:55:533
12597449993,4cyclictest0-21swapper/319:21:013
12597449993,3cyclictest0-21swapper/320:51:073
12597449992,4cyclictest0-21swapper/322:06:443
12597449992,3cyclictest0-21swapper/319:36:033
12597449991,5cyclictest0-21swapper/322:25:253
12597449991,5cyclictest0-21swapper/320:27:363
12597449991,4cyclictest0-21swapper/323:42:543
12597449991,4cyclictest0-21swapper/320:55:553
12597449991,4cyclictest0-21swapper/319:50:573
12597449983,4cyclictest0-21swapper/319:40:573
12597449983,3cyclictest0-21swapper/300:27:343
12597449982,4cyclictest0-21swapper/321:43:293
12597449981,5cyclictest0-21swapper/320:15:563
12597449981,3cyclictest0-21swapper/321:05:543
12597449973,3cyclictest0-21swapper/300:02:503
12597449952,2cyclictest0-21swapper/319:12:353
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional