You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-15 - 09:08
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot6.osadl.org (updated Wed Jan 15, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1539821680,4sleep02187999cyclictest23:47:040
3082521660,4sleep02187999cyclictest00:47:020
2967421660,4sleep32189899cyclictest00:46:533
1675221660,2sleep12188599cyclictest23:07:591
428621610,4sleep02187999cyclictest01:33:320
428621610,4sleep02187999cyclictest01:33:320
1518911010,4ptp4l401ktimersoftd/320:22:363
151891680,1ptp4l401ktimersoftd/301:13:193
21879996010,18cyclictest18542-21load01:52:090
218042600,3sleep12188599cyclictest23:14:211
2187999599,17cyclictest15882-21/usr/sbin/munin22:27:030
21879995911,13cyclictest2603-21latency_hist22:51:430
2187999589,17cyclictest12055-21chrt00:24:340
2187999578,17cyclictest24377-21munin-node20:57:130
217182574,40sleep00-21swapper/020:25:550
151891570,1ptp4l401ktimersoftd/322:09:173
151891570,1ptp4l401ktimersoftd/301:30:493
2187999568,16cyclictest8231-21/usr/sbin/munin23:37:050
2187999568,10cyclictest7869-21snmp_o5powerctl00:17:100
151891560,1ptp4l401ktimersoftd/323:18:303
151891560,1ptp4l401ktimersoftd/323:03:583
2187999556,17cyclictest0-21swapper/001:27:040
21879995510,9cyclictest18789-21cron21:26:410
151891550,1ptp4l401ktimersoftd/322:30:183
151891550,1ptp4l401ktimersoftd/301:33:583
151891550,1ptp4l401ktimersoftd/301:33:583
151891550,1ptp4l401ktimersoftd/301:04:543
2187999549,14cyclictest28625-21ssh22:42:340
2187999549,13cyclictest16037-21ssh01:09:000
2187999548,15cyclictest5513-21ssh23:34:380
2187999548,14cyclictest9515-21ssh01:40:100
2187999548,14cyclictest28615-21apt-get21:47:010
2187999548,14cyclictest20544-21ssh23:52:430
2187999548,14cyclictest20544-21ssh23:52:430
2187999548,14cyclictest18108-21ssh23:10:460
2187999548,14cyclictest11775-21ssh01:44:440
21879995411,11cyclictest15579-21taskset01:48:150
2187999539,12cyclictest12045-21ssh01:04:540
2187999538,9cyclictest22968-21apt-get23:56:570
2187999538,13cyclictest11536-21ssh23:42:280
2187999537,14cyclictest26183-21ssh22:37:560
2187999537,11cyclictest26364-21apt-get01:21:500
21879995311,18cyclictest9011-21ssh00:59:130
2187999528,18cyclictest30039-21apt-get00:06:520
2187999528,14cyclictest25512-21ssh00:39:340
2187999528,14cyclictest20374-21ssh00:33:180
2187999528,13cyclictest19670-21ssh01:12:250
2187999527,14cyclictest13771-21ssh23:05:550
21879995210,14cyclictest209132grep22:32:030
2187999518,9cyclictest6051-21munin-run00:16:450
2187999518,8cyclictest9398-21apt-get21:06:540
2187999518,8cyclictest25172-21apt-get20:31:530
2187999518,18cyclictest6932-21ssh22:56:400
2187999518,14cyclictest28485-21ssh00:45:070
2187999518,14cyclictest27028-21ssh23:20:360
2187999517,17cyclictest14365-21apt-get21:16:540
2187999508,9cyclictest22717-21gltestperf01:17:020
2187999508,9cyclictest10682-21df_inode22:16:570
2187999508,10cyclictest30613-21cpu23:26:520
2187999507,13cyclictest11200-21ssh23:01:280
2187999499,15cyclictest22591-21sh23:15:580
2187999497,13cyclictest4375-21ssh00:53:530
2187999489,8cyclictest2629-21perl20:51:580
2187999488,9cyclictest5674-21apt-get22:06:560
2187999488,9cyclictest14926-21snmp_o5powerctl00:27:090
2187999478,9cyclictest26720-21hddtemp_smartct21:41:570
2187999478,9cyclictest22913-21ssh21:32:100
2187999478,18cyclictest14860-21diskmemload00:05:250
21889994631,8cyclictest33-21ksoftirqd/222:31:532
2187999468,9cyclictest31540-21if_ens321:51:590
2187999467,9cyclictest8379-21snmp_o5powerctl21:02:070
2187999467,17cyclictest12636-21sed21:12:010
2187999457,8cyclictest32554-21df_inode20:46:580
2187999447,8cyclictest19590-21hddtemp_smartct21:27:020
2187999447,13cyclictest28210-21ssh23:22:360
151891440,1ptp4l401ktimersoftd/321:11:433
21889994333,5cyclictest33-21ksoftirqd/201:11:582
151891420,17ptp4l401ktimersoftd/321:56:553
2187999417,9cyclictest13043-21df_inode22:21:550
151891400,2ptp4l32602-21kworker/3:123:42:093
151891400,1ptp4l401ktimersoftd/300:16:323
21889993928,5cyclictest33-21ksoftirqd/223:46:512
21889993926,6cyclictest33-21ksoftirqd/223:31:562
21889993831,4cyclictest33-21ksoftirqd/201:21:502
2187999387,16cyclictest2997-21latency_hist22:01:410
151891380,1ptp4l401ktimersoftd/322:54:143
151891380,1ptp4l401ktimersoftd/320:26:553
151891380,1ptp4l401ktimersoftd/300:29:493
21889993727,8cyclictest33-21ksoftirqd/222:22:092
151891370,2ptp4l32602-21kworker/3:123:59:563
151891370,1ptp4l401ktimersoftd/323:39:233
151891370,1ptp4l401ktimersoftd/323:08:173
151891370,1ptp4l401ktimersoftd/322:58:413
151891370,1ptp4l401ktimersoftd/322:35:263
151891370,1ptp4l401ktimersoftd/300:25:543
151891370,1ptp4l32602-21kworker/3:123:33:393
60512360,4sleep12188599cyclictest22:07:001
21889993631,3cyclictest33-21ksoftirqd/200:16:442
21889993630,3cyclictest33-21ksoftirqd/223:16:442
2187999366,8cyclictest9387-21snmp_o5powerctl22:12:110
2187999366,8cyclictest9387-21snmp_o5powerctl22:12:110
151891360,1ptp4l401ktimersoftd/323:56:443
151891360,1ptp4l401ktimersoftd/323:56:443
151891360,1ptp4l401ktimersoftd/323:12:513
151891360,1ptp4l401ktimersoftd/301:39:013
2187999357,9cyclictest3563-21diskstats22:01:560
151891350,1ptp4l401ktimersoftd/301:43:503
151891350,1ptp4l401ktimersoftd/300:57:193
2187999347,9cyclictest22618-21hddtemp_smartct20:26:550
151891340,1ptp4l401ktimersoftd/300:11:193
151891340,0ptp4l401ktimersoftd/300:45:493
21889993327,3cyclictest33-21ksoftirqd/221:36:412
210782336,11sleep10-21swapper/120:22:131
151891330,1ptp4l401ktimersoftd/322:49:263
151891330,1ptp4l401ktimersoftd/322:44:273
151891330,1ptp4l401ktimersoftd/322:02:003
151891330,1ptp4l401ktimersoftd/321:02:043
151891330,1ptp4l401ktimersoftd/301:08:403
151891330,1ptp4l401ktimersoftd/300:40:323
21889993226,4cyclictest33-21ksoftirqd/222:57:012
21889993226,3cyclictest33-21ksoftirqd/220:46:572
2187999329,13cyclictest27952-21fw_packets20:36:550
151891320,1ptp4l401ktimersoftd/323:27:553
151891320,1ptp4l401ktimersoftd/321:48:553
151891320,1ptp4l401ktimersoftd/321:17:013
151891320,1ptp4l401ktimersoftd/320:57:483
151891320,1ptp4l401ktimersoftd/301:55:193
151891320,1ptp4l401ktimersoftd/301:22:123
151891320,1ptp4l401ktimersoftd/300:52:373
151891320,1ptp4l401ktimersoftd/300:34:523
151891320,1ptp4l401ktimersoftd/300:20:353
151891320,10ptp4l401ktimersoftd/322:15:053
151891320,10ptp4l401ktimersoftd/322:15:053
151891320,0ptp4l401ktimersoftd/323:49:223
151891320,0ptp4l401ktimersoftd/301:21:343
21889993119,6cyclictest33-21ksoftirqd/222:46:512
199752317,11sleep20-21swapper/220:21:532
151891310,1ptp4l401ktimersoftd/321:24:103
151891310,10ptp4l401ktimersoftd/301:49:133
151891310,0ptp4l401ktimersoftd/322:40:483
21889993025,3cyclictest33-21ksoftirqd/201:07:002
21889993024,3cyclictest33-21ksoftirqd/223:57:032
21889993024,3cyclictest33-21ksoftirqd/221:16:412
21889993024,3cyclictest33-21ksoftirqd/220:36:392
151891300,2ptp4l401ktimersoftd/321:36:533
151891300,0ptp4l401ktimersoftd/321:30:363
21889992924,3cyclictest33-21ksoftirqd/201:27:002
21889992923,4cyclictest33-21ksoftirqd/222:56:382
21889992916,6cyclictest33-21ksoftirqd/222:26:572
21889992915,6cyclictest33-21ksoftirqd/222:36:542
21879992914,10cyclictest0-21swapper/021:36:540
151891290,0ptp4l401ktimersoftd/323:22:073
151891290,0ptp4l401ktimersoftd/321:56:363
21889992823,3cyclictest33-21ksoftirqd/201:51:472
21889992823,3cyclictest33-21ksoftirqd/201:06:452
21889992822,3cyclictest33-21ksoftirqd/201:21:462
21889992821,4cyclictest26539-1kworker/u9:000:51:582
21885992821,4cyclictest25-21ksoftirqd/122:25:311
1590912821,3phc2sys41-21ksoftirqd/320:51:583
151891280,1ptp4l401ktimersoftd/322:24:293
21889992721,3cyclictest33-21ksoftirqd/221:17:002
21889992721,3cyclictest33-21ksoftirqd/220:56:402
21889992721,3cyclictest33-21ksoftirqd/201:32:012
21889992721,3cyclictest33-21ksoftirqd/201:32:012
21889992720,4cyclictest33-21ksoftirqd/221:29:032
2188599272,3cyclictest121rcu_preempt00:55:581
181602270,3sleep22188999cyclictest21:22:092
965522619,3sleep233-21ksoftirqd/200:20:032
21889992621,3cyclictest33-21ksoftirqd/221:37:072
21889992621,3cyclictest33-21ksoftirqd/200:46:532
21889992621,3cyclictest33-21ksoftirqd/200:27:092
21889992621,3cyclictest33-21ksoftirqd/200:01:452
21889992620,4cyclictest7964-1kworker/u9:000:57:582
21889992620,4cyclictest33-21ksoftirqd/201:52:002
21889992620,3cyclictest33-21ksoftirqd/222:12:002
21889992620,3cyclictest33-21ksoftirqd/222:12:002
21889992620,3cyclictest33-21ksoftirqd/200:41:382
21889992620,3cyclictest33-21ksoftirqd/200:07:052
21889992619,4cyclictest33-21ksoftirqd/222:20:142
21889992619,4cyclictest33-21ksoftirqd/201:44:502
21885992621,3cyclictest25-21ksoftirqd/121:06:211
21885992621,3cyclictest25-21ksoftirqd/100:06:521
21885992620,3cyclictest25-21ksoftirqd/122:17:441
21885992620,3cyclictest25-21ksoftirqd/122:02:481
21885992620,3cyclictest25-21ksoftirqd/121:19:301
21885992620,3cyclictest25-21ksoftirqd/120:45:371
21885992620,3cyclictest25-21ksoftirqd/101:04:231
21885992619,4cyclictest29616-21cron20:41:391
21885992619,4cyclictest25-21ksoftirqd/121:16:341
21885992617,6cyclictest2601-21sh22:51:431
151891260,1ptp4l401ktimersoftd/321:33:293
151891260,1ptp4l401ktimersoftd/321:10:363
151891260,1ptp4l401ktimersoftd/320:43:383
21898992519,3cyclictest41-21ksoftirqd/320:50:523
21898992518,4cyclictest41-21ksoftirqd/320:34:163
21889992520,3cyclictest33-21ksoftirqd/223:27:412
21889992520,3cyclictest33-21ksoftirqd/222:45:352
21889992520,3cyclictest33-21ksoftirqd/222:07:012
21889992520,3cyclictest33-21ksoftirqd/220:31:392
21889992519,3cyclictest33-21ksoftirqd/221:59:332
21889992519,3cyclictest33-21ksoftirqd/221:51:552
2188999251,21cyclictest29957-21snmp_o5powerctl21:47:152
2188999250,2cyclictest121rcu_preempt23:36:582
21885992519,3cyclictest25-21ksoftirqd/123:29:341
21885992519,3cyclictest25-21ksoftirqd/122:32:211
21885992519,3cyclictest25-21ksoftirqd/120:53:121
21885992519,3cyclictest25-21ksoftirqd/101:17:121
21885992518,4cyclictest25-21ksoftirqd/121:58:451
21885992518,4cyclictest25-21ksoftirqd/120:48:441
21885992517,5cyclictest23657-21irqrtprio23:57:031
21879992513,9cyclictest30356-21sshd20:41:540
151891250,1ptp4l401ktimersoftd/320:41:093
68952240,3sleep17152-21kernelversion00:57:041
21898992418,3cyclictest41-21ksoftirqd/322:19:293
21898992418,3cyclictest41-21ksoftirqd/300:04:213
21889992418,3cyclictest33-21ksoftirqd/221:07:442
21889992418,3cyclictest33-21ksoftirqd/220:38:572
21889992417,4cyclictest33-21ksoftirqd/221:46:062
21889992412,5cyclictest33-21ksoftirqd/200:31:572
2188999240,22cyclictest0-21swapper/223:52:322
2188999240,22cyclictest0-21swapper/223:52:322
2188999240,21cyclictest15084-21if_enp1s023:06:562
21885992419,3cyclictest25-21ksoftirqd/122:16:411
21885992419,3cyclictest25-21ksoftirqd/122:16:411
21885992418,3cyclictest25-21ksoftirqd/123:35:041
21885992418,3cyclictest25-21ksoftirqd/121:34:451
21885992418,3cyclictest25-21ksoftirqd/100:12:591
21885992417,4cyclictest25-21ksoftirqd/121:54:481
151891240,1ptp4l401ktimersoftd/321:41:523
21889992317,3cyclictest33-21ksoftirqd/222:06:412
21889992313,8cyclictest33-21ksoftirqd/201:37:092
2188999231,20cyclictest14860-21diskmemload23:44:572
2188999231,19cyclictest31270-21snmp_o5powerctl20:42:042
21885992318,3cyclictest25-21ksoftirqd/120:31:401
21885992318,3cyclictest25-21ksoftirqd/101:15:531
21885992317,3cyclictest25-21ksoftirqd/121:25:281
21885992317,3cyclictest25-21ksoftirqd/101:52:021
21885992317,3cyclictest25-21ksoftirqd/101:34:311
21885992317,3cyclictest25-21ksoftirqd/101:34:311
21885992317,3cyclictest25-21ksoftirqd/100:44:341
21885992316,4cyclictest25-21ksoftirqd/101:37:101
21889992217,3cyclictest5009-1kworker/u9:123:24:312
21889992217,3cyclictest33-21ksoftirqd/220:57:102
21889992216,3cyclictest33-21ksoftirqd/223:04:492
21889992214,5cyclictest23908-21fschecks_count23:16:592
2188999221,19cyclictest26896-21ssh00:41:542
21885992217,3cyclictest25-21ksoftirqd/122:56:581
21885992217,3cyclictest25-21ksoftirqd/120:34:341
21885992216,3cyclictest25-21ksoftirqd/123:51:591
21885992216,3cyclictest25-21ksoftirqd/123:51:591
21885992216,3cyclictest25-21ksoftirqd/123:18:431
21885992116,3cyclictest25-21ksoftirqd/123:03:341
21885992116,3cyclictest25-21ksoftirqd/101:26:551
21885992115,4cyclictest25-21ksoftirqd/101:24:171
21885992115,3cyclictest25-21ksoftirqd/123:42:471
21885992115,3cyclictest25-21ksoftirqd/123:37:011
21885992114,4cyclictest9763-21grep21:06:571
21885992114,4cyclictest25-21ksoftirqd/121:47:251
21885992114,4cyclictest25-21ksoftirqd/121:41:271
21885992114,4cyclictest20500-21sh00:33:321
21885992114,4cyclictest15856-21sh00:28:191
21885992114,4cyclictest13975-21sh23:46:511
21885992114,4cyclictest12201-21sh01:45:401
21885992113,4cyclictest25616-21timerandwakeup22:37:111
21889992014,3cyclictest6956-21/usr/sbin/munin21:01:562
21885992014,4cyclictest25-21ksoftirqd/121:27:101
21885992013,4cyclictest6977-21basename00:17:011
21885992013,4cyclictest29332-21sh23:24:571
21885992013,4cyclictest28938-21sh22:43:121
21885992013,4cyclictest27649-21sh00:02:311
21885992013,4cyclictest25-21ksoftirqd/120:56:431
21885992013,4cyclictest16812-21sh01:51:031
21885992013,4cyclictest15195-21/usr/sbin/munin22:27:131
21885992012,4cyclictest2938-21acpi22:51:541
2188999190,18cyclictest0-21swapper/200:22:372
21885991913,4cyclictest25-21ksoftirqd/121:42:271
21885991913,3cyclictest14374-21missed_timers01:07:011
21885991912,4cyclictest627-21cron00:26:451
21885991912,4cyclictest22903-21fschecks_time00:37:021
21885991912,4cyclictest1505-21runrttasks00:51:411
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional