You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-02-23 - 10:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Feb 23, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1246263993026,3cyclictest0-21swapper/121:55:001
1246259993026,3cyclictest1265915-21sh20:00:010
1246259993021,7cyclictest1355046-21munin-run22:20:010
124625999291,6cyclictest1332958-21cat21:55:010
124625999286,14cyclictest0-21swapper/021:39:020
1246259992824,2cyclictest0-21swapper/021:10:180
124625999281,7cyclictest940-21runrttasks23:45:390
124625999281,6cyclictest1425672-21tune2fs23:40:420
124625999281,6cyclictest1307826-21ssh21:24:310
1246263992718,6cyclictest699-21dbus-daemon00:30:071
124625999275,2cyclictest1294346-21diskmemload00:25:280
124625999270,25cyclictest1367827-21cat22:35:000
1246269992621,3cyclictest262-21systemd-journal19:59:593
1246263992613,8cyclictest262-21systemd-journal22:15:111
1246259992620,4cyclictest0-21swapper/023:10:000
1246259992617,7cyclictest1463291-21idleruntime-cro00:25:010
124625999261,6cyclictest940-21runrttasks21:16:170
124625999261,6cyclictest1387974-21ssh22:57:010
1246259992615,9cyclictest1274671-21gltestperf20:20:150
1246259992615,9cyclictest0-21swapper/020:30:000
124625999261,2cyclictest0-21swapper/022:52:220
124625999261,2cyclictest0-21swapper/022:52:210
124625999260,18cyclictest1325933-21taskset21:45:560
124625999260,17cyclictest1362992-21ssh22:29:520
1246269992522,2cyclictest699-21dbus-daemon22:30:013
1246269992522,2cyclictest1303674-21idleruntime-cro21:20:013
1246263992516,6cyclictest1442859-21systemctl00:00:001
124625999257,11cyclictest1372655-21apt-get22:40:120
124625999251,6cyclictest1410388-21sh23:22:470
124625999251,5cyclictest1442017-21runrttasks23:58:360
1246259992515,8cyclictest1280253-21apt-key20:35:000
1246259992512,11cyclictest0-21swapper/021:10:010
124625999251,1cyclictest0-21swapper/021:55:050
124625999251,13cyclictest1402862-21/usr/sbin/munin23:15:190
124625999251,13cyclictest1358291-21taskset22:22:430
124625999251,13cyclictest0-21swapper/000:16:050
124625999251,12cyclictest0-21swapper/023:10:020
1246259992510,12cyclictest1344271-21ssh22:07:350
124625999250,7cyclictest141252150ssh23:25:510
124625999250,7cyclictest1248856-21cat19:15:150
124626999241,9cyclictest1391057-21ntpq23:00:193
1246267992419,3cyclictest1265901-21cron20:00:002
1246263992420,2cyclictest1342667-21latency_hist22:05:011
124625999248,7cyclictest1288386-21sshd20:54:460
1246259992415,7cyclictest0-21swapper/023:55:000
124625999241,15cyclictest1294346-21diskmemload00:34:360
124625999241,12cyclictest0-21swapper/021:41:500
1246259992410,13cyclictest0-21swapper/021:33:410
124625999240,6cyclictest1341452-21runrttasks22:03:470
124625999240,17cyclictest1392170-21runrttasks23:01:200
124625999240,15cyclictest1294346-21diskmemload23:31:410
1246269992320,2cyclictest1246255-21cyclictest00:00:003
124626999231,8cyclictest1338565-21ntpq22:00:193
124626799233,17cyclictest1335688-21sh21:57:262
1246267992319,3cyclictest1434004-21gltestperf23:50:102
1246267992316,4cyclictest1301497-21ssh21:16:312
124626399232,18cyclictest1378431-21sh22:47:011
124626399231,7cyclictest1347841-21ssh22:11:101
1246263992311,9cyclictest1356710-21wc22:20:261
124625999239,11cyclictest1379190-21taskset22:48:080
124625999238,6cyclictest722-21gdbus19:45:250
124625999238,6cyclictest1262948-21modprobe19:50:320
124625999235,8cyclictest1424871-21latency_hist23:40:000
124625999231,6cyclictest1453135-21sh00:11:390
124625999231,15cyclictest1267468-21runrttasks20:00:230
124625999231,14cyclictest1294346-21diskmemload21:28:560
1246259992311,10cyclictest1346887-21pmu-power22:10:170
124625999230,7cyclictest1272346-21/usr/sbin/munin20:15:140
1246269992219,2cyclictest1246255-21cyclictest00:21:403
1246269992218,2cyclictest703-21polkitd20:59:593
124626999221,18cyclictest1428296-21ssh23:43:023
124626799225,6cyclictest1454795-21sh00:14:482
1246267992217,4cyclictest421ktimers/221:47:402
1246267992211,7cyclictest1408168-21ntpq23:20:202
124626799220,20cyclictest1246229-21kworker/u16:0+rpciod21:26:282
124626399228,1cyclictest0-21swapper/123:23:581
1246263992219,2cyclictest0-21swapper/119:10:231
1246263992217,3cyclictest1325208-21sh21:45:001
1246263992216,4cyclictest1276046-21sh20:25:001
124625999228,6cyclictest1269235-21ntp_states20:05:210
124625999226,6cyclictest1283820-21grep20:40:320
124625999221,8cyclictest1252700-21grep19:25:140
124625999221,13cyclictest1294346-21diskmemload00:07:250
124625999221,10cyclictest1368962-21ntpq22:35:200
124625999220,8cyclictest1256568-21cut19:35:140
124625999220,12cyclictest0-21swapper/000:03:090
124626999214,5cyclictest0-21swapper/322:10:173
1246269992118,2cyclictest0-21swapper/320:40:003
1246269992117,2cyclictest1465659-21sh00:28:343
1246269992116,3cyclictest0-21swapper/321:10:013
1246269992112,6cyclictest1468901-21sh00:31:543
124626999211,17cyclictest1328591-21/usr/sbin/munin21:50:163
1246267992118,2cyclictest0-21swapper/223:04:592
1246267992117,2cyclictest1274058-21cron20:20:012
1246267992117,2cyclictest1250178-21sendmail19:20:002
1246267992113,5cyclictest699-21dbus-daemon22:40:002
1246267992111,7cyclictest942-21snmpd21:37:562
1246267992111,5cyclictest1454151-21kworker/u16:0+rpciod00:22:122
124626799210,18cyclictest262-21systemd-journal23:14:552
124626399211,7cyclictest1437293-21sh23:53:391
124626399211,7cyclictest1400503-21ssh23:11:431
1246263992116,3cyclictest331ktimers/119:15:201
1246263992116,3cyclictest1279041-21ntp_kernel_pll_20:30:221
1246263992112,6cyclictest1448954-21ssh00:06:591
124626399211,17cyclictest1363140-21ssh22:30:011
124626399211,17cyclictest1289600-21/usr/sbin/munin20:55:221
124625999218,3cyclictest1284664-21sh20:45:440
124625999218,2cyclictest0-21swapper/020:55:250
124625999216,7cyclictest1272045-21seq20:14:320
124625999216,5cyclictest1254801-21cat19:30:510
124625999216,2cyclictest0-21swapper/021:00:120
124625999211,8cyclictest1260027-21basename19:45:000
124625999211,8cyclictest1260027-21basename19:45:000
124625999211,8cyclictest1250762-21sort19:21:080
124626999201,6cyclictest1397748-21sh23:09:373
124626999201,6cyclictest1294346-21diskmemload22:06:323
1246269992016,3cyclictest1294843-21cpuspeed_turbos21:10:153
1246269992015,3cyclictest1252716-21gltestperf19:25:143
1246269992015,2cyclictest1415220-21latency_hist23:30:013
1246269992012,6cyclictest1448417-21runrttasks00:06:093
1246269992011,6cyclictest262-21systemd-journal19:15:093
1246269992010,7cyclictest1372565-21seq22:40:003
124626999200,6cyclictest1315467-21sh21:34:223
124626799201,6cyclictest1427252-21sh23:41:242
124626799201,6cyclictest1389695-21ssh22:59:452
124626799201,6cyclictest1294346-21diskmemload22:01:432
1246267992016,2cyclictest1294346-21diskmemload22:21:382
1246267992012,6cyclictest1455217-21/usr/sbin/munin00:15:532
124626799201,17cyclictest1351536-21ntpq22:15:192
124626799201,17cyclictest1281264-21ntpq20:35:222
1246267992011,6cyclictest940-21runrttasks23:16:062
1246267992010,7cyclictest1445499-21seq00:01:392
1246267992010,7cyclictest1307652-21sh21:24:122
1246267992010,7cyclictest1255073-21ldconfig19:30:542
124626799200,6cyclictest1379602-21ssh22:49:002
124626799200,18cyclictest1448539-21taskset00:06:202
1246263992017,2cyclictest1382551-21ssh22:51:071
1246263992017,2cyclictest1382551-21ssh22:51:061
1246263992016,2cyclictest1415217-21sh23:30:001
1246263992015,3cyclictest1252528-21cpuspeed_turbos19:25:131
1246263992014,2cyclictest1424648-21munin-run23:40:001
124626399201,17cyclictest1316446-21diskstats21:35:141
1246263992011,6cyclictest1444013-21ssh00:00:171
124626399201,16cyclictest1294346-21diskmemload23:30:561
124626399201,14cyclictest1374412-21sh22:40:541
1246263992010,7cyclictest2950362-21gdbus23:08:051
1246263992010,7cyclictest1387303-21ssh22:56:031
1246263992010,7cyclictest1334868-21ssh21:56:021
124625999201,7cyclictest1247175-21grep19:10:200
124626999199,6cyclictest262-21systemd-journal20:40:063
124626999198,7cyclictest1438319-21sed23:55:013
1246269991915,2cyclictest1316955-21sh21:35:183
1246269991915,2cyclictest0-21swapper/322:55:203
1246269991914,3cyclictest1284714-21fschecks_count20:45:463
1246269991914,3cyclictest1270522-21cpuspeed_turbos20:10:143
1246269991912,5cyclictest1336293-21ssh21:58:233
1246269991912,5cyclictest1309830-21sh21:27:303
124626999191,16cyclictest1402754-21sh23:15:023
124626999191,16cyclictest1294346-21diskmemload22:53:583
124626999191,16cyclictest1294346-21diskmemload22:53:583
124626999190,7cyclictest1267230-21ntpq20:00:203
124626999190,17cyclictest1294346-21diskmemload21:48:543
124626799199,7cyclictest1420328-21df23:35:132
124626799198,8cyclictest1312574-21ntpq21:30:152
1246267991917,1cyclictest0-21swapper/223:05:112
124626799191,6cyclictest1345452-21ssh22:09:382
1246267991914,3cyclictest1363526-21cpuspeed_turbos22:30:082
124626799191,2cyclictest0-21swapper/222:13:472
124626799191,16cyclictest1294346-21diskmemload00:37:232
1246267991910,6cyclictest1294653-21ssh21:10:062
1246267991910,6cyclictest1278307-21grep20:30:162
1246267991910,6cyclictest1262013-21munin-run19:50:002
124626799190,17cyclictest1467428-21ntpq00:30:142
124626799190,17cyclictest1417048-21ssh23:30:252
124626799190,16cyclictest1256297-21sshd19:35:062
124626399199,7cyclictest1368121-21/usr/sbin/munin22:35:181
124626399199,7cyclictest1293486-21ld21:06:151
124626399198,7cyclictest1272364-21apt-get20:15:121
124626399192,12cyclictest191rcu_preempt21:46:111
124626399191,6cyclictest1343960-21ssh22:06:531
124626399191,6cyclictest1311014-21taskset21:29:241
124626399191,5cyclictest1300023-21cat21:15:161
124626399191,16cyclictest1464308-21ssh00:26:061
1246263991911,5cyclictest1430948-21ssh23:46:321
1246263991910,6cyclictest726-21gdbus20:50:261
1246263991910,6cyclictest1459059-21cron00:24:561
1246263991910,6cyclictest1314000-21taskset21:31:391
124626399190,6cyclictest1472644-21ssh00:35:471
124625999195,5cyclictest699-21dbus-daemon20:35:000
124626999181,7cyclictest1424615-21ssh23:39:593
124626999181,7cyclictest1355631-21grep22:20:173
124626999181,5cyclictest1294346-21diskmemload23:47:293
1246269991811,5cyclictest1304697-21cut21:20:183
124626999181,14cyclictest1416272-21ntpq23:30:133
1246269991811,2cyclictest191rcu_preempt20:30:003
1246269991810,6cyclictest1378823-21sh22:47:353
124626999180,6cyclictest1446824-21ssh00:03:443
124626999180,6cyclictest1352900-21ssh22:16:353
124626999180,16cyclictest940-21runrttasks00:15:413
124626799189,6cyclictest1332732-21cron21:55:002
124626799188,6cyclictest726-21gdbus19:40:232
124626799188,6cyclictest726-21gdbus19:40:222
1246267991814,2cyclictest1276672-21gltestperf20:25:142
1246267991812,4cyclictest1294346-21diskmemload23:46:212
1246267991811,5cyclictest1283764-21unixbench_multi20:40:322
124626799181,14cyclictest1284822-21grep20:45:452
1246267991810,6cyclictest1382220-21ssh22:50:352
1246267991810,6cyclictest1382220-21ssh22:50:352
1246267991810,6cyclictest1376534-21seq22:44:222
124626399188,7cyclictest1394378-21ssh23:04:511
124626399187,7cyclictest709-21systemd-logind20:48:011
124626399187,7cyclictest288-21systemd-udevd20:00:111
1246263991815,2cyclictest488-21hwrng00:16:231
1246263991815,2cyclictest1276641-21gltestperf20:25:141
124626399181,4cyclictest1270381-21/usr/sbin/munin20:10:241
1246263991813,3cyclictest1451140-21/usr/sbin/munin00:10:231
1246263991813,3cyclictest1364464-21mailstats22:30:171
1246263991811,5cyclictest1405921-21ssh23:18:231
124626399181,15cyclictest1264211-21apt-get19:55:141
124626399180,6cyclictest1294666-21ssh21:10:061
124626999179,6cyclictest1454448-21sh00:14:023
124626999179,6cyclictest1365453-21taskset22:31:153
124626999178,6cyclictest1251219-21ldconfig19:21:133
124626999178,6cyclictest1-21systemd19:35:003
124626999176,9cyclictest1260616-21tail19:45:183
124626999171,3cyclictest1471706-21ntpq00:35:173
1246269991713,3cyclictest1258450-21cpuspeed_turbos19:40:133
1246269991713,3cyclictest1258450-21cpuspeed_turbos19:40:123
1246269991713,3cyclictest1257178-21pmu-power19:35:213
1246269991713,2cyclictest940-21runrttasks20:30:203
1246269991713,2cyclictest0-21swapper/319:55:013
124626999171,14cyclictest1294346-21diskmemload22:37:033
124626999171,14cyclictest1247725-21grep19:10:273
1246269991710,5cyclictest1273180-21/usr/sbin/munin20:15:203
124626799179,6cyclictest1414231-21ssh23:28:302
124626799171,3cyclictest1288050-21/usr/sbin/munin20:50:282
1246267991713,3cyclictest1321125-21cpuspeed_turbos21:40:122
1246267991713,3cyclictest1289015-21gltestperf20:55:142
1246267991710,5cyclictest1463757-21apt-key00:25:012
124626799170,4cyclictest1231028-21munin-node20:05:122
124626799170,14cyclictest1360598-21sh22:26:012
124626399179,5cyclictest932-21ntpd20:35:201
124626399177,7cyclictest1426612-21sed23:40:511
124626399177,7cyclictest1257018-21ld19:35:181
124626399176,7cyclictest1250549-21grep19:20:101
1246263991714,2cyclictest488-21hwrng19:47:041
1246263991713,2cyclictest0-21swapper/119:55:001
124626399171,10cyclictest1284206-21latency_hist20:45:011
1246263991710,4cyclictest1290711-21cpu21:00:131
124626399170,8cyclictest1304869-21ssh21:20:201
124626999166,7cyclictest1399144-21ntpq23:10:183
124626999166,7cyclictest1269171-21ntpq20:05:193
1246269991611,4cyclictest1291482-21ntp_states21:00:213
124626999160,3cyclictest1286184-21apt-get20:50:013
124626799166,6cyclictest699-21dbus-daemon20:24:592
1246267991614,1cyclictest488-21hwrng20:11:382
124626799161,3cyclictest1250803-21tail19:21:092
1246267991612,3cyclictest1262558-21gltestperf19:50:292
1246267991612,2cyclictest0-21swapper/220:00:132
124626799160,14cyclictest1290663-21/usr/sbin/munin21:00:152
1246263991613,2cyclictest1269196-21kworker/u16:120:05:201
124626999151,12cyclictest1274324-21runrttasks20:20:013
124626799157,5cyclictest1292711-21cpuspeed_turbos21:05:122
124626399156,6cyclictest288-21systemd-udevd19:42:121
124626399156,6cyclictest288-21systemd-udevd19:42:111
1246263991510,3cyclictest1255161-21ntpq19:30:551
124626799146,5cyclictest1246917-21kernelversion19:10:182
1246267991411,1cyclictest1252394-21apt-get19:25:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional