You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-23 - 16:41
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sat Nov 23, 2024 12:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23808991076,80cyclictest0-21swapper/008:30:060
23808991063,93cyclictest0-21swapper/011:09:420
23808991063,93cyclictest0-21swapper/011:09:420
23808991022,9cyclictest28480-21apt-get11:54:440
2380899963,10cyclictest0-21swapper/011:29:500
2380899962,16cyclictest12215-21grep10:30:060
2380899962,16cyclictest12215-21grep10:30:060
2380899954,9cyclictest2793-21apt-get10:14:430
23819999429,13cyclictest11770-21chrt08:09:541
23808999414,50cyclictest31429-21apt-get09:09:480
23819999237,10cyclictest0-21swapper/112:34:451
23819999234,9cyclictest0-21swapper/111:19:591
2380899916,31cyclictest5077-21ssh11:14:410
2380899914,9cyclictest5303-21apt-get12:09:450
2380899914,9cyclictest5303-21apt-get12:09:450
2380899912,12cyclictest12205-21/usr/sbin/munin12:20:090
2380899912,12cyclictest12205-21/usr/sbin/munin12:20:090
2380899903,9cyclictest276802sleep007:19:460
2380899902,78cyclictest27589-21latency_hist08:59:310
2380899902,12cyclictest976-21chrt11:07:530
2380899902,12cyclictest976-21chrt11:07:530
23827998949,15cyclictest31309-21/usr/sbin/munin11:04:532
23827998949,15cyclictest31309-21/usr/sbin/munin11:04:532
2380899893,14cyclictest29849-21df_inode10:04:480
2380899892,5cyclictest0-21swapper/010:44:400
23819998829,9cyclictest0-21swapper/109:45:041
2380899885,9cyclictest11414-21apt-get08:09:420
2380899884,13cyclictest26952-21apt-get09:59:420
2380899882,11cyclictest15139-21irqstats12:25:030
2380899882,11cyclictest15139-21irqstats12:25:030
2380899873,12cyclictest11270-21munin-run12:19:280
2380899873,12cyclictest11270-21munin-run12:19:280
2380899872,5cyclictest0-21swapper/012:35:350
2380899863,9cyclictest15466-21ls08:20:020
2380899863,74cyclictest394-21runrttasks07:57:320
2380899863,14cyclictest6435-21ls07:54:320
2380899863,10cyclictest23384-21grep11:45:030
2380899862,75cyclictest16734-21unixbench_singl09:40:040
2368628616,43sleep2366-21dbus-daemon07:09:302
2380899853,75cyclictest1520-21grep07:39:310
2380899853,73cyclictest23799-21cyclictest07:09:330
2380899853,72cyclictest32396-21ssh10:09:350
2380899853,13cyclictest8812-21ssh09:26:060
2380899853,11cyclictest19070-21ntp_states09:44:580
2380899852,9cyclictest8444-21ssh10:24:390
2380899852,11cyclictest23223-21sort08:44:490
2380899852,10cyclictest32072-21cstates12:00:180
2380899852,10cyclictest22559-21df10:49:490
2380899843,14cyclictest17736-21/usr/sbin/munin12:30:220
2380899843,14cyclictest17736-21/usr/sbin/munin12:30:220
23808998431,45cyclictest0-21swapper/009:14:510
2380899842,10cyclictest10268-21ssh09:29:550
2380899833,13cyclictest24858-21awk09:55:010
2380899832,9cyclictest23064-21ssh09:52:150
2380899832,9cyclictest0-21swapper/011:19:410
2380899832,15cyclictest26300-21cut07:14:540
2380899832,15cyclictest26300-21cut07:14:540
2380899832,13cyclictest30846-21modprobe09:09:290
2380899832,11cyclictest5303-21apt-get12:09:410
2380899832,11cyclictest31517-21wc07:30:010
2380899832,11cyclictest31517-21wc07:30:010
23819998244,14cyclictest141rcu_preempt12:22:191
23819998244,14cyclictest141rcu_preempt12:22:191
2380899823,9cyclictest8432-21grep07:59:500
2380899822,14cyclictest30078-21timerandwakeup07:25:010
2380899822,14cyclictest1875-21sort07:39:500
2380899822,10cyclictest13024-21apt-get08:14:420
2380899822,10cyclictest13024-21apt-get08:14:420
23808998214,59cyclictest8150irq/69-2188000.10:54:400
2238228225,23sleep10-21swapper/107:04:491
2380899812,70cyclictest17193-21ls10:39:510
2380899812,70cyclictest17193-21ls10:39:510
2380899812,6cyclictest0-21swapper/010:34:580
2380899812,12cyclictest412-21snmpd11:00:010
23827998059,11cyclictest21112-21basename08:39:292
2380899806,65cyclictest0-21swapper/007:09:500
2380899803,9cyclictest5661-21ssh09:20:030
2380899803,13cyclictest16282-21ls11:34:450
2380899803,12cyclictest25511-21/usr/sbin/munin11:49:450
2380899803,10cyclictest17636-21munin-run08:29:290
23839997958,9cyclictest0-21swapper/309:09:293
2380899793,9cyclictest12864-21apt-get09:34:400
2380899793,10cyclictest28301-21sed08:59:540
2380899792,70cyclictest0-21swapper/008:39:500
2380899792,68cyclictest9779-21/usr/sbin/munin08:05:080
23819997831,8cyclictest0-21swapper/111:29:421
2380899783,12cyclictest19729-21/usr/sbin/munin08:35:010
2380899782,7cyclictest7222-21ssh10:21:400
2380899782,7cyclictest7222-21ssh10:21:400
2380899782,11cyclictest19547-21grep11:39:490
23819997731,13cyclictest815-21threads07:35:031
2380899773,66cyclictest25519-21grep08:50:040
2380899773,65cyclictest3375-21sort07:44:470
2380899773,10cyclictest0-21swapper/011:28:140
23827997458,9cyclictest24367-21kworker/u8:008:49:292
23827997350,13cyclictest23946-21latency_hist12:39:312
23827997154,10cyclictest29347-21unixbench_multi11:00:042
23819997152,12cyclictest28196-21latency_hist10:59:311
23827996952,11cyclictest20300-21unixbench_multi11:40:022
23839996820,14cyclictest141rcu_preempt07:49:313
23839996746,12cyclictest0-21swapper/308:19:293
23839996746,12cyclictest0-21swapper/308:19:293
23839996720,8cyclictest141rcu_preempt09:19:313
23819996639,18cyclictest15390-21munin-plugin-st09:39:291
23839996550,9cyclictest20466-21modprobe12:34:283
23839996550,9cyclictest20466-21modprobe12:34:283
23827996548,10cyclictest15393-21idleruntime-cro09:39:292
23819996545,12cyclictest30671-21tail10:05:051
23819996539,17cyclictest8426-21unixbench_singl09:25:041
23839996445,13cyclictest15372-21modprobe09:39:293
23839996423,13cyclictest41-21ksoftirqd/312:04:313
23827996449,9cyclictest9516-21unixbench_singl10:25:042
23839996247,8cyclictest141rcu_preempt11:15:023
23839996246,9cyclictest0-21swapper/311:49:293
23839996219,14cyclictest141rcu_preempt09:34:313
23827996245,11cyclictest29968-21fschecks_time10:04:512
23827996222,14cyclictest0-21swapper/207:19:472
23819996234,17cyclictest30850-21diskmemload10:53:321
23839996145,9cyclictest25038-21sh10:54:293
23839996145,8cyclictest0-21swapper/312:36:043
23839996127,8cyclictest141rcu_preempt09:19:563
23827996142,12cyclictest11262-21sendmail12:19:282
23827996142,12cyclictest11262-21sendmail12:19:282
23819996146,8cyclictest0-21swapper/112:09:281
23819996145,10cyclictest24084-21kworker/u8:010:04:291
23819996145,10cyclictest15780-21unixbench_multi12:25:171
23819996145,10cyclictest15780-21unixbench_multi12:25:171
23819996138,13cyclictest179-21pr/ttyS007:28:211
23839996024,9cyclictest141rcu_preempt09:09:353
23827996043,11cyclictest5268-21kworker/u8:310:44:492
23827996038,15cyclictest26383-21grep10:55:042
23819996045,8cyclictest116-21pr/ttymxc008:38:161
23839995944,8cyclictest0-21swapper/310:39:283
23839995942,10cyclictest25577-21unixbench_singl08:50:053
23839995920,18cyclictest41-21ksoftirqd/308:34:523
23827995943,9cyclictest26465-21grep11:50:032
23827995943,10cyclictest627-21in:imklog09:59:472
23827995943,10cyclictest32724-21fschecks_time10:09:532
23819995933,18cyclictest28234-21kernelversion08:59:531
23819995931,17cyclictest627-21in:imklog09:12:031
23819995919,8cyclictest0-21swapper/110:09:571
23839995843,9cyclictest627-21in:imklog11:58:443
23839995840,12cyclictest22060-21unin-run11:44:293
23827995842,9cyclictest32486-21unixbench_multi09:10:182
23827995841,11cyclictest30850-21diskmemload12:03:352
23827995840,12cyclictest8229-21threads09:25:022
23819995842,9cyclictest179-21pr/ttyS011:11:321
23819995842,9cyclictest179-21pr/ttyS011:11:321
23819995840,11cyclictest24351-21modprobe08:49:291
23819995839,8cyclictest0-21swapper/108:50:041
23819995834,18cyclictest179-21pr/ttyS011:17:481
23839995742,9cyclictest0-21swapper/309:59:293
23839995742,10cyclictest0-21swapper/310:55:023
23839995741,10cyclictest30850-21diskmemload11:28:393
23839995741,10cyclictest275-21systemd-journal09:45:013
23839995721,12cyclictest14653-21kworker/u8:109:39:483
23827995742,8cyclictest0-21swapper/209:09:292
23827995720,10cyclictest141rcu_preempt11:55:022
2381999577,32cyclictest627-21in:imklog12:29:481
2381999577,32cyclictest627-21in:imklog12:29:481
23819995743,7cyclictest0-21swapper/111:55:051
23819995742,9cyclictest627-21in:imklog11:36:451
23819995741,10cyclictest22317-21unixbench_multi09:50:111
23839995641,9cyclictest14350-21sh12:24:283
23839995641,9cyclictest14350-21sh12:24:283
23839995640,8cyclictest0-21swapper/307:34:503
23839995640,10cyclictest275-21systemd-journal08:54:453
23839995639,11cyclictest275-21systemd-journal11:49:533
23839995637,12cyclictest627-21in:imklog10:41:283
23839995637,12cyclictest627-21in:imklog10:41:283
23827995640,10cyclictest5254-21cpuspeed11:15:022
23827995639,11cyclictest14346-21latency_hist12:24:282
23827995639,11cyclictest14346-21latency_hist12:24:282
23827995638,12cyclictest275-21systemd-journal07:29:472
23827995638,12cyclictest275-21systemd-journal07:29:472
23827995637,12cyclictest0-21swapper/212:30:222
23827995637,12cyclictest0-21swapper/212:30:222
23827995637,11cyclictest10170-21cpuspeed09:29:522
23827995628,11cyclictest275-21systemd-journal11:29:502
23819995636,13cyclictest23799-21cyclictest10:39:511
23819995636,13cyclictest23799-21cyclictest10:39:511
23819995634,13cyclictest275-21systemd-journal07:22:221
23819995631,16cyclictest627-21in:imklog08:04:451
227942569,28sleep322825-21tr07:05:043
23839995536,13cyclictest27377-21cron07:19:293
23839995536,13cyclictest27377-21cron07:19:293
23827995538,10cyclictest275-21systemd-journal09:00:032
23827995537,10cyclictest179-21pr/ttyS008:30:152
23827995515,14cyclictest141rcu_preempt10:14:322
23819995540,8cyclictest0-21swapper/111:59:411
23819995539,10cyclictest11647-21latency_hist12:19:481
23819995538,11cyclictest12770-21latency_hist09:34:311
23819995526,13cyclictest275-21systemd-journal10:18:111
23839995432,16cyclictest10722-21sh11:24:283
23827995438,8cyclictest0-21swapper/207:15:062
23827995438,8cyclictest0-21swapper/207:15:062
23827995438,10cyclictest626-21in:imuxsock12:26:582
23827995438,10cyclictest626-21in:imuxsock12:26:582
23827995436,12cyclictest627-21in:imklog11:09:542
23827995436,12cyclictest627-21in:imklog11:09:542
23819995436,12cyclictest19110-21unixbench_singl08:30:271
23819995417,16cyclictest13553-21ntp_states08:14:591
23819995417,16cyclictest13553-21ntp_states08:14:591
23839995339,7cyclictest0-21swapper/307:25:023
23839995338,9cyclictest179-21pr/ttyS007:20:033
23839995336,10cyclictest0-21swapper/311:04:303
23827995338,8cyclictest0-21swapper/211:24:282
23827995336,11cyclictest627-21in:imklog09:57:522
23827995336,11cyclictest1-21systemd09:44:582
23827995332,14cyclictest16531-21munin-plugin-st10:39:282
23819995339,8cyclictest0-21swapper/112:15:191
23819995337,10cyclictest4259-21unixbench_singl07:45:061
23819995336,11cyclictest27588-21sh08:59:291
23819995334,14cyclictest116-21pr/ttymxc007:49:581
23819995330,17cyclictest275-21systemd-journal09:41:101
23819995326,8cyclictest141rcu_preempt08:24:301
23819995314,12cyclictest25-21ksoftirqd/107:40:041
23839995237,9cyclictest8212-21ssh12:14:293
23839995237,9cyclictest8212-21ssh12:14:293
23839995237,9cyclictest275-21systemd-journal10:13:143
23839995237,9cyclictest275-21systemd-journal09:51:443
23839995236,10cyclictest626-21in:imuxsock07:54:293
23827995236,8cyclictest0-21swapper/208:19:292
23827995236,8cyclictest0-21swapper/208:19:292
23827995236,10cyclictest30850-21diskmemload09:19:452
23827995236,10cyclictest0-21swapper/210:34:282
23827995236,10cyclictest0-21swapper/210:34:282
23827995234,11cyclictest627-21in:imklog11:28:172
23827995231,13cyclictest0-21swapper/209:19:292
23827995230,13cyclictest275-21systemd-journal08:49:552
23819995235,8cyclictest0-21swapper/108:39:491
23819995233,9cyclictest0-21swapper/110:25:041
23819995231,14cyclictest275-21systemd-journal11:52:481
23819995212,17cyclictest394-21runrttasks11:25:221
23839995134,11cyclictest275-21systemd-journal11:38:183
23839995134,10cyclictest0-21swapper/308:29:293
23839995133,11cyclictest2576-21/usr/sbin/munin11:10:023
23839995133,11cyclictest2576-21/usr/sbin/munin11:10:023
2383999512,25cyclictest2501-21/usr/sbin/munin12:04:473
2383999512,22cyclictest14876-21diskstats12:24:503
2383999512,22cyclictest14876-21diskstats12:24:503
23839995114,15cyclictest141rcu_preempt09:24:493
23827995137,9cyclictest179-21pr/ttyS007:59:312
23827995134,11cyclictest275-21systemd-journal09:50:192
23827995134,11cyclictest179-21pr/ttyS012:07:142
23827995134,10cyclictest275-21systemd-journal08:19:392
23827995132,12cyclictest17967-21apt-key08:29:322
23827995132,11cyclictest0-21swapper/208:10:042
23827995126,18cyclictest4667-21munin-plugin-st07:49:282
23819995136,8cyclictest0-21swapper/109:09:321
23819995134,11cyclictest275-21systemd-journal10:19:531
23819995134,11cyclictest275-21systemd-journal10:19:531
23819995134,10cyclictest275-21systemd-journal07:14:341
23819995134,10cyclictest275-21systemd-journal07:14:341
23819995130,11cyclictest0-21swapper/109:20:021
23819995129,13cyclictest22085-21kworker/u8:211:09:311
23819995129,13cyclictest22085-21kworker/u8:211:09:311
23819995120,9cyclictest30850-21diskmemload12:09:501
23819995120,9cyclictest30850-21diskmemload12:09:501
2383999505,21cyclictest30850-21diskmemload10:48:563
2383999503,41cyclictest18135-21apt-get08:29:493
23839995034,10cyclictest275-21systemd-journal10:09:233
23839995033,9cyclictest0-21swapper/310:04:323
23839995031,11cyclictest0-21swapper/308:09:543
23839995030,12cyclictest0-21swapper/308:41:253
23827995034,10cyclictest275-21systemd-journal12:11:492
23827995034,10cyclictest275-21systemd-journal12:11:492
23827995031,10cyclictest0-21swapper/210:24:302
23827995031,10cyclictest0-21swapper/210:24:302
23827995030,11cyclictest0-21swapper/209:39:582
23819995035,9cyclictest179-21pr/ttyS009:58:491
23819995035,8cyclictest116-21pr/ttymxc009:16:161
23819995030,10cyclictest0-21swapper/111:44:281
23819995028,8cyclictest24084-21kworker/u8:010:31:431
23819995028,8cyclictest24084-21kworker/u8:010:31:431
23819995023,7cyclictest141rcu_preempt11:04:311
23819995012,11cyclictest141rcu_preempt10:34:491
23839994934,7cyclictest0-21swapper/310:24:293
23839994934,7cyclictest0-21swapper/310:24:293
23839994933,10cyclictest0-21swapper/308:09:323
2383999493,21cyclictest31720-21cat07:30:043
2383999493,21cyclictest31720-21cat07:30:043
23839994930,11cyclictest0-21swapper/310:24:453
23839994916,8cyclictest41-21ksoftirqd/308:44:453
23827994933,10cyclictest275-21systemd-journal08:54:392
23827994932,10cyclictest0-21swapper/211:34:502
23827994931,12cyclictest627-21in:imklog08:00:002
23827994931,11cyclictest116-21pr/ttymxc008:09:322
2381999495,14cyclictest141rcu_preempt07:10:031
23819994912,14cyclictest141rcu_preempt07:59:501
2383999483,34cyclictest394-21runrttasks09:02:423
23839994832,10cyclictest627-21in:imklog10:34:103
23839994832,10cyclictest627-21in:imklog10:34:103
23839994831,10cyclictest275-21systemd-journal11:29:373
23839994828,14cyclictest0-21swapper/307:09:493
23839994810,10cyclictest0-21swapper/307:39:403
23827994831,10cyclictest17878-21unixbench_multi10:40:032
23827994831,10cyclictest17878-21unixbench_multi10:40:032
23827994829,10cyclictest0-21swapper/207:44:292
2382799482,23cyclictest0-21swapper/207:34:402
23819994833,8cyclictest0-21swapper/110:44:491
23819994833,10cyclictest0-21swapper/107:55:021
2383999474,19cyclictest7877-21modprobe07:59:283
23839994721,13cyclictest0-21swapper/311:09:283
23839994721,13cyclictest0-21swapper/311:09:283
2382799473,18cyclictest394-21runrttasks07:24:452
23827994731,8cyclictest0-21swapper/211:47:442
23827994731,10cyclictest394-21runrttasks07:11:482
23827994726,12cyclictest0-21swapper/208:40:002
23827994725,15cyclictest0-21swapper/207:49:482
23839994627,11cyclictest0-21swapper/307:59:563
23819994632,9cyclictest30850-21diskmemload11:44:491
23839994530,9cyclictest275-21systemd-journal10:15:413
23839994525,9cyclictest0-21swapper/308:20:003
23827994526,12cyclictest0-21swapper/210:53:012
23819994526,12cyclictest17629-21cron08:29:291
23839994421,13cyclictest0-21swapper/312:15:193
23839994421,13cyclictest0-21swapper/312:15:193
23819994123,12cyclictest275-21systemd-journal07:29:561
23819994123,12cyclictest275-21systemd-journal07:29:561
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional