You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 06:42
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7.osadl.org (updated Wed Apr 02, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
294615299722,50cyclictest3144934-21sendmail-msp00:22:390
294615299468,26cyclictest11050irq/107-eth%d00:28:330
294615299439,25cyclictest3039073-21unixbench-2d22:16:030
294615299432,32cyclictest0-21swapper/023:36:010
294615299393,23cyclictest11050irq/107-eth%d22:50:550
294615299357,19cyclictest0-21swapper/021:21:000
294615299351,21cyclictest0-21swapper/022:33:210
294615299325,21cyclictest0-21swapper/023:21:000
2946152993217,4cyclictest0-21swapper/023:55:530
2946152993126,4cyclictest0-21swapper/022:10:570
294615299310,19cyclictest0-21swapper/023:47:110
2946154993025,3cyclictest1006-21mta-sts-daemon19:51:062
294615299305,14cyclictest0-21swapper/022:30:550
294615299302,17cyclictest0-21swapper/023:14:060
294615299302,17cyclictest0-21swapper/023:14:060
294615299291,23cyclictest0-21swapper/020:55:590
2946152992912,10cyclictest3112535-21ssh23:42:340
294615499288,14cyclictest3141639-21ssh00:18:542
294615399283,6cyclictest11050irq/107-eth%d23:55:531
294615299282,23cyclictest0-21swapper/000:21:030
2946152992817,1cyclictest2972003-21/usr/sbin/munin20:37:340
294615299272,17cyclictest967-21snmpd19:15:580
2946152992714,11cyclictest0-21swapper/000:36:080
294615499263,10cyclictest3122475-21/usr/sbin/munin23:56:022
294615399268,2cyclictest11050irq/107-eth%d00:16:011
2946153992619,5cyclictest2976147-21fschecks_count20:51:011
294615299261,5cyclictest3081091-21ssh23:06:300
2946152992614,2cyclictest0-21swapper/021:54:230
294615299260,16cyclictest0-21swapper/021:59:570
2946152992522,2cyclictest3043156-21ntpq22:21:000
294615299252,22cyclictest0-21swapper/021:38:290
294615299251,4cyclictest3127057-21/usr/sbin/munin00:00:580
294615299250,17cyclictest3085078-21cat23:10:580
294615399244,17cyclictest3152640-21ssh00:32:361
2946152992421,2cyclictest3003816-21grep21:35:560
2946152992415,4cyclictest0-21swapper/023:32:040
294615299241,17cyclictest0-21swapper/021:29:190
294615599234,7cyclictest0-21swapper/321:47:343
294615599233,7cyclictest0-21swapper/323:56:023
294615499233,16cyclictest3085504-21ssh23:11:232
294615399231,9cyclictest11050irq/107-eth%d20:22:341
2946153992317,3cyclictest3150078-21fschecks_count00:30:561
294615299235,8cyclictest1007-21mta-sts-daemon21:42:360
294615299235,15cyclictest0-21swapper/020:08:050
294615299230,14cyclictest0-21swapper/021:13:250
112699230,4rtkit-daemon1125-21rtkit-daemon21:06:022
294615499226,13cyclictest3067677-21ntpq22:50:552
2946154992217,4cyclictest3008315-21fschecks_count21:40:552
2946153992219,2cyclictest3044679-21sh22:22:331
2946153992215,6cyclictest2957994-21munin-plugin-st19:47:341
294615399220,19cyclictest3033449-21fschecks_time22:10:511
294615299229,5cyclictest0-21swapper/022:25:400
294615299228,12cyclictest11050irq/107-eth%d00:16:020
2946152992216,5cyclictest2957045-21grep19:45:550
2946152992213,6cyclictest2984280-21cut21:12:340
294615299220,17cyclictest0-21swapper/022:43:230
294615299220,15cyclictest3071949-21sort22:55:550
112699220,0rtkit-daemon0-21swapper/123:20:381
2946154992115,4cyclictest0-21swapper/222:41:032
294615499210,16cyclictest3098952-21ssh23:28:072
294615399212,14cyclictest3077073-21ssh23:00:591
294615299214,14cyclictest0-21swapper/023:52:340
294615299212,16cyclictest11050irq/107-eth%d00:07:340
2946152992114,5cyclictest2961950-21gltestperf20:00:570
2946152992113,6cyclictest2967133-21grep20:20:560
112699210,6rtkit-daemon1008-21mta-sts-daemon23:44:382
294615599203,8cyclictest0-21swapper/300:22:393
294615499203,11cyclictest3051231-21ssh22:30:592
294615399205,10cyclictest3038856-21/usr/sbin/munin22:16:021
294615399201,5cyclictest11050irq/107-eth%d22:31:351
294615299203,10cyclictest0-21swapper/019:12:330
2946152992012,6cyclictest3157570-21tune2fs00:40:570
2946152992012,3cyclictest0-21swapper/023:26:480
294615599196,2cyclictest0-21swapper/300:28:333
294615499196,6cyclictest3132135-21apt-get00:07:362
294615499195,10cyclictest3038663-21ntpq22:16:002
294615499193,12cyclictest3069798-21ssh22:52:362
294615399193,13cyclictest3069799-21apt-get22:52:361
294615399192,10cyclictest2968122-21sendmail-msp20:22:361
2946153991913,4cyclictest3021182-21gltestperf21:55:571
294615299192,9cyclictest11050irq/107-eth%d20:40:590
2946152991916,2cyclictest0-21swapper/023:00:520
2946152991915,3cyclictest2974693-21grep20:45:560
2946152991912,5cyclictest2979578-21grep21:00:560
2946152991912,5cyclictest2948532-21/usr/sbin/munin19:20:560
294615299191,13cyclictest3060660-21ssh22:41:110
294615299191,12cyclictest3132722-21ssh00:07:360
294615299190,10cyclictest2959078-21ntpq19:51:050
112699190,1rtkit-daemon0-21swapper/100:24:001
294615499189,3cyclictest2984286-21latency_hist21:12:342
294615499183,3cyclictest11050irq/107-eth%d23:40:542
294615499182,14cyclictest3013981-21idleruntime-cro21:47:332
2946154991813,3cyclictest3147919-21latency_hist00:27:342
294615399189,6cyclictest261rcuc/122:00:541
294615399185,8cyclictest3084174-21/usr/sbin/munin23:10:511
294615399183,9cyclictest3157516-21grep00:40:571
294615299187,3cyclictest3018672-21date21:52:340
2946152991814,3cyclictest2952111-21sed19:30:590
294615299180,14cyclictest0-21swapper/021:27:130
112699180,3rtkit-daemon1125-21rtkit-daemon21:24:232
112699180,0rtkit-daemon0-21swapper/123:12:361
112699180,0rtkit-daemon0-21swapper/123:12:361
294615499173,1cyclictest11050irq/107-eth%d23:37:342
294615499171,4cyclictest11050irq/107-eth%d23:24:172
2946154991713,4cyclictest2964214-21ntp_kernel_pll_20:10:572
2946154991711,3cyclictest3044770-21latency_hist22:22:342
2946154991710,4cyclictest3092904-21/usr/sbin/munin23:20:562
294615399178,6cyclictest0-21swapper/122:40:561
294615399177,5cyclictest11050irq/107-eth%d23:28:071
294615399173,8cyclictest3111057-21ssh23:40:581
294615399173,12cyclictest3004424-21/usr/sbin/munin21:36:021
294615399173,11cyclictest3009198-21/usr/sbin/munin21:41:021
294615399172,11cyclictest2972298-21latency_hist20:37:341
2946153991715,1cyclictest0-21swapper/100:07:341
2946153991712,4cyclictest3144095-21unixbench_singl00:21:041
294615299175,3cyclictest2976772-21ntpq20:51:070
294615299175,10cyclictest2965041-21/usr/sbin/munin20:16:030
2946152991713,3cyclictest2955374-21grep19:40:560
2946152991712,3cyclictest2950337-21/usr/sbin/munin19:25:570
294615299170,10cyclictest0-21swapper/022:02:530
294615599164,7cyclictest0-21swapper/322:50:553
294615499169,4cyclictest3088687-21grep23:15:562
294615499169,4cyclictest3088687-21grep23:15:562
294615499166,5cyclictest2998814-21/usr/sbin/munin21:30:532
294615499163,2cyclictest0-21swapper/200:32:372
294615499163,11cyclictest2962664-21cat20:01:042
294615499162,13cyclictest3127054-21sh00:00:532
294615499162,12cyclictest2987124-21/usr/sbin/munin21:15:572
2946154991613,3cyclictest2950455-21ntp_kernel_err19:25:592
2946154991611,4cyclictest0-21swapper/221:01:032
294615499161,13cyclictest3003783-21grep21:35:562
2946154991611,3cyclictest2949412-21cron19:22:342
2946154991611,3cyclictest0-21swapper/200:40:562
294615399163,10cyclictest2980538-21idleruntime-cro21:02:331
294615399162,10cyclictest3119717-21ssh23:51:351
2946153991613,2cyclictest619-21mta-sts-daemon21:22:041
2946153991613,2cyclictest2983210-21gltestperf21:10:581
2946153991612,2cyclictest0-21swapper/122:36:031
2946152991613,2cyclictest2953600-21grep19:35:550
2946152991611,4cyclictest2982277-21find21:06:030
2946152991610,5cyclictest2969013-21aten2.4-expect20:30:490
294615299160,2cyclictest2960686-21ntpq19:55:590
294615499158,5cyclictest3152280-21latency_hist00:32:352
294615499157,7cyclictest2952654-21/usr/sbin/munin19:31:042
294615499155,9cyclictest2972557-21apt-get20:37:362
294615499154,5cyclictest3080116-21ssh23:04:542
294615499153,7cyclictest3048137-21latency_hist22:27:352
294615499152,9cyclictest2965180-21aten_r2power_po20:15:552
294615499152,9cyclictest2953648-21tune2fs19:35:552
2946154991512,2cyclictest2968034-21unixbench_singl20:21:032
2946154991511,3cyclictest2972308-21latency_hist20:37:342
2946154991511,2cyclictest3033538-21grep22:10:512
294615499151,11cyclictest3025682-21tune2fs22:00:532
294615499151,11cyclictest2960331-21sed19:55:562
294615499150,11cyclictest2956694-21aten2.4-expect19:45:532
294615399159,5cyclictest2949324-21unixbench_multi19:21:031
294615399158,4cyclictest0-21swapper/120:47:341
294615399153,9cyclictest2955302-21grep19:40:561
294615399153,9cyclictest1-21systemd21:27:341
2946153991511,3cyclictest2986744-21fschecks_count21:15:551
2946153991511,3cyclictest2978751-21grep20:56:031
294615399151,10cyclictest3128525-21grep00:01:041
2946153991510,3cyclictest2965353-21fschecks_count20:15:561
2946153991510,3cyclictest0-21swapper/121:07:341
2946153991510,3cyclictest0-21swapper/119:25:541
112699150,2rtkit-daemon1125-21rtkit-daemon19:39:262
294615599144,6cyclictest0-21swapper/322:22:333
294615599144,5cyclictest0-21swapper/322:35:583
294615599143,8cyclictest0-21swapper/323:10:513
294615599142,8cyclictest0-21swapper/319:27:363
294615599142,5cyclictest0-21swapper/300:15:403
294615499149,4cyclictest0-21swapper/223:50:552
294615499146,5cyclictest2979089-21wc20:57:342
294615499143,9cyclictest3022088-21/usr/sbin/munin21:56:042
294615499143,7cyclictest3076689-21ssh23:00:562
294615499142,8cyclictest3063797-21ssh22:46:122
294615399149,4cyclictest2963692-21fschecks_count20:10:531
294615399146,7cyclictest11050irq/107-eth%d00:11:221
294615399145,6cyclictest3105211-21/usr/sbin/munin23:36:061
294615399142,7cyclictest2999119-21/usr/sbin/munin21:30:551
294615399142,4cyclictest2958656-21grep19:51:011
294615399141,2cyclictest3031073-21latency_hist22:07:341
2946153991412,1cyclictest0-21swapper/121:50:531
2946153991412,1cyclictest0-21swapper/119:31:001
2946153991410,2cyclictest0-21swapper/121:47:341
294615599135,3cyclictest0-21swapper/322:27:493
294615599134,4cyclictest0-21swapper/320:45:593
294615599134,4cyclictest0-21swapper/300:10:493
294615599134,3cyclictest0-21swapper/320:10:573
294615599133,6cyclictest0-21swapper/300:21:033
294615599133,5cyclictest0-21swapper/321:42:343
294615599133,4cyclictest0-21swapper/322:00:543
294615599132,6cyclictest0-21swapper/323:21:033
294615599132,6cyclictest0-21swapper/321:16:013
294615599132,6cyclictest0-21swapper/300:32:373
294615599132,5cyclictest0-21swapper/323:42:593
294615599132,5cyclictest0-21swapper/323:35:553
294615599131,7cyclictest0-21swapper/321:36:023
294615499138,3cyclictest0-21swapper/200:07:342
294615499135,8cyclictest0-21swapper/222:35:532
294615499134,6cyclictest3017536-21grep21:50:582
294615499133,7cyclictest0-21swapper/221:20:592
294615499132,6cyclictest2946105-21munin-run19:12:332
294615499131,9cyclictest2970241-21sed20:30:592
294615499131,8cyclictest3139172-21/usr/sbin/munin00:15:582
294615399137,3cyclictest0-21swapper/122:50:521
294615399134,8cyclictest2960687-21awk19:55:591
294615399132,6cyclictest2974179-21grep20:42:341
294615399131,9cyclictest2968982-21aten_r2power_cu20:30:491
294615299136,6cyclictest2968702-21cron20:27:340
294615599125,5cyclictest0-21swapper/322:55:543
294615599123,5cyclictest0-21swapper/323:40:583
294615599123,4cyclictest0-21swapper/321:11:003
294615599122,7cyclictest0-21swapper/321:02:373
294615599122,6cyclictest0-21swapper/322:15:573
294615599121,6cyclictest0-21swapper/300:01:023
294615599121,4cyclictest0-21swapper/319:51:013
294615599120,6cyclictest0-21swapper/323:30:483
294615499128,3cyclictest111788-21systemd-journal20:07:342
294615499127,3cyclictest0-21swapper/219:15:552
294615499121,8cyclictest2976379-21/usr/sbin/munin20:51:022
294615499120,5cyclictest11050irq/107-eth%d20:45:532
294615499120,10cyclictest967-21snmpd20:22:362
294615399123,6cyclictest2962783-21unixbench_multi20:01:041
294615399122,7cyclictest11050irq/107-eth%d23:06:301
294615399121,6cyclictest2963306-21latency_hist20:07:341
2946153991210,1cyclictest2954632-21grep19:36:031
294615399120,3cyclictest3098408-21munin-plugin-st23:27:341
294615299123,4cyclictest762-21in:imuxsock20:02:420
294615599116,4cyclictest0-21swapper/323:24:173
294615599113,4cyclictest0-21swapper/322:44:363
294615599113,4cyclictest0-21swapper/321:55:583
294615599113,4cyclictest0-21swapper/319:35:593
294615599113,3cyclictest0-21swapper/320:35:533
294615599112,6cyclictest0-21swapper/321:52:343
294615599112,5cyclictest0-21swapper/323:50:553
294615599112,5cyclictest0-21swapper/321:30:573
294615599112,5cyclictest0-21swapper/321:20:543
294615599112,5cyclictest0-21swapper/321:01:003
294615599112,5cyclictest0-21swapper/319:45:543
294615599111,5cyclictest0-21swapper/320:55:593
294615599110,5cyclictest0-21swapper/323:00:563
294615499114,3cyclictest0-21swapper/222:02:422
294615399114,3cyclictest1005-21mta-sts-daemon22:22:591
294615399113,4cyclictest3064807-21latency_hist22:47:341
294615399111,7cyclictest111788-21systemd-journal19:12:331
294615399111,6cyclictest2946680-21fschecks_count19:15:551
294615399111,5cyclictest11050irq/107-eth%d23:44:281
294615599104,3cyclictest0-21swapper/320:51:093
294615599103,4cyclictest0-21swapper/319:41:003
294615599103,4cyclictest0-21swapper/300:05:043
294615599102,7cyclictest0-21swapper/322:10:523
294615599102,4cyclictest0-21swapper/319:26:033
294615599102,3cyclictest0-21swapper/319:15:533
294615599100,5cyclictest0-21swapper/323:14:043
294615599100,5cyclictest0-21swapper/323:14:043
294615599100,5cyclictest0-21swapper/322:40:573
29461559994,4cyclictest0-21swapper/323:06:393
29461559994,4cyclictest0-21swapper/320:16:023
29461559993,3cyclictest0-21swapper/320:37:363
29461559993,3cyclictest0-21swapper/300:39:343
29461559992,4cyclictest0-21swapper/319:56:033
29461559991,6cyclictest0-21swapper/320:22:363
29461559991,4cyclictest0-21swapper/319:20:543
29461559990,4cyclictest0-21swapper/320:20:533
29461559983,3cyclictest0-21swapper/320:00:543
29461559982,4cyclictest0-21swapper/322:26:133
29461559981,5cyclictest0-21swapper/320:30:513
29461559981,4cyclictest0-21swapper/322:02:503
29461559972,3cyclictest0-21swapper/321:25:073
29461559972,3cyclictest0-21swapper/320:07:343
29461559951,3cyclictest0-21swapper/319:12:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional