You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-25 - 14:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Nov 25, 2024 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3770921995854,3cyclictest3942663-21kworker/u8:0+flush-259:022:07:273
3770910994541,3cyclictest75-21kswapd022:07:270
3770914994439,4cyclictest3932015-21cat21:45:001
3770914994236,4cyclictest3864450-21latency_hist20:40:011
3770921993732,3cyclictest3854173-21kworker/u8:1+flush-259:022:10:253
3770921993730,4cyclictest4019682-21ntpq23:05:203
3770919993734,2cyclictest3900716-21kworker/u8:4+flush-259:022:10:012
3770919993633,2cyclictest3942663-21kworker/u8:0+flush-259:022:20:002
3770914993631,3cyclictest4026147-21latency_hist23:15:011
3770914993530,4cyclictest3894432-21ntpq21:05:211
3770910993528,5cyclictest3937475-21latency_hist21:50:010
3770921993430,3cyclictest3932519-21kworker/u8:5+flush-259:022:40:003
3770921993228,3cyclictest3889829-21kworker/u8:0+flush-259:021:10:193
3770921993228,3cyclictest3854173-21kworker/u8:1+flush-259:022:20:213
3770921993228,3cyclictest3854173-21kworker/u8:1+flush-259:022:20:203
3770921993228,3cyclictest3786544-21kworker/u8:3+events_unbound20:25:203
3770921993226,5cyclictest3832681-21ntpq20:05:223
3770921993128,2cyclictest4005280-21kworker/u8:3+events_unbound00:30:263
3770921993127,2cyclictest3932519-21kworker/u8:5+events_unbound22:30:193
3770921993126,3cyclictest3905892-21/usr/sbin/munin21:20:143
3770921993125,3cyclictest1117191-21systemd-journal23:55:383
3770919993127,3cyclictest4021178-21base6423:10:012
3770919993126,3cyclictest4073923-21/usr/sbin/munin00:05:222
3770914993127,3cyclictest3854173-21kworker/u8:1+flush-259:021:00:151
3770914993126,4cyclictest4083163-21ntpq00:10:221
3770910993124,4cyclictest4031310-21latency_hist23:20:010
3770921993026,2cyclictest4046953-21kworker/u8:4+flush-259:023:45:003
3770914993027,2cyclictest3942663-21kworker/u8:0+flush-259:022:45:161
3770921992926,2cyclictest3926297-21kworker/u8:0+flush-259:021:40:203
3770921992925,3cyclictest4068545-21kworker/u8:1+flush-259:000:05:143
3770921992925,2cyclictest4005280-21kworker/u8:3+flush-259:023:20:173
3770921992825,2cyclictest3786544-21kworker/u8:3+flush-259:020:55:243
3770921992825,2cyclictest3776239-21kworker/u8:2+flush-259:019:20:193
3770921992824,3cyclictest4005280-21kworker/u8:3+events_unbound23:35:153
3770921992824,3cyclictest3994527-21kworker/u8:1+flush-259:022:50:233
3770921992824,3cyclictest3797741-21kworker/u8:0+flush-259:020:10:213
3770914992824,3cyclictest3874992-21/usr/sbin/munin20:50:201
3770910992822,4cyclictest3792164-21/usr/sbin/munin19:30:200
3770910992821,5cyclictest1117191-21systemd-journal21:57:380
3770921992723,3cyclictest3797741-21kworker/u8:0+events_unbound19:55:203
3770921992722,3cyclictest4026374-21kworker/u8:2+events_unbound23:30:223
3770921992623,2cyclictest4005280-21kworker/u8:3+flush-259:000:00:203
3770921992620,4cyclictest3999599-21gpgv22:50:003
3770914992622,3cyclictest4005280-21kworker/u8:3+flush-259:000:25:151
3770910992623,2cyclictest3854173-21kworker/u8:1+flush-259:021:30:010
3770921992522,2cyclictest3854173-21kworker/u8:1+flush-259:021:45:223
3770921992522,2cyclictest3786544-21kworker/u8:3+events_unbound20:00:133
3770921992520,3cyclictest3854173-21kworker/u8:1+events_unbound20:50:183
3770910992518,5cyclictest3977913-21ntpq22:25:200
3770921992421,2cyclictest3854173-21kworker/u8:1+flush-259:021:30:273
3770921992420,3cyclictest3854173-21kworker/u8:1+flush-259:020:30:163
3770921992420,3cyclictest3843863-21kworker/u8:4+flush-259:020:20:263
3770921992420,2cyclictest4031311-21latency_hist23:20:003
3770921992420,2cyclictest3786544-21kworker/u8:3+flush-259:019:35:253
3770919992419,4cyclictest4051889-21apt-get23:40:012
3770914992421,2cyclictest3889829-21kworker/u8:0+flush-259:021:10:181
3770914992419,4cyclictest3915051-21ntpq21:25:201
3770914992418,4cyclictest3770841-21latency_hist19:10:001
3770921992320,2cyclictest4068545-21kworker/u8:1+flush-259:000:35:233
3770921992320,2cyclictest4068545-21kworker/u8:1+flush-259:000:35:233
3770921992320,2cyclictest4005280-21kworker/u8:3+flush-259:023:50:153
3770921992320,2cyclictest3776239-21kworker/u8:2+flush-259:019:15:203
3770921992318,3cyclictest3717301-21kworker/u8:5+events_unbound19:25:183
3770919992320,2cyclictest4026374-21kworker/u8:2+events_unbound23:15:192
3770919992317,4cyclictest1117191-21systemd-journal21:30:002
3770910992318,4cyclictest4005280-21kworker/u8:3+flush-259:000:30:190
3770910992318,3cyclictest3864876-21diskstats20:40:140
3770921992219,2cyclictest3900716-21kworker/u8:4+events_unbound21:50:253
3770921992219,2cyclictest3797741-21kworker/u8:0+events_unbound19:40:173
3770921992219,2cyclictest3786544-21kworker/u8:3+events_unbound21:28:383
3770921992218,3cyclictest3942663-21kworker/u8:0+flush-259:023:00:593
3770919992219,2cyclictest3797741-21kworker/u8:0+flush-259:020:05:252
377091999220,20cyclictest1-21systemd19:20:002
3770914992218,3cyclictest3989399-21apt-get22:40:121
3770910992215,5cyclictest1117191-21systemd-journal20:25:000
377091099221,20cyclictest1-21systemd21:43:380
3770921992118,2cyclictest3786544-21kworker/u8:3+flush-259:020:45:233
3770921992117,3cyclictest4036445-21kworker/u8:0+events_unbound23:25:253
3770921992116,4cyclictest4084258-21apt-get00:15:113
3770914992117,3cyclictest3843863-21kworker/u8:4+flush-259:020:25:001
377091099211,18cyclictest1-21systemd00:39:390
377091099211,18cyclictest1-21systemd00:39:380
3770921992017,2cyclictest4005280-21kworker/u8:3+flush-259:000:25:173
3770921992016,3cyclictest4084019-21kworker/u8:0+flush-259:000:20:183
3770921992016,3cyclictest3973683-21kworker/u8:2+flush-259:022:40:233
3770921992016,3cyclictest3889829-21kworker/u8:0+flush-259:021:15:233
3770921992015,4cyclictest3771135-21apt-get19:10:133
3770919992017,2cyclictest3999493-21munin-run22:50:002
3770919992017,2cyclictest3989138-21turbostat.cron22:40:002
3770919992017,2cyclictest3786544-21kworker/u8:3+flush-259:020:00:242
3770919992016,3cyclictest3854173-21kworker/u8:1+flush-259:020:35:192
3770919992016,3cyclictest3786544-21kworker/u8:3+events_unbound20:55:012
3770919992016,2cyclictest3786544-21kworker/u8:3+flush-259:019:39:592
3770919992015,3cyclictest3968786-21apt-get22:20:112
3770919992015,3cyclictest3968786-21apt-get22:20:112
3770914992017,2cyclictest3932519-21kworker/u8:5+flush-259:022:15:201
3770914992015,4cyclictest4068428-21latency_hist00:00:001
3770910992015,4cyclictest3932519-21kworker/u8:5+flush-259:022:30:290
3770910992015,4cyclictest3854173-21kworker/u8:1+flush-259:021:25:010
3770910992014,4cyclictest3797741-21kworker/u8:0+flush-259:019:55:000
3770921991916,2cyclictest3854173-21kworker/u8:1+flush-259:022:00:193
3770921991916,2cyclictest3786544-21kworker/u8:3+events_unbound19:45:243
3770921991915,3cyclictest4046953-21kworker/u8:4+events_unbound23:46:383
3770921991915,3cyclictest3932519-21kworker/u8:5+events_unbound22:27:383
3770919991915,3cyclictest3776239-21kworker/u8:2+flush-259:019:20:202
3770919991915,2cyclictest4058109-21cron23:49:592
3770914991915,3cyclictest4036661-21/usr/sbin/munin23:25:241
3770914991915,3cyclictest3854173-21kworker/u8:1+events_unbound22:20:151
3770914991915,3cyclictest3854173-21kworker/u8:1+events_unbound22:20:151
3770914991915,3cyclictest3792189-21apt-get19:30:131
3770914991915,2cyclictest4005280-21kworker/u8:3+events_unbound23:40:171
3770914991913,4cyclictest3833541-21latency_hist20:10:011
3770914991912,5cyclictest3942639-21latency_hist21:55:001
3770914991910,7cyclictest0-21swapper/122:07:271
3770910991915,3cyclictest3854173-21kworker/u8:1+flush-259:020:40:000
3770921991815,2cyclictest3854173-21kworker/u8:1+flush-259:020:40:213
3770921991815,2cyclictest3854173-21kworker/u8:1+events_unbound21:05:113
3770921991815,2cyclictest3854173-21kworker/u8:1+events_unbound21:00:213
3770921991814,3cyclictest3932519-21kworker/u8:5+events_unbound22:15:173
3770921991814,3cyclictest3854173-21kworker/u8:1+flush-259:021:55:203
3770921991814,3cyclictest3854173-21kworker/u8:1+events_unbound20:35:153
3770919991815,2cyclictest4005068-21kworker/u8:2+flush-259:022:55:162
3770919991815,2cyclictest3942663-21kworker/u8:0+events_unbound22:50:192
3770919991815,2cyclictest3942663-21kworker/u8:0+events_unbound22:25:232
3770919991814,3cyclictest4031569-21apt-get23:20:112
3770919991814,3cyclictest3854173-21kworker/u8:1+events_unbound21:20:232
3770914991815,2cyclictest4005068-21kworker/u8:2+events_unbound22:55:181
3770914991814,3cyclictest4068545-21kworker/u8:1+events_unbound00:35:161
3770914991814,3cyclictest4068545-21kworker/u8:1+events_unbound00:35:151
3770914991814,3cyclictest4010500-21apt-get23:00:121
3770910991815,2cyclictest3786310-21sh19:25:000
3770910991811,5cyclictest3925412-21ntpq21:35:210
3770921991714,2cyclictest4068545-21kworker/u8:1+events_unbound00:13:293
3770921991714,2cyclictest3776239-21kworker/u8:2+events_unbound19:30:143
3770921991713,3cyclictest3813017-21kworker/u8:2+events_unbound19:50:393
3770919991713,3cyclictest3854173-21kworker/u8:1+events_unbound22:10:282
3770919991712,4cyclictest3792020-21base6419:30:012
3770914991714,2cyclictest3900716-21kworker/u8:4+events_unbound21:35:201
3770914991714,2cyclictest3797741-21kworker/u8:0+events_unbound19:55:241
3770914991714,2cyclictest3786544-21kworker/u8:3+events_unbound20:25:241
3770914991714,2cyclictest1702837-21kworker/u8:4+flush-259:019:10:241
3770914991713,3cyclictest4026374-21kworker/u8:2+events_unbound23:30:181
3770914991713,3cyclictest3942663-21kworker/u8:0+events_unbound22:40:001
3770914991713,3cyclictest3932519-21kworker/u8:5+events_unbound22:30:261
3770914991713,3cyclictest3786544-21kworker/u8:3+flush-259:019:35:311
3770914991712,4cyclictest3791499-21latency_hist19:30:001
3770910991714,2cyclictest60650irq/133-enp0s29f1:rx-022:20:140
3770910991714,2cyclictest60650irq/133-enp0s29f1:rx-022:20:140
3770910991714,2cyclictest60650irq/133-enp0s29f1:rx-021:50:200
3770910991714,2cyclictest60650irq/133-enp0s29f1:rx-000:20:210
3770910991711,4cyclictest4046953-21kworker/u8:4+flush-259:023:50:220
3770910991710,5cyclictest4099395-21latency_hist00:30:010
377091099170,15cyclictest4046723-21turbostat23:35:010
3770921991612,3cyclictest3797741-21kworker/u8:0+events_unbound20:17:073
3770919991613,2cyclictest4005280-21kworker/u8:3+events_unbound00:10:212
3770919991613,1cyclictest0-21swapper/223:45:002
3770919991612,3cyclictest4005280-21kworker/u8:3+flush-259:023:00:252
3770919991612,2cyclictest3900716-21kworker/u8:4+flush-259:021:50:302
3770914991613,2cyclictest0-21swapper/120:15:001
3770914991612,3cyclictest4026828-21irqcore23:15:151
3770914991612,3cyclictest3786544-21kworker/u8:3+events_unbound21:17:391
3770914991612,3cyclictest3786544-21kworker/u8:3+events_unbound20:04:381
3770914991612,3cyclictest1117191-21systemd-journal20:55:381
3770914991611,4cyclictest3942663-21kworker/u8:0+flush-259:022:54:431
3770914991611,4cyclictest3932714-21diskstats21:45:131
3770914991611,4cyclictest3854173-21kworker/u8:1+events_unbound20:40:181
377091099169,5cyclictest3885478-21diskstats21:00:120
377091099168,7cyclictest0-21swapper/023:55:180
3770910991614,1cyclictest60650irq/133-enp0s29f1:rx-023:10:190
3770910991614,1cyclictest60650irq/133-enp0s29f1:rx-022:10:170
3770910991613,2cyclictest60650irq/133-enp0s29f1:rx-020:30:260
3770910991613,2cyclictest60650irq/133-enp0s29f1:rx-019:56:320
3770910991612,3cyclictest4005068-21kworker/u8:2+flush-259:022:55:160
3770910991612,2cyclictest4073694-21turbostat00:05:000
3770910991610,5cyclictest1117191-21systemd-journal22:50:000
3770910991610,4cyclictest1117191-21systemd-journal00:16:380
377091099160,14cyclictest862-21sshd22:15:380
3770921991511,3cyclictest3900716-21kworker/u8:4+flush-259:021:38:433
3770919991512,2cyclictest3951988-21ntp_kernel_err22:00:202
3770919991512,2cyclictest3843863-21kworker/u8:4+flush-259:020:20:262
3770919991511,3cyclictest3889829-21kworker/u8:0+events_unbound21:10:242
3770919991511,3cyclictest3813017-21kworker/u8:2+events_unbound19:50:202
3770919991511,3cyclictest1117191-21systemd-journal21:37:382
3770914991510,3cyclictest1117191-21systemd-journal23:54:381
377091099159,5cyclictest1117191-21systemd-journal23:50:000
3770910991513,1cyclictest60650irq/133-enp0s29f1:rx-023:40:250
3770910991513,1cyclictest60650irq/133-enp0s29f1:rx-023:35:160
3770910991513,1cyclictest60650irq/133-enp0s29f1:rx-023:20:150
3770910991513,1cyclictest60650irq/133-enp0s29f1:rx-022:35:150
3770910991513,1cyclictest60650irq/133-enp0s29f1:rx-021:15:230
3770910991513,1cyclictest60650irq/133-enp0s29f1:rx-019:35:250
3770910991512,2cyclictest60650irq/133-enp0s29f1:rx-023:00:160
3770910991512,2cyclictest60650irq/133-enp0s29f1:rx-022:03:570
3770910991512,2cyclictest60650irq/133-enp0s29f1:rx-021:05:440
3770910991512,2cyclictest60650irq/133-enp0s29f1:rx-020:15:100
3770910991512,2cyclictest60650irq/133-enp0s29f1:rx-019:25:240
3770910991512,2cyclictest3989504-21cpuspeed_turbos22:40:130
3770910991511,3cyclictest60650irq/133-enp0s29f1:rx-000:10:140
377091999148,4cyclictest509-21rs:main2
377091999147,5cyclictest3895362-21latency_hist21:10:012
3770919991411,2cyclictest4036445-21kworker/u8:0+flush-259:023:25:202
3770919991411,2cyclictest3932519-21kworker/u8:5+events_unbound21:55:232
3770919991411,2cyclictest3838912-21kworker/u8:1+events_unbound20:15:192
3770919991410,3cyclictest3932264-21gpgv21:45:002
3770919991410,3cyclictest3854173-21kworker/u8:1+flush-259:021:05:002
377091999140,8cyclictest3874764-21munin-run20:50:002
377091999140,13cyclictest4072834-21ntpq00:00:222
377091499149,3cyclictest4005068-21kworker/u8:2+flush-259:023:05:161
3770914991411,2cyclictest3854173-21kworker/u8:1+flush-259:022:10:291
3770914991410,3cyclictest3813017-21kworker/u8:2+flush-259:019:50:281
3770910991413,1cyclictest60650irq/133-enp0s29f1:rx-023:09:560
3770910991413,1cyclictest60650irq/133-enp0s29f1:rx-020:52:430
3770910991412,1cyclictest60650irq/133-enp0s29f1:rx-023:25:270
3770910991412,1cyclictest60650irq/133-enp0s29f1:rx-021:30:140
3770910991412,1cyclictest60650irq/133-enp0s29f1:rx-020:29:140
3770910991411,2cyclictest60650irq/133-enp0s29f1:rx-020:55:200
3770910991411,2cyclictest60650irq/133-enp0s29f1:rx-020:45:410
3770910991411,2cyclictest60650irq/133-enp0s29f1:rx-020:10:190
3770910991411,2cyclictest60650irq/133-enp0s29f1:rx-019:40:130
3770910991411,2cyclictest60650irq/133-enp0s29f1:rx-019:17:010
3770910991411,2cyclictest60650irq/133-enp0s29f1:rx-000:10:000
3770910991411,2cyclictest0-21swapper/021:15:000
377092199139,3cyclictest4005280-21kworker/u8:3+events_unbound23:11:243
3770921991310,2cyclictest4005280-21kworker/u8:3+events_unbound22:57:303
377091999139,3cyclictest4005280-21kworker/u8:3+flush-259:000:30:092
3770919991311,1cyclictest0-21swapper/219:30:002
3770919991310,2cyclictest4063284-21idleruntime-cro23:55:002
3770919991310,2cyclictest4026374-21kworker/u8:2+events_unbound23:30:282
377091499139,3cyclictest4084019-21kworker/u8:0+events_unbound00:20:201
377091499139,3cyclictest4068545-21kworker/u8:1+events_unbound00:05:231
377091499139,3cyclictest3854173-21kworker/u8:1+flush-259:021:55:391
377091499138,4cyclictest4084019-21kworker/u8:0+flush-259:000:15:411
377091499131,10cyclictest1117191-21systemd-journal21:25:001
3770914991310,2cyclictest3839208-21fschecks_time20:15:161
3770910991311,1cyclictest60650irq/133-enp0s29f1:rx-019:10:260
3770910991310,2cyclictest60650irq/133-enp0s29f1:rx-022:53:230
377091999129,2cyclictest4084019-21kworker/u8:0+flush-259:000:20:222
377091999129,2cyclictest3854491-21cpuspeed_turbos20:30:142
377091999128,3cyclictest3786544-21kworker/u8:3+flush-259:019:45:252
377091999127,4cyclictest4089139-21cron00:20:002
377091499129,2cyclictest0-21swapper/100:30:231
377091499128,3cyclictest3942663-21kworker/u8:0+flush-259:022:29:441
377091499128,3cyclictest3854173-21kworker/u8:1+flush-259:022:00:241
377091499128,2cyclictest0-21swapper/119:49:591
377091499126,4cyclictest4036420-21latency_hist23:25:021
377091499117,3cyclictest3854173-21kworker/u8:1+events_unbound21:30:231
377091499117,3cyclictest3797741-21kworker/u8:0+events_unbound19:40:291
377091999108,1cyclictest0-21swapper/221:30:222
377091999107,2cyclictest1702837-21kworker/u8:4+events_unbound19:10:192
377091999107,2cyclictest0-21swapper/222:40:162
37709149997,1cyclictest0-21swapper/123:50:011
37709149996,2cyclictest3870275-21irqcore20:45:151
37709199984,3cyclictest3820860-21cut19:55:202
37709149986,1cyclictest0-21swapper/120:30:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional