You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-02 - 04:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Wed Apr 02, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
612282994336,5cyclictest892421-21latency_hist00:25:011
612287994036,3cyclictest724186-21kworker/u8:4+flush-259:021:35:152
612287994035,3cyclictest876810-21kworker/u8:2+events_unbound00:15:012
612287993834,3cyclictest659233-21kworker/u8:4+flush-259:020:20:242
612287993629,5cyclictest638202-21latency_hist19:35:012
612282993630,4cyclictest805367-21latency_hist22:40:021
612282993532,2cyclictest676268-21kworker/u8:3+flush-259:020:15:261
612287993330,2cyclictest794529-21kworker/u8:0+flush-259:022:45:002
612282993328,3cyclictest893707-21kworker/u8:4+flush-259:000:30:181
612279993327,4cyclictest729170-21latency_hist21:10:010
612287993229,2cyclictest848942-21kworker/u8:3+flush-259:023:50:002
612287993229,2cyclictest848942-21kworker/u8:3+flush-259:023:49:592
612287993228,3cyclictest724186-21kworker/u8:4+flush-259:021:40:242
612287993127,3cyclictest848942-21kworker/u8:3+events_unbound23:45:002
612287993127,3cyclictest843096-21kworker/u8:0+events_unbound23:35:242
612287993127,3cyclictest811215-21kworker/u8:1+events_unbound23:00:222
612287993127,3cyclictest638436-21kworker/u8:0+flush-259:019:35:252
612287993125,4cyclictest3000860-21systemd-journal20:09:232
612279993124,5cyclictest697961-21diskstats20:40:150
612289993023,5cyclictest843717-21latency_hist23:30:003
612287993027,2cyclictest617019-21kworker/u8:2+events_unbound19:45:132
612287993026,3cyclictest573914-21kworker/u8:1+events_unbound19:40:182
612287993025,4cyclictest676268-21kworker/u8:3+flush-259:020:20:002
612279993023,5cyclictest893055-21latency_hist00:30:000
612287992927,1cyclictest617019-21kworker/u8:2+flush-259:019:15:232
612287992924,3cyclictest794529-21kworker/u8:0+events_unbound22:30:142
612287992824,3cyclictest860009-21kworker/u8:0+flush-259:000:20:212
612282992824,3cyclictest628054-21kworker/u8:4+flush-259:019:25:151
612287992724,2cyclictest686702-21kworker/u8:2+flush-259:020:55:242
612282992724,2cyclictest860009-21kworker/u8:0+events_unbound00:10:141
612282992724,2cyclictest686702-21kworker/u8:2+events_unbound21:20:161
612279992722,4cyclictest795172-21/usr/sbin/munin22:30:110
612287992623,2cyclictest876810-21kworker/u8:2+flush-259:000:30:262
612287992623,2cyclictest617019-21kworker/u8:2+flush-259:019:55:202
612287992622,3cyclictest848941-21kworker/u8:1+events_unbound00:15:262
612287992620,2cyclictest659233-21kworker/u8:4+events_unbound20:40:192
612279992623,2cyclictest767408-21apt-get21:55:140
612287992522,2cyclictest848941-21kworker/u8:1+flush-259:000:00:192
612287992522,2cyclictest827658-21kworker/u8:4+flush-259:023:10:152
612287992522,2cyclictest724186-21kworker/u8:4+flush-259:021:50:102
612287992522,2cyclictest617019-21kworker/u8:2+flush-259:019:20:202
612287992521,3cyclictest676268-21kworker/u8:3+flush-259:020:30:202
612282992521,3cyclictest767168-21kworker/u8:0+flush-259:022:15:011
612289992419,3cyclictest848941-21kworker/u8:1+events_unbound00:15:163
612287992421,2cyclictest761335-21kworker/u8:1+flush-259:022:15:232
612287992420,3cyclictest794959-21kworker/u8:2+flush-259:023:20:212
612287992420,2cyclictest724186-21kworker/u8:4+events_unbound22:35:162
612282992421,2cyclictest783844-21kworker/u8:5+flush-259:022:20:001
612282992421,2cyclictest670470-21kworker/u8:2+flush-259:020:10:161
612282992420,3cyclictest708233-21apt-get20:50:181
612282992420,3cyclictest686702-21kworker/u8:2+events_unbound21:15:151
612289992319,3cyclictest723527-21apt-get21:04:593
612287992320,2cyclictest848941-21kworker/u8:1+flush-259:023:55:252
612287992320,2cyclictest751001-21kworker/u8:2+events_unbound21:55:192
612282992318,4cyclictest865441-21/usr/sbin/munin23:55:151
612282992318,4cyclictest3000860-21systemd-journal19:19:231
612279992320,2cyclictest745175-21kworker/u8:1+events_unbound21:30:010
612279992320,2cyclictest745175-21kworker/u8:1+events_unbound21:30:000
612279992319,3cyclictest772986-21kworker/u8:3+flush-259:022:00:140
612289992218,3cyclictest713393-21apt-get20:55:123
612287992220,2cyclictest772986-21kworker/u8:3+events_unbound22:00:202
612287992219,2cyclictest617019-21kworker/u8:2+events_unbound19:25:192
612287992219,2cyclictest607040-21kworker/u8:0+events_unbound19:10:262
612287992218,3cyclictest734389-21kworker/u8:0+events_unbound21:25:242
612287992218,3cyclictest734389-21kworker/u8:0+events_unbound21:25:242
612282992218,3cyclictest573914-21kworker/u8:1+events_unbound20:20:131
612282992217,4cyclictest756405-21/usr/sbin/munin21:40:151
612279992218,3cyclictest843096-21kworker/u8:0+events_unbound23:30:200
612289992118,2cyclictest686702-21kworker/u8:2+events_unbound20:45:263
612289992118,2cyclictest573914-21kworker/u8:1+events_unbound19:45:243
612289992117,3cyclictest686702-21kworker/u8:2+flush-259:021:10:293
612287992118,2cyclictest686702-21kworker/u8:2+flush-259:020:50:302
612287992118,2cyclictest670470-21kworker/u8:2+flush-259:020:10:212
612287992117,3cyclictest723560-21kworker/u8:1+events_unbound21:06:252
612282992118,2cyclictest827658-21kworker/u8:4+flush-259:023:55:001
612282992117,3cyclictest648847-21/usr/sbin/munin19:45:201
612279992119,1cyclictest0-21swapper/023:09:590
612289992013,6cyclictest875733-21apt-get00:05:023
612287992017,2cyclictest794529-21kworker/u8:0+flush-259:022:50:162
612287992016,3cyclictest724186-21kworker/u8:4+events_unbound22:49:232
612287992016,3cyclictest724186-21kworker/u8:4+events_unbound22:22:262
612287992014,4cyclictest3000860-21systemd-journal19:54:232
612282992017,3cyclictest676268-21kworker/u8:3+events_unbound20:30:111
612282992017,2cyclictest761335-21kworker/u8:1+flush-259:022:30:001
612282992017,2cyclictest734389-21kworker/u8:0+events_unbound21:30:151
612282992016,3cyclictest794959-21kworker/u8:2+events_unbound23:20:241
612282992016,3cyclictest723560-21kworker/u8:1+flush-259:021:05:211
612282992016,3cyclictest638436-21kworker/u8:0+events_unbound19:35:231
612279992016,3cyclictest676268-21kworker/u8:3+events_unbound21:05:000
612279992015,3cyclictest876378-21rred00:05:040
612289991915,3cyclictest794959-21kworker/u8:2+flush-259:022:40:173
612289991913,5cyclictest893717-21munin-node00:30:123
612287991915,3cyclictest827658-21kworker/u8:4+events_unbound23:32:252
612287991914,4cyclictest669500-21gpgv20:05:002
612279991916,2cyclictest648838-21kworker/u8:3+flush-259:019:50:280
612289991815,2cyclictest648838-21kworker/u8:3+events_unbound19:55:013
612289991814,3cyclictest622875-21apt-get19:20:133
612289991812,4cyclictest3000860-21systemd-journal21:33:233
612287991815,2cyclictest827658-21kworker/u8:4+flush-259:023:29:592
612287991815,2cyclictest686702-21kworker/u8:2+events_unbound21:00:162
612287991812,4cyclictest0-21swapper/222:59:592
612282991815,2cyclictest827658-21kworker/u8:4+events_unbound23:40:141
612282991815,2cyclictest724186-21kworker/u8:4+flush-259:021:35:281
612282991814,3cyclictest860009-21kworker/u8:0+flush-259:000:15:311
612279991814,3cyclictest3000061-21in:imuxsock20:49:590
612289991715,1cyclictest794529-21kworker/u8:0+events_unbound23:00:143
612289991713,3cyclictest573914-21kworker/u8:1+flush-259:020:00:193
612289991712,3cyclictest772331-21apt-get22:00:003
612287991714,2cyclictest893707-21kworker/u8:4+flush-259:000:35:212
612287991714,2cyclictest676268-21kworker/u8:3+flush-259:020:35:212
612287991713,3cyclictest686702-21kworker/u8:2+events_unbound21:19:552
612282991714,2cyclictest573914-21kworker/u8:1+events_unbound19:30:261
612282991711,5cyclictest3000860-21systemd-journal21:00:001
612279991715,2cyclictest617019-21kworker/u8:2+events_unbound19:45:140
612279991714,2cyclictest827658-21kworker/u8:4+events_unbound23:47:240
612279991714,2cyclictest827658-21kworker/u8:4+events_unbound23:47:230
612279991714,2cyclictest794959-21kworker/u8:2+events_unbound23:20:260
612279991713,3cyclictest767144-21latency_hist21:55:000
612279991713,3cyclictest691460-21ntpq20:30:170
612289991610,4cyclictest740069-21base6421:20:003
612287991613,2cyclictest686702-21kworker/u8:2+events_unbound21:21:572
612287991613,2cyclictest659233-21kworker/u8:4+flush-259:020:45:212
612287991611,3cyclictest827658-21kworker/u8:4+events_unbound23:16:032
612282991613,2cyclictest3000063-21rs:main1
612279991614,1cyclictest60650irq/133-enp0s29f1:rx-020:54:110
612279991613,2cyclictest761335-21kworker/u8:1+events_unbound22:15:190
612279991613,2cyclictest756174-21turbostat21:40:000
612279991613,2cyclictest676268-21kworker/u8:3+events_unbound20:20:010
612279991613,2cyclictest60650irq/133-enp0s29f1:rx-021:16:290
612279991613,2cyclictest60650irq/133-enp0s29f1:rx-020:28:430
612279991612,3cyclictest659233-21kworker/u8:4+events_unbound20:20:170
612279991612,3cyclictest492-21dbus-daemon21:35:000
612279991610,5cyclictest627828-21latency_hist19:25:010
612279991610,4cyclictest669390-21latency_hist20:05:010
612289991512,2cyclictest573914-21kworker/u8:1+flush-259:020:15:173
612289991511,3cyclictest893051-21apt-get00:30:003
612289991511,3cyclictest724186-21kworker/u8:4+flush-259:022:10:133
612287991512,2cyclictest686702-21kworker/u8:2+events_unbound21:10:152
612287991511,3cyclictest848942-21kworker/u8:3+events_unbound23:50:202
612287991511,3cyclictest794529-21kworker/u8:0+events_unbound22:25:472
612287991511,3cyclictest724186-21kworker/u8:4+events_unbound21:30:432
612282991512,2cyclictest893707-21kworker/u8:4+flush-259:000:35:161
612282991512,2cyclictest851521-21kthreadcore23:35:201
612282991512,2cyclictest843096-21kworker/u8:0+events_unbound23:30:121
612282991512,2cyclictest724186-21kworker/u8:4+flush-259:022:20:161
612282991510,4cyclictest3000860-21systemd-journal22:44:221
612279991513,2cyclictest848941-21kworker/u8:1+flush-259:000:00:260
612279991513,1cyclictest60650irq/133-enp0s29f1:rx-023:55:110
612279991513,1cyclictest60650irq/133-enp0s29f1:rx-023:40:230
612279991513,1cyclictest60650irq/133-enp0s29f1:rx-023:35:210
612279991513,1cyclictest60650irq/133-enp0s29f1:rx-023:13:240
612279991513,1cyclictest60650irq/133-enp0s29f1:rx-022:40:390
612279991513,1cyclictest60650irq/133-enp0s29f1:rx-021:14:150
612279991513,1cyclictest60650irq/133-enp0s29f1:rx-019:15:250
612279991512,2cyclictest724186-21kworker/u8:4+flush-259:021:40:220
612279991512,2cyclictest724186-21kworker/u8:4+events_unbound22:20:150
612279991512,2cyclictest638436-21kworker/u8:0+events_unbound19:35:210
612279991512,2cyclictest617019-21kworker/u8:2+flush-259:019:55:190
612279991512,2cyclictest61450irq/134-enp0s29f1:tx-022:55:260
612279991512,2cyclictest60650irq/133-enp0s29f1:rx-023:53:460
612279991512,2cyclictest60650irq/133-enp0s29f1:rx-022:11:480
612279991512,2cyclictest0-21swapper/022:30:000
612289991411,2cyclictest849364-21fschecks_count23:35:173
612289991410,3cyclictest881525-21unixbench_multi00:10:213
61228799149,3cyclictest724186-21kworker/u8:4+events_unbound21:47:092
612287991410,3cyclictest761335-21kworker/u8:1+events_unbound22:10:262
612287991410,3cyclictest3000061-21in:imuxsock00:30:002
612287991410,2cyclictest0-21swapper/222:10:002
612282991410,3cyclictest848942-21kworker/u8:3+flush-259:023:46:291
612282991410,3cyclictest848942-21kworker/u8:3+flush-259:023:46:281
612282991410,3cyclictest724186-21kworker/u8:4+events_unbound21:50:151
612279991412,2cyclictest60650irq/133-enp0s29f1:rx-020:07:410
612279991412,1cyclictest60650irq/133-enp0s29f1:rx-023:29:230
612279991412,1cyclictest60650irq/133-enp0s29f1:rx-022:50:130
612279991412,1cyclictest60650irq/133-enp0s29f1:rx-021:20:240
612279991412,1cyclictest60650irq/133-enp0s29f1:rx-020:35:240
612279991412,1cyclictest60650irq/133-enp0s29f1:rx-020:14:000
612279991411,2cyclictest811215-21kworker/u8:1+flush-259:023:15:160
612279991410,3cyclictest893618-21apt-key00:30:020
61228999139,2cyclictest0-21swapper/323:10:003
61228999137,4cyclictest832658-21awk23:14:593
612289991310,2cyclictest875255-21unixbench_multi00:00:263
612289991310,2cyclictest839192-21kthreadcore23:20:213
612289991310,2cyclictest817066-21irqcore22:55:183
612289991310,2cyclictest687503-21cpuspeed_turbos20:30:103
61228799139,3cyclictest860009-21kworker/u8:0+events_unbound00:08:222
61228799139,3cyclictest811215-21kworker/u8:1+events_unbound23:05:222
61228799139,3cyclictest676268-21kworker/u8:3+events_unbound20:27:502
612282991311,2cyclictest686702-21kworker/u8:2+flush-259:020:35:191
612282991310,2cyclictest827884-21fschecks_count23:10:101
612282991310,2cyclictest724186-21kworker/u8:4+events_unbound21:45:231
612282991310,2cyclictest617019-21kworker/u8:2+flush-259:019:20:151
612279991312,1cyclictest60650irq/133-enp0s29f1:rx-023:02:190
612279991312,1cyclictest60650irq/133-enp0s29f1:rx-019:14:130
612279991312,1cyclictest60650irq/133-enp0s29f1:rx-000:14:330
612279991311,2cyclictest60650irq/133-enp0s29f1:rx-019:32:340
612279991311,1cyclictest60650irq/133-enp0s29f1:rx-019:42:190
612279991310,2cyclictest60650irq/133-enp0s29f1:rx-022:36:450
61228999129,2cyclictest751465-21fschecks_time21:35:163
61228999128,3cyclictest617019-21kworker/u8:2+flush-259:019:55:253
61228999128,3cyclictest573914-21kworker/u8:1+flush-259:019:30:283
61228999126,5cyclictest3000860-21systemd-journal21:19:593
61228799127,3cyclictest612196-21latency_hist19:10:012
61228799127,3cyclictest612196-21latency_hist19:10:002
61228299129,3cyclictest811215-21kworker/u8:1+events_unbound23:15:171
61228299129,2cyclictest648838-21kworker/u8:3+flush-259:019:54:281
61228299128,3cyclictest686702-21kworker/u8:2+flush-259:021:10:261
61228299127,4cyclictest3000860-21systemd-journal22:33:231
612282991210,2cyclictest659233-21kworker/u8:4+events_unbound20:40:251
61227999129,2cyclictest60650irq/133-enp0s29f1:rx-021:47:510
61227999129,2cyclictest60650irq/133-enp0s29f1:rx-000:24:220
612279991210,2cyclictest60650irq/133-enp0s29f1:rx-022:48:590
612279991210,1cyclictest60650irq/133-enp0s29f1:rx-022:07:390
61228999119,1cyclictest612274-21cyclictest22:00:243
61228999117,3cyclictest708299-21cpuspeed_turbos20:50:203
61228999116,4cyclictest800217-21cat22:35:003
61228999110,4cyclictest686457-21cron20:25:003
61228999110,10cyclictest860002-21awk23:50:003
61228999110,10cyclictest860002-21awk23:50:003
61228299117,3cyclictest848941-21kworker/u8:1+flush-259:000:00:041
61228299117,3cyclictest772986-21kworker/u8:3+flush-259:022:00:151
61228299116,4cyclictest3000860-21systemd-journal23:29:591
61228299115,4cyclictest516-21gdbus20:49:591
61227999119,1cyclictest60650irq/133-enp0s29f1:rx-020:55:160
61227999118,2cyclictest60650irq/133-enp0s29f1:rx-000:19:390
61228999107,2cyclictest800828-21irqcore22:35:163
61228999107,2cyclictest616920-21unixbench_multi19:10:263
61228999106,3cyclictest491ktimers/319:15:123
61228999105,3cyclictest888310-21sed00:20:173
61228999105,3cyclictest3000860-21systemd-journal23:54:593
61228999105,3cyclictest3000860-21systemd-journal21:30:013
61228999105,3cyclictest3000860-21systemd-journal21:30:003
61228999104,4cyclictest833077-21df23:15:133
61228999104,4cyclictest3000860-21systemd-journal19:44:553
61228299106,3cyclictest676268-21kworker/u8:3+flush-259:020:28:291
61228299106,3cyclictest573914-21kworker/u8:1+flush-259:019:40:261
61228299101,2cyclictest875748-21http00:05:021
6122899996,2cyclictest0-21swapper/319:25:173
6122899995,3cyclictest3000860-21systemd-journal21:45:003
6122899995,2cyclictest793454-21ntpq22:20:223
6122899994,3cyclictest696662-21ntpq20:35:213
6122899990,8cyclictest794307-21turbostat22:29:563
6122899990,2cyclictest767144-21latency_hist21:55:003
6122899990,2cyclictest687116-21latency_hist20:30:013
6122829995,3cyclictest811215-21kworker/u8:1+events_unbound23:06:241
6122829995,3cyclictest761335-21kworker/u8:1+events_unbound22:07:241
6122829994,3cyclictest3000061-21in:imuxsock00:26:221
6122899985,3cyclictest491ktimers/323:40:103
6122899984,3cyclictest811627-21cut22:50:113
6122899984,3cyclictest727429-21tr21:05:193
6122899984,3cyclictest491ktimers/319:35:133
6122899984,3cyclictest491ktimers/300:35:143
6122899984,2cyclictest481rcuc/322:15:163
6122899983,3cyclictest761952-21sh21:50:013
6122899980,6cyclictest811185-21awk22:50:003
6122899980,3cyclictest835-21runrttasks20:05:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional