You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-25 - 14:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Nov 25, 2024 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113412239163,25sleep30-21swapper/307:09:033
112852239164,24sleep10-21swapper/107:08:201
94592202165,25sleep20-21swapper/207:05:112
112632202163,25sleep00-21swapper/007:08:030
11518998279,2cyclictest0-21swapper/210:37:342
11518996258,3cyclictest0-21swapper/209:56:242
117892610,1sleep10-21swapper/112:05:161
171762520,4sleep017173-21crond08:35:010
144052500,1sleep00-21swapper/012:10:250
11506994912,20cyclictest15618-21idleruntime-cro09:45:010
1151299441,31cyclictest23942-21taskset12:33:421
1150699444,21cyclictest12013-21missed_timers12:05:190
1151299433,26cyclictest780-21rngd07:14:071
1151299421,32cyclictest0-21swapper/112:22:031
1151299421,32cyclictest0-21swapper/112:22:031
1151299421,31cyclictest780-21rngd08:47:471
1151299411,30cyclictest0-21swapper/107:32:491
1151299411,11cyclictest1475-21cat10:30:011
1151299393,23cyclictest0-21swapper/109:08:571
1151299393,23cyclictest0-21swapper/109:08:571
11512993927,6cyclictest0-21swapper/109:35:131
1151299391,30cyclictest0-21swapper/112:37:351
1151299391,28cyclictest0-21swapper/109:40:441
1151299391,16cyclictest0-21swapper/109:30:221
1151299380,24cyclictest0-21swapper/108:32:011
1150699383,6cyclictest28061-21/usr/sbin/munin11:30:250
1150699381,26cyclictest25033-21kworker/0:208:09:050
11506993812,13cyclictest24054-21munin-run12:35:010
1151899372,7cyclictest17406-21/usr/sbin/munin08:35:172
1151899371,6cyclictest0-21swapper/211:55:082
11512993721,5cyclictest0-21swapper/107:58:571
1151299371,27cyclictest9232-21kworker/u8:009:51:271
1151299371,27cyclictest780-21rngd11:47:111
1150699377,14cyclictest25014-21munin-run10:10:010
1150699372,24cyclictest21769-21cat11:15:120
1150699372,24cyclictest21769-21cat11:15:120
1150699371,25cyclictest27441-21/usr/sbin/munin10:15:150
1151899365,5cyclictest21249-21awk11:15:012
1151899361,6cyclictest0-21swapper/210:26:402
1151299362,21cyclictest780-21rngd07:25:371
1151299361,3cyclictest22724-21sensors_temp11:15:231
1151299361,3cyclictest22724-21sensors_temp11:15:231
1151299361,32cyclictest28167-21awk09:00:001
1150699361,6cyclictest16353-21ls12:15:190
1150699361,23cyclictest26902-21idleruntime08:55:160
1151899351,5cyclictest0-21swapper/210:21:402
1151299356,3cyclictest14632-21turbostat11:00:001
11512993521,4cyclictest0-21swapper/111:55:081
1151299351,8cyclictest18833-21sshd11:07:271
1151299350,3cyclictest0-21swapper/110:52:031
1150699358,12cyclictest20622-21perf10:00:000
1150699356,14cyclictest14984-21munin-run08:30:010
11506993515,2cyclictest2107-21kworker/0:109:16:490
1151899341,5cyclictest11769-21df09:35:132
1151299349,4cyclictest780-21rngd08:27:371
1151299349,4cyclictest780-21rngd07:22:571
1151299347,5cyclictest0-21swapper/111:00:381
1151299342,4cyclictest27832-21munin-run11:30:011
11512993413,3cyclictest25629-21sh11:25:011
1151299341,17cyclictest0-21swapper/110:03:511
1151299341,16cyclictest0-21swapper/109:14:521
1151299340,15cyclictest0-21swapper/110:40:401
1150699341,25cyclictest0-21swapper/012:00:200
1151899331,5cyclictest0-21swapper/211:05:202
1151899331,4cyclictest792-21avahi-daemon10:40:542
1151299339,3cyclictest25094-21sh08:50:351
1151299337,3cyclictest86750irq/128-enp1s0-12:00:141
1151299332,11cyclictest4689-21irqstats10:35:181
1151299331,4cyclictest31323-21ntp_states11:35:221
11512993312,3cyclictest23555-21cut07:35:161
11512993310,3cyclictest18639-21/usr/sbin/munin09:55:121
11512993310,3cyclictest10197-21users08:15:261
1151299330,2cyclictest86650irq/127-enp1s0-08:03:431
1151299330,12cyclictest0-21swapper/110:30:501
1150699332,24cyclictest22835-21kworker/0:007:35:310
1150699331,4cyclictest9573-21df09:30:160
1150699331,21cyclictest30254-21/usr/sbin/munin11:35:160
1151899321,17cyclictest0-21swapper/209:10:122
1151299324,3cyclictest86850irq/129-enp1s0-08:35:271
1151299323,18cyclictest0-21swapper/108:20:551
1151299322,20cyclictest780-21rngd07:16:441
1151299321,3cyclictest5441-21gunzip08:05:231
1151299321,14cyclictest16537-21kworker/u8:209:00:141
1151299320,6cyclictest0-21swapper/109:23:201
1151299320,13cyclictest0-21swapper/111:32:341
1150699327,12cyclictest3649-21munin-run10:35:010
11506993220,5cyclictest14391-21kworker/0:111:00:230
11506993214,15cyclictest1461-21ntpq09:10:210
1150699321,20cyclictest32172-21/usr/sbin/munin07:55:170
1150699321,20cyclictest25632-21sh11:25:010
1151899311,6cyclictest0-21swapper/212:06:232
1151899311,11cyclictest0-21swapper/210:52:012
1151299319,4cyclictest9111-21/usr/sbin/munin10:45:171
1151299313,19cyclictest0-21swapper/109:17:301
1151299312,20cyclictest86750irq/128-enp1s0-11:40:241
1151299311,9cyclictest24882-21sshd10:08:281
11512993113,3cyclictest13974-21hddtemp_smartct12:10:191
11512993112,4cyclictest26512-21/usr/sbin/munin10:10:271
11512993112,4cyclictest26512-21/usr/sbin/munin10:10:271
11512993112,3cyclictest7463-21ntp_kernel_pll_08:10:201
1151299311,20cyclictest0-21swapper/107:46:511
1151299311,10cyclictest0-21swapper/110:15:511
11512993110,3cyclictest19282-21acpi11:10:131
1151299310,3cyclictest0-21swapper/111:54:511
1151299310,14cyclictest0-21swapper/112:28:051
1151299310,10cyclictest0-21swapper/108:40:151
1150699319,19cyclictest32733-21cat10:25:180
1150699318,20cyclictest17440-21sed07:20:150
1150699318,20cyclictest12278-21cat08:20:340
1150699316,21cyclictest24385-21cpuspeed_turbos12:35:130
1150699316,17cyclictest88150irq/134-enp2s0-08:35:370
1150699312,3cyclictest86750irq/128-enp1s0-10:35:130
1150699312,3cyclictest2129-21cat11:45:010
1150699312,10cyclictest6239-21kworker/0:111:53:310
1151899301,4cyclictest28394-21/usr/sbin/munin09:00:182
1151899301,21cyclictest0-21swapper/208:18:082
1151899301,11cyclictest0-21swapper/212:34:192
1151899301,10cyclictest0-21swapper/211:42:112
1151299303,7cyclictest0-21swapper/107:50:231
1150699308,19cyclictest11786-21cat07:10:110
1150699304,2cyclictest86550irq/126-enp1s0-07:50:260
1150699302,4cyclictest30246-40dnf09:03:290
11506993020,2cyclictest27292-21kworker/0:107:42:110
11506993018,7cyclictest23655-21kworker/0:009:08:110
11506993018,7cyclictest23655-21kworker/0:009:08:110
11506993013,14cyclictest11431-21/usr/sbin/munin09:35:190
1150699301,26cyclictest19275-21/usr/sbin/munin11:10:220
1150699301,19cyclictest836-21runrttasks11:26:380
1151899291,2cyclictest32-21ksoftirqd/207:58:582
1151899291,10cyclictest0-21swapper/207:28:282
1151299299,3cyclictest8643-21sshd09:25:291
11512992919,3cyclictest0-21swapper/112:17:161
1151299290,12cyclictest0-21swapper/110:22:321
1150699299,7cyclictest16526-21kworker/0:107:32:350
1150699298,18cyclictest7039-21/usr/sbin/munin09:25:250
1150699297,19cyclictest3800-21sed11:45:250
1150699296,20cyclictest581-21systemd-journal08:02:280
1150699295,21cyclictest848-21sssd_nss09:20:270
1150699295,21cyclictest0-21swapper/010:00:150
1150699295,20cyclictest28911-21pmu-power07:45:240
1150699292,4cyclictest19938-21df08:40:130
1150699292,4cyclictest13347-21/usr/sbin/munin10:55:160
1150699292,19cyclictest21438-21kworker/0:108:46:030
1150699291,4cyclictest6753-21/usr/sbin/munin08:10:130
1150699291,3cyclictest14272-21smart_sda07:15:030
11506992913,13cyclictest11139-21expr10:50:200
1150699291,20cyclictest19648-21kworker/0:012:25:190
1150699290,17cyclictest24748-21kworker/0:110:11:210
1150699290,17cyclictest24748-21kworker/0:110:11:210
1150699290,17cyclictest0-21swapper/011:05:200
1151899287,3cyclictest86650irq/127-enp1s0-10:18:042
1151899281,7cyclictest0-21swapper/210:08:262
1151899281,5cyclictest0-21swapper/210:32:592
1151899281,11cyclictest0-21swapper/209:54:402
11512992810,3cyclictest25935-21dnf07:40:141
1151299280,20cyclictest0-21swapper/109:47:451
1150699288,17cyclictest836-21runrttasks09:46:120
1150699282,5cyclictest7753-21ntp_states11:55:210
1150699282,4cyclictest24334-21df08:50:110
1150699282,4cyclictest17702-21smart_sda09:50:210
1150699282,3cyclictest18029-21df12:20:140
1150699282,3cyclictest18029-21df12:20:140
1150699281,5cyclictest6072-21/usr/sbin/munin10:40:160
1150699281,3cyclictest9171-21/usr/sbin/munin10:45:180
1150699281,3cyclictest20145-21/usr/sbin/munin07:25:220
1150699281,19cyclictest30977-21/usr/sbin/munin10:20:250
1151899278,6cyclictest0-21swapper/212:00:332
1151899272,10cyclictest0-21swapper/208:47:442
1151899271,4cyclictest0-21swapper/211:37:452
1151899271,22cyclictest0-21swapper/211:16:192
1151899271,22cyclictest0-21swapper/211:16:192
1151899271,11cyclictest0-21swapper/208:11:322
1151899270,9cyclictest30-21rcuc/207:33:402
1150699276,18cyclictest8612-21/usr/sbin/munin08:15:150
1151899262,10cyclictest780-21rngd07:54:342
1151899261,19cyclictest0-21swapper/212:13:392
1151899261,10cyclictest0-21swapper/212:22:392
1151899261,10cyclictest0-21swapper/212:22:392
1151899261,10cyclictest0-21swapper/212:16:172
1151899261,10cyclictest0-21swapper/211:00:332
1151899260,11cyclictest2862-21kworker/2:007:48:022
1152299251,6cyclictest0-21swapper/312:10:243
1152299251,10cyclictest0-21swapper/308:56:173
1151899252,9cyclictest0-21swapper/208:57:042
1151899251,13cyclictest0-21swapper/207:35:182
1151899251,12cyclictest0-21swapper/211:28:252
1151899251,10cyclictest32-21ksoftirqd/208:00:042
1151899250,13cyclictest16835-21kworker/2:212:35:232
1152299241,9cyclictest24006-21df_inode11:20:153
1151899241,12cyclictest0-21swapper/209:49:382
1151899241,11cyclictest86850irq/129-enp1s0-10:12:322
1151899241,11cyclictest86850irq/129-enp1s0-10:12:322
1151899241,11cyclictest0-21swapper/210:01:382
1151899241,11cyclictest0-21swapper/209:19:322
1152299231,4cyclictest0-21swapper/312:33:033
1152299231,16cyclictest0-21swapper/310:24:593
1152299230,16cyclictest0-21swapper/311:10:103
1151899238,3cyclictest0-21swapper/208:30:162
1151899233,2cyclictest86850irq/129-enp1s0-07:11:452
1151899231,8cyclictest0-21swapper/209:08:362
1151899231,8cyclictest0-21swapper/209:08:362
1151899231,13cyclictest0-21swapper/208:29:322
1151899231,12cyclictest0-21swapper/210:48:272
1151899231,10cyclictest0-21swapper/209:27:042
1152299227,3cyclictest0-21swapper/309:58:163
1152299224,7cyclictest0-21swapper/310:08:583
1152299221,6cyclictest0-21swapper/312:27:293
1152299221,6cyclictest0-21swapper/309:32:453
1152299221,5cyclictest0-21swapper/312:24:023
1152299221,5cyclictest0-21swapper/312:24:023
1152299221,14cyclictest0-21swapper/310:36:283
1151899222,13cyclictest32-21ksoftirqd/207:23:442
1151899222,10cyclictest32-21ksoftirqd/208:20:102
1151899222,10cyclictest0-21swapper/209:33:462
1151899221,9cyclictest32-21ksoftirqd/207:44:042
1151899221,9cyclictest0-21swapper/211:45:452
1151899221,4cyclictest0-21swapper/212:29:002
1151899221,4cyclictest0-21swapper/211:24:222
1151899221,4cyclictest0-21swapper/209:20:322
1151899221,11cyclictest0-21swapper/207:15:312
1151899221,10cyclictest86650irq/127-enp1s0-11:30:112
1151899221,10cyclictest0-21swapper/208:05:222
11518992210,3cyclictest0-21swapper/209:40:202
1152299213,5cyclictest16790-40dnf11:04:323
1152299213,16cyclictest401ktimersoftd/307:10:183
1152299211,4cyclictest0-21swapper/311:57:423
1152299211,4cyclictest0-21swapper/310:55:373
1152299211,3cyclictest0-21swapper/312:19:573
11522992112,2cyclictest32583-21awk09:10:013
11522992112,2cyclictest32583-21awk09:10:013
1152299211,10cyclictest5422-21cut09:20:143
1152299210,9cyclictest0-21swapper/309:18:513
11518992113,3cyclictest5390-21latency11:50:212
1152299209,3cyclictest28061-21/usr/sbin/munin11:30:293
1152299209,3cyclictest14595-21/usr/sbin/munin07:15:123
1152299208,4cyclictest0-21swapper/310:01:123
1152299207,7cyclictest86550irq/126-enp1s0-12:04:243
1152299207,6cyclictest0-21swapper/311:39:553
1152299206,4cyclictest0-21swapper/310:16:013
1152299205,6cyclictest0-21swapper/309:28:243
1152299205,4cyclictest0-21swapper/309:10:183
1152299205,3cyclictest0-21swapper/312:38:103
1152299202,10cyclictest6419-21/usr/sbin/munin08:10:193
1152299201,9cyclictest6511-21cat11:55:023
1152299201,7cyclictest26314-21ntp_states10:10:243
1152299201,7cyclictest26314-21ntp_states10:10:243
11522992014,3cyclictest19372-21dump-pmu-power08:40:013
1152299201,3cyclictest0-21swapper/309:52:513
1152299201,2cyclictest0-21swapper/307:40:253
1152299201,11cyclictest14699-21ntp_states09:40:203
1152299201,11cyclictest11431-21/usr/sbin/munin09:35:223
1152299201,11cyclictest0-21swapper/311:28:593
1152299201,11cyclictest0-21swapper/310:49:483
1152299201,11cyclictest0-21swapper/310:30:223
1152299201,11cyclictest0-21swapper/310:28:053
1152299201,10cyclictest0-21swapper/311:08:193
1152299201,10cyclictest0-21swapper/309:46:263
1152299201,10cyclictest0-21swapper/308:45:453
1152299200,7cyclictest0-21swapper/310:42:383
1151899206,3cyclictest86550irq/126-enp1s0-08:50:202
1151899201,10cyclictest0-21swapper/210:58:092
1151899201,10cyclictest0-21swapper/208:44:202
1152299196,4cyclictest21424-21cstates07:30:143
1152299196,3cyclictest0-21swapper/311:48:363
1152299192,14cyclictest19609-21tail07:25:153
1152299191,3cyclictest17177-21grep07:20:143
1152299191,10cyclictest32575-21diskmemload11:40:323
1152299191,10cyclictest31202-21gunzip07:50:233
1152299191,10cyclictest22726-21smart_sda11:15:243
1152299191,10cyclictest22726-21smart_sda11:15:243
1152299191,10cyclictest0-21swapper/312:05:053
1152299191,10cyclictest0-21swapper/310:53:593
1152299181,7cyclictest1305-21grep07:55:293
1152299181,5cyclictest842-21lldpd07:47:443
11522991813,3cyclictest14975-21idleruntime-cro08:30:013
1152299181,15cyclictest0-21swapper/308:31:243
1152299181,10cyclictest2010-21/usr/sbin/munin08:00:183
1152299181,10cyclictest10807-21/usr/sbin/munin08:20:163
1152299181,10cyclictest0-21swapper/308:15:143
1152299171,5cyclictest0-21swapper/307:35:193
1152299171,15cyclictest0-21swapper/309:02:543
1152299164,4cyclictest19604-21/usr/sbin/munin08:40:133
11522991611,3cyclictest24896-21cat08:50:173
11522991510,3cyclictest4221-21/usr/sbin/munin08:05:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional