You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-11 - 23:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Apr 11, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39572240162,24sleep10-21swapper/107:07:111
38672237164,24sleep30-21swapper/307:05:593
40702234163,58sleep20-21swapper/207:08:362
39992222146,24sleep00-21swapper/007:07:410
515421660,4sleep0427199cyclictest08:20:270
2869721510,4sleep0427199cyclictest09:15:190
4283997774,2cyclictest0-21swapper/208:55:152
6152630,4sleep325305-21diskmemload09:30:113
262242610,1sleep00-21swapper/009:10:220
287872550,2sleep30-21swapper/312:15:183
4283994643,2cyclictest0-21swapper/211:51:542
4283994643,2cyclictest0-21swapper/211:51:542
4283994643,2cyclictest0-21swapper/209:26:422
427199387,13cyclictest20900-21munin-run09:00:010
427199376,10cyclictest6755-21awk09:45:010
427199376,10cyclictest6755-21awk09:45:010
427199369,11cyclictest8245-21perf11:20:010
427199368,13cyclictest14577-21munin-run10:10:010
427199335,13cyclictest23983-21idleruntime-cro10:35:010
427699320,23cyclictest0-21swapper/111:30:101
158152320,4sleep1427699cyclictest10:10:151
427199313,12cyclictest7736-21awk08:30:010
427199311,5cyclictest22857-21/usr/sbin/munin12:00:170
427199311,11cyclictest14544-21/usr/sbin/munin08:45:160
428899301,19cyclictest0-21swapper/309:25:133
4288993010,12cyclictest0-21swapper/309:48:113
427199301,9cyclictest3785-21ntp_states12:35:220
427199290,12cyclictest25321-21kworker/0:012:15:180
428899282,17cyclictest0-21swapper/312:38:333
428899281,18cyclictest0-21swapper/312:31:583
428899281,18cyclictest0-21swapper/311:54:593
428899281,18cyclictest0-21swapper/311:54:593
428899281,18cyclictest0-21swapper/310:01:213
428899281,17cyclictest0-21swapper/310:56:513
427699284,18cyclictest11696-21kworker/u8:110:01:491
4276992824,3cyclictest0-21swapper/112:01:101
4276992824,3cyclictest0-21swapper/111:36:301
427699281,24cyclictest25912-21strings07:55:201
427699281,24cyclictest13478-21pmu-power08:40:231
427699281,21cyclictest0-21swapper/109:06:361
427699281,21cyclictest0-21swapper/108:07:531
427199281,13cyclictest18241-21munin-run11:50:010
427199281,12cyclictest6964-21kworker/0:109:59:340
427199280,11cyclictest21709-21kworker/0:310:35:180
428899271,18cyclictest0-21swapper/311:24:083
428899271,18cyclictest0-21swapper/309:56:303
428899271,16cyclictest0-21swapper/310:22:543
4276992723,3cyclictest0-21swapper/112:12:081
427699272,11cyclictest22673-21hddtemp_smartct10:30:121
427699271,23cyclictest14708-21tr08:45:151
427699271,23cyclictest11516-21turbostat.cron07:25:001
427699271,20cyclictest0-21swapper/108:56:431
427699271,20cyclictest0-21swapper/108:22:281
427699271,20cyclictest0-21swapper/107:50:451
427699271,20cyclictest0-21swapper/107:31:291
427199270,2cyclictest29471-21kworker/0:009:20:170
428899263,16cyclictest86550irq/126-enp1s0-11:11:533
4288992614,4cyclictest0-21swapper/311:33:033
428899261,19cyclictest0-21swapper/310:49:433
428899261,18cyclictest0-21swapper/312:11:213
428899260,19cyclictest0-21swapper/310:41:433
428399268,5cyclictest0-21swapper/209:50:092
427699263,5cyclictest30962-21sshd10:52:251
4276992624,1cyclictest0-21swapper/111:44:491
4276992623,2cyclictest0-21swapper/112:39:431
4276992623,2cyclictest0-21swapper/112:20:121
4276992623,2cyclictest0-21swapper/112:09:121
427699262,13cyclictest6474-21munin-node07:15:011
427699261,23cyclictest0-21swapper/109:35:221
427699261,22cyclictest32505-21crond09:30:001
427699261,20cyclictest0-21swapper/109:45:581
427699261,11cyclictest21783-21sed09:00:131
427199269,2cyclictest10147-21kworker/0:211:26:140
427199268,15cyclictest17632-21munin-run11:45:020
427199262,13cyclictest14318-21munin-run08:45:020
427199261,13cyclictest10147-21kworker/0:211:30:220
427199261,12cyclictest18960-21idleruntime-cro10:20:010
427199261,12cyclictest11538-21basename07:25:010
428899259,9cyclictest0-21swapper/312:21:353
428899259,7cyclictest0-21swapper/311:02:383
428899257,11cyclictest0-21swapper/310:39:343
428899254,2cyclictest86550irq/126-enp1s0-10:10:113
428899251,18cyclictest0-21swapper/311:57:393
428899251,17cyclictest0-21swapper/311:38:493
428899251,17cyclictest0-21swapper/310:05:383
428899251,17cyclictest0-21swapper/309:16:543
428899251,16cyclictest0-21swapper/311:25:063
428899251,16cyclictest0-21swapper/310:17:413
4276992522,2cyclictest0-21swapper/111:12:171
427699252,11cyclictest1200-21idleruntime12:30:121
427699251,19cyclictest0-21swapper/109:32:391
427699251,19cyclictest0-21swapper/109:14:031
427699251,19cyclictest0-21swapper/107:38:361
427699251,15cyclictest0-21swapper/107:45:021
4276992510,3cyclictest28074-21df_inode09:15:131
4276992510,3cyclictest17939-21sed10:15:211
427699250,20cyclictest0-21swapper/109:41:161
427699250,20cyclictest0-21swapper/109:41:161
427199254,3cyclictest86750irq/128-enp1s0-12:30:590
427199252,12cyclictest13944-21kworker/0:107:35:290
186722251,3sleep118685-21diskstats07:40:141
428899248,9cyclictest0-21swapper/311:42:533
4288992414,5cyclictest0-21swapper/309:51:573
428899241,17cyclictest0-21swapper/312:07:483
428899241,17cyclictest0-21swapper/310:50:473
428899241,17cyclictest0-21swapper/309:20:083
428899241,15cyclictest0-21swapper/309:40:043
428899241,15cyclictest0-21swapper/309:40:043
427699241,6cyclictest10692-21kworker/u8:211:26:291
427699241,3cyclictest13574-21chrt07:28:211
427699241,3cyclictest13574-21chrt07:28:211
4276992413,6cyclictest0-21swapper/110:24:151
4276992412,7cyclictest0-21swapper/109:59:561
427699241,20cyclictest8252-21sh09:50:121
427699241,20cyclictest7984-21cpu08:30:151
427699241,20cyclictest16741-21/usr/sbin/munin08:50:221
427699241,20cyclictest0-21swapper/110:35:181
427699241,15cyclictest0-21swapper/108:15:531
427699241,11cyclictest32504-21needreboot08:10:231
427199249,3cyclictest18130-21kworker/0:011:50:120
427199249,3cyclictest18130-21kworker/0:011:50:120
427199248,3cyclictest25321-21kworker/0:012:10:130
427199247,2cyclictest86750irq/128-enp1s0-11:11:200
427199243,11cyclictest6964-21kworker/0:110:00:180
427199241,16cyclictest6964-21kworker/0:110:10:080
428899238,9cyclictest0-21swapper/311:49:013
428899231,17cyclictest0-21swapper/312:03:133
428899231,17cyclictest0-21swapper/309:36:023
428899231,16cyclictest86550irq/126-enp1s0-08:40:153
427699234,16cyclictest7324-21ntp_states11:15:211
427699232,9cyclictest32470-21ntp_states10:55:221
427699231,16cyclictest0-21swapper/108:04:321
427699231,15cyclictest0-21swapper/110:29:521
427699231,10cyclictest8548-21wc11:20:131
427699231,10cyclictest18331-21date11:50:011
427699231,10cyclictest0-21swapper/111:05:021
427199232,9cyclictest29471-21kworker/0:009:27:090
427199232,10cyclictest11440-21kworker/0:207:33:190
428899225,5cyclictest780-21rngd08:18:493
4288992219,2cyclictest0-21swapper/309:11:493
4288992218,3cyclictest0-21swapper/307:45:223
428899221,16cyclictest0-21swapper/310:33:313
428899221,16cyclictest0-21swapper/310:26:463
428899221,15cyclictest0-21swapper/312:26:373
428899221,15cyclictest0-21swapper/311:18:583
427699228,3cyclictest32593-21seq12:29:241
427699227,3cyclictest6103-21diskstats08:25:141
427699221,9cyclictest0-21swapper/109:20:181
4276992215,2cyclictest0-21swapper/110:05:101
4276992212,2cyclictest0-21swapper/107:16:341
427699221,19cyclictest0-21swapper/112:18:411
427699221,19cyclictest0-21swapper/110:45:071
427699221,18cyclictest0-21swapper/111:55:251
427699221,10cyclictest20334-21chrt11:53:521
427699221,10cyclictest20334-21chrt11:53:521
427199224,3cyclictest87950irq/132-enp2s0-11:21:380
427199221,9cyclictest285862grep08:00:260
427199221,13cyclictest751-21kworker/0:111:06:590
134162220,5sleep2311ktimersoftd/210:00:242
55162211,17sleep3581-21systemd-journal08:25:003
428899213,5cyclictest6125-21sort08:25:153
428899212,11cyclictest23334-21/usr/sbin/munin09:05:243
428899212,11cyclictest11749-21/usr/sbin/munin07:25:173
428899212,11cyclictest11749-21/usr/sbin/munin07:25:173
428899212,10cyclictest27117-21/usr/sbin/munin08:00:113
428899211,7cyclictest20895-21latency_hist09:00:023
428899211,15cyclictest0-21swapper/311:07:053
428899211,13cyclictest86550irq/126-enp1s0-07:15:163
428899211,11cyclictest17820-21sshd07:36:253
428899210,14cyclictest86550irq/126-enp1s0-08:30:263
428399215,4cyclictest0-21swapper/212:21:402
428399214,8cyclictest780-21rngd07:25:222
428399214,8cyclictest780-21rngd07:25:222
428399214,7cyclictest985-21grep11:00:012
428399213,7cyclictest2867-21unixbench_multi08:15:262
428399211,7cyclictest20944-21runrttasks10:22:222
428399211,4cyclictest0-21swapper/210:27:502
427699211,12cyclictest0-21swapper/110:41:451
427199217,3cyclictest2270-21kworker/0:209:39:560
427199217,2cyclictest20242-21kworker/0:312:06:460
427199215,6cyclictest88050irq/133-enp2s0-10:59:100
427199213,8cyclictest31232-21kworker/0:108:11:400
427199211,2cyclictest15895-21kworker/0:307:40:180
427199211,13cyclictest25839-21kworker/0:110:50:200
427199211,11cyclictest15895-21kworker/0:307:56:060
295592210,5sleep339-21rcuc/308:05:123
225642210,3sleep081ktimersoftd/009:00:250
428899207,11cyclictest86550irq/126-enp1s0-07:30:253
4288992017,2cyclictest17314-21kworker/u8:007:14:343
4288992012,5cyclictest23603-21/usr/sbin/munin07:50:213
428899201,11cyclictest16741-21/usr/sbin/munin08:50:253
4283992014,3cyclictest1100-21dump-pmu-power08:15:002
4276992016,3cyclictest0-21swapper/108:36:141
4276992016,3cyclictest0-21swapper/108:36:141
427699201,12cyclictest0-21swapper/111:05:281
427199206,2cyclictest29471-21kworker/0:009:30:150
427199205,3cyclictest13015-21kworker/0:011:35:220
427199201,10cyclictest25839-21kworker/0:110:42:000
428899196,3cyclictest12050irq/123-ahci[0008:45:193
428899191,11cyclictest30954-21kworker/u8:208:36:303
428899191,11cyclictest30954-21kworker/u8:208:36:303
428399197,4cyclictest23127-21ls10:30:202
428399193,13cyclictest8979-21grep11:20:162
428399191,8cyclictest31181-21munin-run10:55:012
428399191,7cyclictest28543-21irqstats09:15:172
428399191,7cyclictest0-21swapper/209:38:572
4283991913,3cyclictest1155-21gltestperf12:30:112
427199194,9cyclictest751-21kworker/0:111:01:310
427199194,2cyclictest86750irq/128-enp1s0-09:47:520
427199194,2cyclictest86650irq/127-enp1s0-08:09:430
427199194,2cyclictest31966-21kworker/0:112:27:440
427199193,2cyclictest86750irq/128-enp1s0-07:52:240
427199193,2cyclictest86750irq/128-enp1s0-07:12:100
427199192,4cyclictest15895-21kworker/0:307:46:170
427199192,3cyclictest0-21swapper/008:50:290
4271991915,3cyclictest28340-21kworker/u8:010:48:560
427199190,9cyclictest3968-21kworker/0:007:27:290
427199190,9cyclictest3968-21kworker/0:007:27:290
428899183,4cyclictest19680-21timerandwakeup07:40:253
4288991813,2cyclictest31530-21/usr/sbin/munin08:10:183
428399183,8cyclictest0-21swapper/209:30:262
428399182,3cyclictest17733-21users07:35:272
428399182,3cyclictest0-21swapper/207:16:122
428399182,15cyclictest0-21swapper/208:42:522
428399181,7cyclictest0-21swapper/212:15:122
428399181,7cyclictest0-21swapper/210:14:382
428399181,6cyclictest25305-21diskmemload10:35:212
428399181,6cyclictest0-21swapper/210:47:182
428399181,5cyclictest0-21swapper/208:32:542
4283991812,3cyclictest21212-21ls09:00:092
428399181,10cyclictest0-21swapper/209:55:492
427199185,4cyclictest20242-21kworker/0:311:58:580
427199182,8cyclictest6964-21kworker/0:110:26:380
427199181,4cyclictest9144-21kworker/0:107:19:500
4271991813,3cyclictest9191-21sendmail_mailqu08:30:280
120562181,2sleep20-21swapper/208:39:252
120562181,2sleep20-21swapper/208:39:252
428899171,3cyclictest0-21swapper/309:01:543
428899171,3cyclictest0-21swapper/307:20:103
428899171,13cyclictest25271-21diskstats07:55:153
428399176,3cyclictest86550irq/126-enp1s0-09:10:182
428399174,4cyclictest6823-21cat09:45:012
428399174,4cyclictest6823-21cat09:45:012
428399173,8cyclictest88150irq/134-enp2s0-07:50:282
428399172,8cyclictest86750irq/128-enp1s0-11:10:312
428399172,2cyclictest0-21swapper/207:20:382
428399171,7cyclictest18922-21gltestperf07:40:162
428399171,7cyclictest14572-21latency_hist10:10:012
428399171,7cyclictest0-21swapper/207:10:172
428399171,6cyclictest18963-21crond10:20:012
428399171,6cyclictest15370-21uname08:45:212
428399171,5cyclictest0-21swapper/212:00:022
428399171,4cyclictest2101-21sed11:00:192
428399171,4cyclictest0-21swapper/211:35:352
428399171,3cyclictest0-21swapper/208:28:312
4283991713,3cyclictest0-21swapper/211:57:482
4283991713,3cyclictest0-21swapper/208:01:282
4283991712,3cyclictest836-21runrttasks11:49:012
4283991712,3cyclictest3889-21df08:20:142
4283991712,3cyclictest29932-21/usr/sbin/munin09:20:252
4283991712,3cyclictest29326-21/usr/sbin/munin08:05:172
428399171,14cyclictest0-21swapper/211:25:242
4283991711,3cyclictest13230-21latency_hist11:35:012
428399171,10cyclictest0-21swapper/212:26:052
427199174,2cyclictest86750irq/128-enp1s0-10:20:100
427199171,3cyclictest31232-21kworker/0:108:18:240
4271991711,3cyclictest24110-21if_err_enp1s009:05:160
427199171,13cyclictest10151-21/usr/sbin/munin08:35:190
427199171,13cyclictest10151-21/usr/sbin/munin08:35:180
250642171,3sleep20-21swapper/212:05:142
428399164,2cyclictest28340-21kworker/u8:011:06:402
428399162,7cyclictest0-21swapper/208:52:232
428399161,9cyclictest0-21swapper/211:40:162
428399161,7cyclictest0-21swapper/209:46:522
428399161,6cyclictest0-21swapper/212:38:382
428399161,6cyclictest0-21swapper/210:40:162
428399161,4cyclictest14207-1kworker/u9:007:46:192
4283991611,3cyclictest25707-21cpu12:10:112
4283991611,3cyclictest23537-21expr09:05:122
4283991610,3cyclictest25102-21cat07:55:132
428399160,15cyclictest0-21swapper/211:15:392
427199164,2cyclictest86750irq/128-enp1s0-12:20:350
427199161,12cyclictest7576-21/usr/sbin/munin09:50:140
139382160,2sleep20-21swapper/207:30:082
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional