You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-21 - 21:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 21, 2024 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
159122241163,25sleep10-21swapper/107:08:471
156502240165,24sleep20-21swapper/207:05:262
158312238162,25sleep00-21swapper/007:07:450
140362234163,58sleep30-21swapper/307:05:033
126992600,1sleep10-21swapper/109:30:131
291872560,3sleep229191-21gltestperf08:50:152
37582500,2sleep20-21swapper/207:50:262
16110993827,2cyclictest9086-21kworker/2:010:04:172
1611099362,21cyclictest9086-21kworker/2:010:40:342
1611099351,21cyclictest9086-21kworker/2:012:20:182
1610599356,20cyclictest12615-21sshd08:10:311
1609899358,13cyclictest29951-21munin-run07:40:010
16110993430,3cyclictest0-21swapper/212:02:082
16110993323,3cyclictest30552-21kworker/2:208:33:392
16110993315,5cyclictest9086-21kworker/2:012:13:382
1611099322,19cyclictest9086-21kworker/2:010:47:282
16110993218,7cyclictest9086-21kworker/2:012:15:062
16110993020,2cyclictest30552-21kworker/2:208:25:152
16098993014,14cyclictest81ktimersoftd/007:25:010
1611099297,3cyclictest30-21rcuc/207:30:092
16110992913,3cyclictest9086-21kworker/2:010:56:512
16110992820,3cyclictest9086-21kworker/2:009:25:472
16110992818,2cyclictest9086-21kworker/2:010:27:152
16110992816,7cyclictest9086-21kworker/2:010:18:242
16110992816,7cyclictest9086-21kworker/2:010:18:242
16110992813,3cyclictest9086-21kworker/2:011:40:022
1611099281,11cyclictest9086-21kworker/2:009:19:452
1610599281,18cyclictest6954-21sshd09:14:381
1610599281,18cyclictest6954-21sshd09:14:381
1609899285,10cyclictest32370-21/usr/sbin/munin07:45:240
16110992717,2cyclictest30552-21kworker/2:209:08:432
16110992716,3cyclictest9086-21kworker/2:009:59:202
16110992714,7cyclictest9086-21kworker/2:009:41:392
16110992712,3cyclictest9086-21kworker/2:010:31:382
1611099270,19cyclictest9086-21kworker/2:011:18:422
1610599271,10cyclictest0-21swapper/112:30:101
1610599271,10cyclictest0-21swapper/112:30:101
1609899275,12cyclictest0-21swapper/009:47:150
1609899272,11cyclictest17881-21/usr/sbin/munin12:10:170
1611599261,6cyclictest0-21swapper/309:16:563
1611599261,3cyclictest14014-21irqstats08:15:193
1611099262,13cyclictest9086-21kworker/2:011:55:382
16110992616,3cyclictest9086-21kworker/2:011:02:592
16110992612,4cyclictest30552-21kworker/2:207:28:252
16110992611,3cyclictest9086-21kworker/2:011:50:082
16110992611,2cyclictest30552-21kworker/2:207:10:112
1610599261,17cyclictest8172-21switchtime11:45:251
1609899261,11cyclictest28587-21/usr/sbin/munin08:50:180
1611599253,9cyclictest0-21swapper/307:19:383
16110992518,2cyclictest9086-21kworker/2:012:06:122
16110992517,2cyclictest30552-21kworker/2:207:48:272
16110992516,2cyclictest9086-21kworker/2:012:34:042
16110992516,2cyclictest9086-21kworker/2:012:34:042
16110992514,3cyclictest9086-21kworker/2:012:28:042
16110992513,3cyclictest9086-21kworker/2:010:06:352
16110992511,3cyclictest9086-21kworker/2:009:20:152
16110992511,3cyclictest30552-21kworker/2:208:36:432
16110992510,3cyclictest9086-21kworker/2:010:23:122
16105992521,3cyclictest0-21swapper/107:35:011
16105992510,7cyclictest0-21swapper/108:45:371
1609899251,5cyclictest0-21swapper/010:06:230
16098992510,12cyclictest1648-21awk10:20:010
16098992510,12cyclictest1648-21awk10:20:010
1611599241,3cyclictest25422-21/usr/sbin/munin10:00:213
1611599241,11cyclictest8449-21/usr/sbin/munin10:35:163
1611099249,3cyclictest9086-21kworker/2:009:36:562
1611099249,3cyclictest30552-21kworker/2:207:15:192
16110992414,3cyclictest9086-21kworker/2:011:25:362
16110992414,3cyclictest9086-21kworker/2:011:25:362
16110992414,3cyclictest30552-21kworker/2:208:24:172
16110992413,2cyclictest9086-21kworker/2:012:37:162
16110992412,3cyclictest9086-21kworker/2:009:32:562
16110992410,3cyclictest9086-21kworker/2:010:38:422
1611099240,13cyclictest9086-21kworker/2:010:50:562
1610599248,7cyclictest0-21swapper/110:30:141
1610599245,9cyclictest8296-21ntp_states09:15:231
1610599245,7cyclictest0-21swapper/109:50:211
1610599241,14cyclictest0-21swapper/107:28:391
1609899248,3cyclictest2205-21/usr/sbin/munin07:50:180
1609899242,8cyclictest14634-21snmpd11:13:490
1609899242,10cyclictest21655-21/usr/sbin/munin11:05:140
1609899241,10cyclictest30781-21/usr/sbin/munin08:55:260
1609899241,10cyclictest30443-21/usr/sbin/munin11:25:240
1609899241,10cyclictest30443-21/usr/sbin/munin11:25:230
1611599239,3cyclictest8804-21/usr/sbin/munin08:05:143
1611599239,3cyclictest8804-21/usr/sbin/munin08:05:143
1611599237,3cyclictest15935-21expr12:05:123
1611599233,10cyclictest1864-21/usr/sbin/munin10:20:263
1611599231,6cyclictest10653-21/usr/sbin/munin10:40:123
1611599231,3cyclictest22324-21switchtime09:50:283
1611599231,11cyclictest4797-21latency_hist09:10:013
1611599231,11cyclictest19799-21/usr/sbin/munin08:30:263
1611599231,11cyclictest11264-21/usr/sbin/munin11:55:253
1611599231,10cyclictest9422-21/usr/sbin/munin09:20:173
1611099235,8cyclictest32-21ksoftirqd/208:04:292
1611099233,3cyclictest30552-21kworker/2:207:40:212
1611099232,10cyclictest30552-21kworker/2:208:16:472
16110992313,3cyclictest30552-21kworker/2:207:37:452
16110992311,2cyclictest30552-21kworker/2:207:55:152
1610599237,7cyclictest16946-21runrttasks10:54:111
1610599232,11cyclictest12008-21munin-run09:30:011
1610599231,19cyclictest8871-21fschecks_count10:35:131
1610599230,17cyclictest0-21swapper/109:58:341
1610599230,10cyclictest0-21swapper/107:39:071
299752220,4sleep230-21rcuc/210:10:122
1611599229,3cyclictest86850irq/129-enp1s0-12:36:013
1611599226,4cyclictest27797-21cat07:35:023
1611599226,4cyclictest0-21swapper/312:14:013
1611599222,10cyclictest14634-21snmpd08:58:223
1611099221,12cyclictest9086-21kworker/2:011:33:062
16110992210,3cyclictest30552-21kworker/2:208:14:332
16110992210,3cyclictest30552-21kworker/2:208:05:072
16110992210,3cyclictest30552-21kworker/2:208:05:072
16110992210,2cyclictest9086-21kworker/2:011:47:122
16110992210,2cyclictest9086-21kworker/2:011:44:322
1610599221,9cyclictest0-21swapper/112:16:181
16105992214,3cyclictest0-21swapper/111:00:501
16105992214,2cyclictest86850irq/129-enp1s0-11:50:231
1610599220,9cyclictest0-21swapper/110:26:451
1610599220,17cyclictest0-21swapper/112:09:461
1610599220,16cyclictest0-21swapper/109:35:151
1609899227,12cyclictest6046-21basename10:30:010
1609899226,14cyclictest0-21swapper/009:30:010
1609899224,10cyclictest779-21ntpd10:10:330
1609899222,5cyclictest0-21swapper/010:21:090
1609899221,12cyclictest0-21swapper/008:10:150
1609899220,6cyclictest0-21swapper/009:19:340
16162211,16sleep31774-21unixbench_multi11:30:263
1611599215,4cyclictest86850irq/129-enp1s0-11:20:153
1611599211,6cyclictest13462-21/usr/sbin/munin12:00:233
16115992116,3cyclictest23310-21ls09:55:123
16115992114,4cyclictest26519-21grep08:45:113
1611599211,14cyclictest9950-21kworker/3:209:43:383
1611599211,10cyclictest0-21swapper/311:36:243
1611099218,3cyclictest86750irq/128-enp1s0-08:45:512
1611099216,3cyclictest30552-21kworker/2:208:44:152
16110992112,2cyclictest9086-21kworker/2:011:21:382
1611099210,9cyclictest9-21rcu_preempt08:58:552
1610599217,3cyclictest848-21sssd_nss09:05:131
1610599216,3cyclictest20800-21munin-run09:50:011
1610599212,11cyclictest7862-21smart_sda08:00:271
16105992117,3cyclictest0-21swapper/107:49:021
1610599211,2cyclictest31366-21processes07:40:231
1610599211,15cyclictest0-21swapper/112:20:371
1610599211,15cyclictest0-21swapper/111:05:571
1610599211,13cyclictest86850irq/129-enp1s0-10:10:331
1610599211,11cyclictest0-21swapper/112:04:431
1610599211,11cyclictest0-21swapper/107:22:571
1610599210,10cyclictest0-21swapper/107:50:231
1609899216,4cyclictest0-21swapper/010:55:590
1609899213,4cyclictest0-21swapper/009:20:140
1609899212,5cyclictest4808-40dnf09:10:010
1609899212,11cyclictest8048-21kworker/u8:010:51:400
1609899211,6cyclictest0-21swapper/011:19:350
1609899211,14cyclictest0-21swapper/012:00:030
1609899211,11cyclictest0-21swapper/011:00:230
1609899211,10cyclictest0-21swapper/009:40:220
1609899210,11cyclictest0-21swapper/011:39:070
65392200,4sleep341-21ksoftirqd/310:30:123
323912201,3sleep10-21swapper/108:55:331
1611599203,7cyclictest30443-21/usr/sbin/munin11:25:253
1611599203,7cyclictest30443-21/usr/sbin/munin11:25:253
1611599201,9cyclictest27626-21/usr/sbin/munin10:05:243
1611599201,7cyclictest0-21swapper/312:25:213
1611599201,7cyclictest0-21swapper/307:50:123
1611599201,6cyclictest19893-21ntp_states09:45:213
1611599201,3cyclictest26673-21/usr/sbin/munin12:30:163
1611599201,3cyclictest26673-21/usr/sbin/munin12:30:163
16115992013,4cyclictest25048-21grep07:25:263
1611599201,16cyclictest9222-21grep11:50:193
16115992010,2cyclictest17559-21smart_sda07:10:233
1611099208,3cyclictest30552-21kworker/2:209:12:152
1611099208,3cyclictest30552-21kworker/2:209:12:152
1611099208,2cyclictest9086-21kworker/2:009:54:492
1611099205,9cyclictest9086-21kworker/2:009:48:452
1611099203,3cyclictest30552-21kworker/2:207:20:032
1610599202,7cyclictest86850irq/129-enp1s0-11:20:191
1610599201,7cyclictest16088-21cyclictest08:15:251
1610599201,4cyclictest848-21sssd_nss12:10:261
16105992013,3cyclictest0-21swapper/111:44:351
16105992013,3cyclictest0-21swapper/111:31:481
1610599201,18cyclictest0-21swapper/111:35:141
1610599201,17cyclictest0-21swapper/108:50:041
1610599201,17cyclictest0-21swapper/108:35:151
1610599201,14cyclictest0-21swapper/111:16:161
1610599201,14cyclictest0-21swapper/111:10:531
1610599201,11cyclictest12241-21users10:40:281
1610599201,10cyclictest16480-21ntp_states08:20:251
1610599200,2cyclictest86850irq/129-enp1s0-11:56:201
1610599200,18cyclictest0-21swapper/110:20:241
1610599200,18cyclictest0-21swapper/107:10:211
1609899207,4cyclictest0-21swapper/012:07:500
1609899206,5cyclictest4672-21/usr/sbin/munin11:40:110
1609899205,5cyclictest0-21swapper/008:41:380
1609899204,7cyclictest4794-21diskmemload12:25:140
16098992012,6cyclictest7-21ksoftirqd/007:35:010
1609899201,12cyclictest0-21swapper/008:47:340
1609899201,11cyclictest0-21swapper/009:50:260
1609899201,10cyclictest19799-21/usr/sbin/munin08:30:250
1609899201,10cyclictest0-21swapper/012:36:270
1609899201,10cyclictest0-21swapper/009:14:200
1609899201,10cyclictest0-21swapper/009:14:200
1609899201,10cyclictest0-21swapper/009:00:380
1609899200,3cyclictest0-21swapper/011:52:510
233942191,2sleep30-21swapper/311:07:143
220032190,3sleep30-21swapper/312:19:403
1611599195,3cyclictest86850irq/129-enp1s0-08:00:253
1611599192,5cyclictest15045-21/usr/sbin/munin10:50:143
1611599191,9cyclictest1845-21gunzip09:00:223
1611599191,9cyclictest15404-21/usr/sbin/munin08:20:223
1611599191,7cyclictest0-21swapper/311:40:213
1611599191,6cyclictest7785-21memory11:45:203
1611599191,6cyclictest22378-21tr12:20:133
1611599191,5cyclictest12575-21df_inode09:30:113
1611599191,10cyclictest29828-21/usr/sbin/munin10:10:133
1611599191,10cyclictest26773-21idleruntime11:15:173
1611099199,2cyclictest86750irq/128-enp1s0-11:05:072
1611099197,3cyclictest9086-21kworker/2:011:10:102
1611099192,8cyclictest30552-21kworker/2:209:03:212
1610599191,9cyclictest0-21swapper/108:05:131
1610599191,9cyclictest0-21swapper/108:05:131
1610599191,8cyclictest0-21swapper/107:15:171
16105991916,2cyclictest0-21swapper/110:45:371
16105991916,2cyclictest0-21swapper/110:06:301
16105991915,3cyclictest0-21swapper/112:27:141
16105991915,3cyclictest0-21swapper/110:00:541
16105991915,3cyclictest0-21swapper/108:32:161
1610599191,4cyclictest5212-21irqstats07:55:171
1610599191,17cyclictest0-21swapper/112:35:051
1610599191,17cyclictest0-21swapper/111:26:111
1610599191,17cyclictest0-21swapper/111:26:111
1610599191,17cyclictest0-21swapper/110:16:231
1610599191,17cyclictest0-21swapper/110:16:231
1610599191,17cyclictest0-21swapper/109:00:481
1610599191,17cyclictest0-21swapper/108:41:111
1610599191,16cyclictest0-21swapper/109:44:101
1610599191,10cyclictest19213-21latency_hist11:00:021
1609899199,3cyclictest0-21swapper/010:30:240
1609899198,3cyclictest30174-21/usr/sbin/munin07:40:240
1609899196,3cyclictest0-21swapper/010:02:330
1609899194,9cyclictest0-21swapper/007:26:250
1609899191,9cyclictest13851-21sshd09:30:380
1609899191,8cyclictest8449-21/usr/sbin/munin10:35:180
16098991915,2cyclictest7-21ksoftirqd/010:50:010
16098991914,2cyclictest7-21ksoftirqd/009:40:010
16098991913,3cyclictest20677-21grep12:15:170
1609899191,10cyclictest0-21swapper/012:23:380
1609899190,4cyclictest25798-21kworker/u8:011:30:480
107532191,2sleep00-21swapper/008:09:520
107532191,2sleep00-21swapper/008:09:520
299592181,2sleep00-21swapper/011:21:480
253612180,3sleep3401ktimersoftd/308:40:213
191402181,2sleep30-21swapper/310:59:113
183932180,14sleep00-21swapper/008:25:180
1611599187,3cyclictest21994-21/usr/sbin/munin08:35:133
1611599185,3cyclictest86850irq/129-enp1s0-09:35:223
1611599182,7cyclictest0-21swapper/310:45:203
1611599181,7cyclictest0-21swapper/311:11:113
1611599181,6cyclictest0-21swapper/307:55:183
1611599181,5cyclictest4794-21diskmemload09:27:053
1611599181,5cyclictest21387-21/usr/sbin/munin07:20:233
1611599181,3cyclictest0-21swapper/310:25:123
16115991812,3cyclictest32453-21cat07:45:123
1611599181,11cyclictest41-21ksoftirqd/309:13:543
1611599181,11cyclictest41-21ksoftirqd/309:13:543
1611599181,10cyclictest0-21swapper/310:15:253
1611599181,10cyclictest0-21swapper/310:15:253
16105991811,4cyclictest836-21runrttasks09:20:051
1609899187,8cyclictest27289-21gltestperf12:30:200
1609899187,8cyclictest27289-21gltestperf12:30:200
1609899182,2cyclictest32366-21kworker/0:208:21:270
1609899181,6cyclictest0-21swapper/009:57:010
1609899181,4cyclictest8582-21perf08:05:010
16098991812,3cyclictest15140-21seq08:19:470
1609899181,16cyclictest0-21swapper/010:40:270
85752171,2sleep04794-21diskmemload11:46:450
1611599171,3cyclictest0-21swapper/307:37:313
16115991711,3cyclictest11189-21cstates08:10:133
1611599171,10cyclictest0-21swapper/311:00:113
16105991714,2cyclictest0-21swapper/108:25:421
16098991713,2cyclictest7-21ksoftirqd/007:20:000
1609899171,2cyclictest0-21swapper/007:55:380
16098991711,3cyclictest12004-21if_enp1s011:55:160
1611599169,4cyclictest17824-21ls08:25:123
1611599168,5cyclictest848-21sssd_nss08:55:013
1609899165,3cyclictest86550irq/126-enp1s0-08:35:210
16098991610,3cyclictest17564-21swap07:10:230
16115991510,3cyclictest31245-21ntp_states07:40:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional