You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-07 - 11:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot6s.osadl.org (updated Mon Apr 07, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20895212096,19sleep00-21swapper/019:06:240
210072117106,6sleep30-21swapper/319:07:503
210372116105,6sleep20-21swapper/219:08:132
21073211397,11sleep10-21swapper/119:08:411
155292860,1sleep00-21swapper/023:41:060
227112820,0sleep10-21swapper/121:29:151
136552800,1sleep00-21swapper/021:18:160
313672770,0sleep00-21swapper/023:23:450
158142730,0sleep20-21swapper/223:41:322
205042710,0sleep00-21swapper/021:25:540
160252640,1sleep10-21swapper/121:20:171
21204993817,19cyclictest0-21swapper/222:40:092
306912290,0sleep10-21swapper/120:50:281
41082220,0sleep20-21swapper/200:05:012
2120499220,4cyclictest0-21swapper/222:22:042
2120399221,19cyclictest23-21ksoftirqd/122:10:131
2120499212,18cyclictest0-21swapper/221:09:052
21204992117,2cyclictest0-21swapper/219:10:112
2120499210,4cyclictest0-21swapper/220:00:052
2120499210,4cyclictest0-21swapper/220:00:052
297632200,0sleep30-21swapper/321:35:273
2120499200,18cyclictest0-21swapper/221:55:112
21203992017,2cyclictest0-21swapper/100:10:071
21202992016,2cyclictest0-21swapper/019:10:110
2120299200,18cyclictest0-21swapper/019:35:120
2120599190,13cyclictest0-21swapper/323:44:043
2120499193,2cyclictest4965-21diskmemload21:12:032
21204991916,2cyclictest7875-21kworker/u8:100:12:052
21204991916,2cyclictest1921-21kworker/u8:000:37:052
21204991916,2cyclictest16942-21kworker/u8:022:53:052
2120499190,3cyclictest0-21swapper/222:57:032
2120299190,2cyclictest0-21swapper/020:40:120
21205991815,2cyclictest29476-21kworker/u8:123:33:033
21205991815,2cyclictest27451-21kworker/u8:019:26:053
2120499180,4cyclictest0-21swapper/222:00:052
2120499180,3cyclictest0-21swapper/223:26:052
2120499180,2cyclictest0-21swapper/200:30:072
21205991714,2cyclictest7875-21kworker/u8:100:20:043
21204991714,2cyclictest31058-21kworker/u8:121:42:042
21204991714,2cyclictest20414-21kworker/u8:223:57:032
21204991714,2cyclictest17873-21kworker/u8:123:15:052
21204991714,2cyclictest13312-21kworker/u8:020:30:042
2120499170,2cyclictest0-21swapper/200:23:032
2120499170,16cyclictest0-21swapper/222:05:042
21205991613,2cyclictest1921-21kworker/u8:000:32:023
2120599160,2cyclictest0-21swapper/322:22:043
2120599160,2cyclictest0-21swapper/320:00:053
2120599160,2cyclictest0-21swapper/320:00:043
21204991613,2cyclictest31073-21kworker/u8:120:10:052
21204991613,2cyclictest24509-21kworker/u8:122:30:022
21204991613,2cyclictest19425-21kworker/u8:120:38:052
21204991613,2cyclictest19021-21kworker/u8:119:21:052
21204991612,2cyclictest0-21swapper/223:22:522
2120499160,2cyclictest0-21swapper/200:17:052
2120499160,11cyclictest0-21swapper/200:04:052
2120399161,13cyclictest19621-21ssh22:35:211
2120399160,3cyclictest0-21swapper/119:40:241
2120399160,14cyclictest0-21swapper/121:12:101
2120299161,13cyclictest0-21swapper/021:14:010
21205991512,2cyclictest20058-21kworker/u8:221:31:053
21205991512,2cyclictest1921-21kworker/u8:000:39:053
21205991512,2cyclictest16942-21kworker/u8:023:03:053
21205991512,2cyclictest16942-21kworker/u8:022:54:043
2120599150,2cyclictest0-21swapper/321:12:033
2120599150,11cyclictest0-21swapper/321:52:033
21204991512,2cyclictest9099-21kworker/u8:121:26:052
21204991512,2cyclictest5331-21kworker/u8:019:58:042
21204991512,2cyclictest31073-21kworker/u8:119:51:042
21204991512,2cyclictest24509-21kworker/u8:122:28:032
21204991512,2cyclictest20058-21kworker/u8:221:36:052
21204991512,2cyclictest19425-21kworker/u8:120:59:052
21204991512,2cyclictest19425-21kworker/u8:120:45:042
21204991512,2cyclictest19359-21apt-get20:25:112
21204991512,2cyclictest19021-21kworker/u8:119:16:042
21204991512,2cyclictest15453-21kworker/u8:220:17:032
21204991512,2cyclictest0-21swapper/223:34:112
21204991511,2cyclictest23529-21apt-get21:30:122
21204991510,3cyclictest12471-21ssh23:38:562
2120499150,3cyclictest3379-21tr19:45:162
2120499150,3cyclictest0-21swapper/222:47:352
2120499150,13cyclictest4965-21diskmemload23:48:222
2120499150,13cyclictest13961-21apt-get23:05:062
2120499150,13cyclictest0-21swapper/223:00:112
2120499150,10cyclictest0-21swapper/223:50:052
2120399155,7cyclictest3594-21Xorg20:27:591
2120399152,11cyclictest21-21rcuc/119:25:201
2120399151,12cyclictest0-21swapper/120:05:061
2120399151,12cyclictest0-21swapper/100:00:391
21203991510,3cyclictest20561-21sh23:12:141
21203991510,3cyclictest0-21swapper/123:00:351
21203991510,3cyclictest0-21swapper/121:30:131
2120399150,13cyclictest0-21swapper/122:51:391
21202991510,3cyclictest0-21swapper/000:27:140
2120299150,3cyclictest1411-21grep22:15:250
276222140,2sleep00-21swapper/022:10:040
21205991411,2cyclictest8768-21kworker/u8:223:36:043
21205991411,2cyclictest31073-21kworker/u8:120:06:053
21205991411,2cyclictest27451-21kworker/u8:019:32:053
21205991411,2cyclictest20414-21kworker/u8:200:08:033
21205991411,2cyclictest16774-21kworker/u8:023:55:043
2120599140,3cyclictest0-21swapper/319:43:033
2120599140,2cyclictest19234-21ssh22:00:053
2120599140,2cyclictest0-21swapper/320:58:053
2120599140,12cyclictest0-21swapper/323:13:123
2120599140,11cyclictest0-21swapper/320:33:053
2120599140,10cyclictest0-21swapper/322:16:033
21204991411,2cyclictest31073-21kworker/u8:120:05:052
21204991411,2cyclictest29063-21kworker/u8:020:52:032
21204991411,2cyclictest27451-21kworker/u8:019:31:042
21204991411,2cyclictest25185-21kworker/u8:021:45:052
21204991411,2cyclictest19425-21kworker/u8:120:40:032
21204991411,2cyclictest16942-21kworker/u8:022:35:042
21204991411,2cyclictest0-21swapper/200:29:212
2120499140,12cyclictest11500-21ssh21:51:072
2120499140,10cyclictest0-21swapper/221:22:052
2120399149,3cyclictest5213-21iostat_ios22:55:151
2120399142,2cyclictest23-21ksoftirqd/122:21:101
21203991411,2cyclictest21-21rcuc/121:18:541
21203991411,2cyclictest0-21swapper/123:36:201
21203991411,2cyclictest0-21swapper/122:46:101
2120399141,11cyclictest4965-21diskmemload23:15:111
2120399141,11cyclictest30599-21ssh21:36:211
21203991410,2cyclictest0-21swapper/123:50:141
2120399140,2cyclictest0-21swapper/123:48:581
2120399140,2cyclictest0-21swapper/123:30:191
2120399140,13cyclictest0-21swapper/123:26:301
2120399140,13cyclictest0-21swapper/123:20:091
2120399140,12cyclictest4965-21diskmemload23:55:471
2120399140,12cyclictest0-21swapper/122:17:561
2120399140,12cyclictest0-21swapper/100:20:501
2120299142,7cyclictest0-21swapper/021:45:000
2120299142,7cyclictest0-21swapper/021:34:590
2120299142,6cyclictest0-21swapper/020:30:000
2120299141,3cyclictest24442-21munin-node0
2120299141,11cyclictest10606-21ssh22:25:510
2120299140,3cyclictest8333-21latency_hist23:35:010
2120299140,3cyclictest23975-21cut22:05:000
2120299140,12cyclictest0-21swapper/023:55:130
2120299140,12cyclictest0-21swapper/020:00:160
2120299140,12cyclictest0-21swapper/020:00:160
21205991310,2cyclictest4965-21diskmemload00:14:013
21205991310,2cyclictest31462-21kworker/u8:221:15:033
21205991310,2cyclictest2503-21kworker/u8:219:47:043
21205991310,2cyclictest24509-21kworker/u8:122:12:043
21205991310,2cyclictest23392-21kworker/u8:122:42:043
21205991310,2cyclictest19425-21kworker/u8:120:50:043
21205991310,2cyclictest19401-21kworker/u8:222:09:053
21205991310,2cyclictest14219-21kworker/u8:021:21:043
21205991310,2cyclictest13720-21kworker/u8:021:56:053
21205991310,2cyclictest10613-21kworker/u8:022:26:053
2120599130,2cyclictest0-21swapper/323:53:043
2120599130,2cyclictest0-21swapper/323:16:023
2120599130,2cyclictest0-21swapper/320:36:053
2120599130,11cyclictest1031-21aten2.4_r3power21:00:133
2120499131,2cyclictest32248-21munin-node2
21204991310,2cyclictest24509-21kworker/u8:122:15:052
2120499130,2cyclictest0-21swapper/221:15:122
2120499130,12cyclictest19654-21ssh23:10:522
2120399139,2cyclictest0-21swapper/121:47:201
2120399132,7cyclictest0-21swapper/120:50:011
2120399132,6cyclictest4965-21diskmemload23:10:011
2120399131,10cyclictest3024-21munin-node1
21203991310,2cyclictest4965-21diskmemload00:31:031
21203991310,2cyclictest21-21rcuc/119:10:141
21203991310,2cyclictest0-21swapper/122:29:251
21203991310,2cyclictest0-21swapper/121:41:591
21203991310,2cyclictest0-21swapper/119:20:201
21203991310,2cyclictest0-21swapper/100:38:521
2120399130,2cyclictest221ktimersoftd/122:42:471
2120399130,2cyclictest221ktimersoftd/100:15:161
2120399130,2cyclictest0-21swapper/120:55:161
2120399130,1cyclictest0-21swapper/120:00:201
2120399130,1cyclictest0-21swapper/120:00:201
2120399130,12cyclictest0-21swapper/122:01:061
2120399130,11cyclictest221ktimersoftd/123:40:241
2120299132,6cyclictest0-21swapper/022:55:010
2120299131,10cyclictest0-21swapper/021:50:290
21202991310,2cyclictest0-21swapper/023:02:100
21202991310,2cyclictest0-21swapper/022:56:290
21202991310,2cyclictest0-21swapper/000:31:540
2120299130,2cyclictest0-21swapper/021:35:510
2120299130,12cyclictest4965-21diskmemload00:05:500
2120299130,11cyclictest32195-21apt-get23:25:070
2120299130,11cyclictest0-21swapper/023:54:340
2120599129,2cyclictest7875-21kworker/u8:100:19:043
2120599129,2cyclictest2753-21kworker/u8:222:56:053
2120599129,2cyclictest19021-21kworker/u8:119:24:053
2120599129,2cyclictest19021-21kworker/u8:119:18:033
2120599129,2cyclictest0-21swapper/323:26:033
2120599129,2cyclictest0-21swapper/322:35:403
2120599129,2cyclictest0-21swapper/322:30:123
2120599129,2cyclictest0-21swapper/321:40:503
2120599124,7cyclictest8422-21nscd00:26:593
2120599122,7cyclictest0-21swapper/320:28:013
2120599120,7cyclictest4572-21gdm-simple-gree19:37:593
2120599120,2cyclictest0-21swapper/323:05:033
2120599120,2cyclictest0-21swapper/320:20:053
2120599120,11cyclictest4965-21diskmemload21:45:103
2120599120,10cyclictest0-21swapper/322:47:053
2120499120,11cyclictest0-21swapper/222:11:432
2120399129,2cyclictest0-21swapper/122:05:171
2120399129,2cyclictest0-21swapper/121:55:461
2120399129,2cyclictest0-21swapper/120:10:261
2120399129,2cyclictest0-21swapper/100:25:481
2120399121,9cyclictest0-21swapper/120:15:121
2120399121,9cyclictest0-21swapper/119:45:211
2120399121,3cyclictest0-21swapper/119:35:111
2120399120,11cyclictest4965-21diskmemload00:06:031
2120399120,11cyclictest0-21swapper/122:30:041
2120399120,11cyclictest0-21swapper/121:50:141
2120399120,11cyclictest0-21swapper/121:00:151
2120299129,2cyclictest0-21swapper/023:45:200
2120299129,2cyclictest0-21swapper/022:36:410
2120299129,2cyclictest0-21swapper/021:48:300
2120299129,2cyclictest0-21swapper/021:05:120
2120299129,2cyclictest0-21swapper/000:10:500
2120299122,8cyclictest0-21swapper/020:05:000
2120299122,7cyclictest0-21swapper/021:04:590
2120299122,7cyclictest0-21swapper/019:43:020
2120299122,6cyclictest0-21swapper/021:25:000
2120299120,2cyclictest0-21swapper/022:49:300
2120299120,2cyclictest0-21swapper/022:07:050
2120299120,2cyclictest0-21swapper/019:15:150
2120299120,2cyclictest0-21swapper/000:35:190
2120299120,2cyclictest0-21swapper/000:15:530
2120299120,11cyclictest4965-21diskmemload23:35:510
2120299120,11cyclictest0-21swapper/023:10:390
2120299120,11cyclictest0-21swapper/022:20:490
2120299120,11cyclictest0-21swapper/021:56:490
2120299120,11cyclictest0-21swapper/000:03:000
155722120,1sleep00-21swapper/022:30:550
227992110,1sleep00-21swapper/023:15:140
2120599118,2cyclictest5331-21kworker/u8:019:57:053
2120599118,2cyclictest2503-21kworker/u8:219:50:053
2120599118,2cyclictest16942-21kworker/u8:023:20:043
2120599118,2cyclictest13357-21kworker/u8:019:10:053
2120599118,2cyclictest13312-21kworker/u8:020:12:053
2120599110,2cyclictest4965-21diskmemload23:46:123
2120599110,10cyclictest0-21swapper/321:05:053
2120599110,10cyclictest0-21swapper/300:03:203
2120399112,6cyclictest0-21swapper/119:16:591
2120399111,8cyclictest0-21swapper/119:30:201
2120399110,9cyclictest5230-21munin-node1
2120399110,9cyclictest221ktimersoftd/120:00:001
2120399110,9cyclictest0-21swapper/120:20:161
2120399110,3cyclictest23378-21munin-node1
2120299118,2cyclictest0-21swapper/023:05:230
2120299118,2cyclictest0-21swapper/020:55:240
2120299118,2cyclictest0-21swapper/000:21:010
2120299113,3cyclictest15724-1kworker/0:1H20:55:010
2120299112,6cyclictest0-21swapper/019:24:010
2120299111,8cyclictest3745-21runrttasks19:58:220
2120299110,10cyclictest0-21swapper/020:20:170
2120599107,2cyclictest13312-21kworker/u8:020:15:043
2120599100,8cyclictest20468-21ssh21:25:493
2120499107,2cyclictest0-21swapper/219:40:282
2120399107,2cyclictest0-21swapper/120:30:211
2120399101,7cyclictest0-21swapper/120:40:241
2120299105,3cyclictest28747-21grep20:45:260
2120299102,6cyclictest277-21jbd2/sda1-819:45:300
2120299101,7cyclictest7013-21cron19:55:000
2120299101,6cyclictest0-21swapper/019:25:250
212059996,2cyclictest19425-21kworker/u8:120:46:043
212059990,6cyclictest0-21swapper/320:41:593
212049996,2cyclictest0-21swapper/221:04:562
212049994,3cyclictest3745-21runrttasks19:27:462
212029994,3cyclictest23429-21aten2.4_r3power20:35:130
212029994,3cyclictest19751-21df20:25:140
212029994,3cyclictest13573-21tr20:10:150
212029991,3cyclictest0-21swapper/020:15:140
212049983,3cyclictest17574-21grep20:20:132
212029983,3cyclictest29754-21cut19:30:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional