You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-15 - 06:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Jan 15, 2025 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2118991063,13cyclictest32383-21apt-get21:46:133
20949910143,8cyclictest0-21swapper/223:46:292
20949910143,8cyclictest0-21swapper/223:46:292
2067991002,60cyclictest8342-21apt-get21:01:121
2094999949,43cyclictest0-21swapper/221:15:582
2067999743,46cyclictest25354-21unixbench_multi21:31:341
2067999743,46cyclictest25354-21unixbench_multi21:31:341
2118999277,8cyclictest0-21swapper/321:06:193
2094999234,10cyclictest0-21swapper/219:41:442
2094999041,11cyclictest0-21swapper/221:31:022
206799892,18cyclictest2077-21cmp19:06:031
211899843,49cyclictest26474-21apt-get20:16:123
17632845,13sleep20-21swapper/219:05:322
2094998363,14cyclictest5015-21idleruntime-cro22:55:582
2094998353,8cyclictest33-21ksoftirqd/200:21:012
2094998353,8cyclictest33-21ksoftirqd/200:21:012
16582836,13sleep00-21swapper/019:04:070
2067998221,10cyclictest0-21swapper/120:06:541
188528213,37sleep31949-21gpgv19:06:013
2118998143,30cyclictest0-21swapper/322:16:323
2067998127,45cyclictest0-21swapper/122:26:021
211899802,42cyclictest22980-21apt-get20:06:153
2118997955,15cyclictest32619-21unixbench_multi23:46:293
2118997955,15cyclictest32619-21unixbench_multi23:46:293
211899792,42cyclictest8342-21apt-get21:01:123
206799793,18cyclictest3093-21head20:41:331
206799793,18cyclictest3093-21head20:41:331
2037997947,13cyclictest141rcu_preempt21:36:160
2067997821,8cyclictest0-21swapper/122:16:201
2067997821,7cyclictest0-21swapper/119:36:321
2067997817,12cyclictest0-21swapper/119:55:591
2067997727,41cyclictest0-21swapper/119:46:261
2067997721,8cyclictest0-21swapper/119:15:591
2067997721,8cyclictest0-21swapper/119:15:591
2037997759,12cyclictest3441-21unixbench_multi23:52:110
2037997759,12cyclictest3441-21unixbench_multi23:52:110
2037997743,13cyclictest141rcu_preempt22:53:430
2118997640,10cyclictest10202-21diskmemload21:30:593
211899763,12cyclictest15494-21runrttasks23:15:243
211899763,12cyclictest15494-21runrttasks23:15:243
2067997632,36cyclictest2026-21cyclictest23:11:221
2067997632,36cyclictest2026-21cyclictest23:11:221
206799763,18cyclictest13664-21chrt21:11:111
2067997622,7cyclictest0-21swapper/120:46:351
206799762,22cyclictest25207-21grep22:31:351
206799762,21cyclictest23468-21latency_hist00:31:001
206799762,21cyclictest23468-21latency_hist00:31:001
2067997613,12cyclictest9080-21unin-run19:26:001
2037997659,11cyclictest31660-21kworker/u8:123:15:580
2037997659,11cyclictest31660-21kworker/u8:123:15:580
2037997646,9cyclictest141rcu_preempt23:59:020
2037997646,9cyclictest141rcu_preempt23:59:020
132692760,6chrt0-21swapper/200:11:212
132692760,6chrt0-21swapper/200:11:212
211899752,13cyclictest12929-21taskset22:10:133
206799757,21cyclictest23805-21/usr/sbin/munin23:31:361
2067997521,47cyclictest0-21swapper/119:31:011
2067997519,11cyclictest0-21swapper/123:06:271
2067997519,11cyclictest0-21swapper/123:06:271
2037997550,17cyclictest29382-21idleruntime-cro23:40:590
203799753,45cyclictest31859-21apt-get21:46:130
211899742,8cyclictest13518-21ssh22:11:123
211899742,63cyclictest22781-21chrt20:06:023
206799743,20cyclictest18681-21latency_hist00:21:001
206799743,20cyclictest18681-21latency_hist00:21:001
2067997418,8cyclictest0-21swapper/122:51:191
2067997415,12cyclictest2026-21cyclictest23:41:021
2037997450,17cyclictest5790-21munin-plugin-st19:15:590
2037997450,17cyclictest5790-21munin-plugin-st19:15:590
2118997346,16cyclictest412-21snmpd23:51:183
2118997346,16cyclictest412-21snmpd23:51:183
211899733,59cyclictest394-21runrttasks20:26:243
211899733,59cyclictest394-21runrttasks20:26:243
2094997344,18cyclictest33-21ksoftirqd/219:41:022
206799733,18cyclictest6835-21grep21:56:341
206799733,18cyclictest24111-21latency_hist21:31:011
2067997327,38cyclictest0-21swapper/122:11:011
2067997322,8cyclictest0-21swapper/121:46:141
206799732,20cyclictest21636-21/usr/sbin/munin22:26:191
206799732,20cyclictest21636-21/usr/sbin/munin22:26:191
2067997321,43cyclictest0-21swapper/122:11:211
2067997319,8cyclictest0-21swapper/123:55:031
2067997319,8cyclictest0-21swapper/123:55:031
211899723,15cyclictest5540-21seq23:56:223
211899723,15cyclictest5540-21seq23:56:223
211899723,14cyclictest15982-21munin-plugin-st19:45:593
211899723,13cyclictest20896-21sh23:26:083
211899723,13cyclictest20896-21sh23:26:083
211899722,13cyclictest19172-21taskset21:20:553
2067997222,10cyclictest376-21systemd-logind23:25:591
206799722,17cyclictest28348-21grep21:36:331
2067997220,8cyclictest0-21swapper/100:06:281
2037997214,37cyclictest8844-21apt-get21:01:170
211899713,13cyclictest12897-21latency_hist23:11:013
211899713,13cyclictest12897-21latency_hist23:11:013
211899713,11cyclictest3055-21/usr/sbin/munin22:51:293
2118997126,19cyclictest141rcu_preempt22:31:023
2118997126,19cyclictest141rcu_preempt22:31:023
209499717,17cyclictest0-21swapper/221:06:192
2094997156,9cyclictest6702-21idleruntime-cro20:55:592
206799713,19cyclictest31519-21ssh23:46:021
206799713,19cyclictest21138-21modprobe21:25:591
206799713,18cyclictest32257-21proc_pri23:46:251
206799713,18cyclictest32257-21proc_pri23:46:251
206799713,17cyclictest9909-21latency_hist00:06:011
206799713,17cyclictest3081-21fschecks_count21:51:191
206799713,17cyclictest26378-21latency_hist20:16:031
2067997122,9cyclictest0-21swapper/121:06:301
2067997122,7cyclictest0-21swapper/122:46:131
206799712,22cyclictest27451-21fschecks_time22:36:211
206799712,18cyclictest6284-21grep20:51:361
203799714,47cyclictest23021-21cpu20:06:200
2118997050,13cyclictest0-21swapper/321:41:003
211899703,15cyclictest25249-21sh21:31:343
211899703,15cyclictest25249-21sh21:31:343
211899703,11cyclictest23738-21apt-get00:31:133
211899702,13cyclictest21839-21wc00:26:303
211899702,13cyclictest21839-21wc00:26:303
211899702,12cyclictest16144-21ssh23:16:163
211899702,11cyclictest13569-21/usr/sbin/munin19:36:343
2118997016,25cyclictest141rcu_preempt20:36:253
206799704,15cyclictest2587-21fschecks_count19:06:371
206799703,21cyclictest8768-21latency_hist21:01:021
206799703,18cyclictest11440-21ls19:31:241
206799703,18cyclictest11440-21ls19:31:241
206799703,17cyclictest5146-21/usr/sbin/munin23:56:201
206799703,17cyclictest5146-21/usr/sbin/munin23:56:201
206799703,16cyclictest23499-21modprobe23:30:581
206799703,16cyclictest23499-21modprobe23:30:581
2067997030,15cyclictest10202-21diskmemload22:56:331
206799702,18cyclictest21731-21find20:01:311
206799702,17cyclictest16879-21fschecks_count21:16:201
2067997016,10cyclictest0-21swapper/120:21:301
2037997041,12cyclictest141rcu_preempt20:41:310
2037997041,12cyclictest141rcu_preempt20:41:310
2037997012,31cyclictest12-21ksoftirqd/022:36:000
211899695,47cyclictest6629-21/usr/sbin/munin19:16:303
211899693,11cyclictest5758-21/usr/sbin/munin20:51:263
211899693,10cyclictest13855-21timerandwakeup00:11:333
211899693,10cyclictest13855-21timerandwakeup00:11:333
211899692,17cyclictest31983-21apt-key21:46:013
211899692,16cyclictest10317-21modprobe22:05:593
211899692,16cyclictest10317-21modprobe22:05:593
211899692,13cyclictest17221-21/usr/sbin/munin19:46:353
211899692,12cyclictest11726-21/usr/sbin/munin19:31:323
211899692,12cyclictest11726-21/usr/sbin/munin19:31:323
211899692,10cyclictest29914-21latency_hist22:41:023
2118996918,40cyclictest0-21swapper/321:51:123
206799693,20cyclictest19729-21fschecks_time19:56:211
2067996915,8cyclictest0-21swapper/121:46:021
2067996911,12cyclictest0-21swapper/120:36:341
2037996934,16cyclictest141rcu_preempt20:51:340
211899683,9cyclictest0-21swapper/323:36:193
211899683,6cyclictest0-21swapper/319:06:293
211899683,58cyclictest25534-21head20:11:353
211899683,17cyclictest20395-21grep19:56:343
211899683,15cyclictest32515-21ntp_kernel_pll_22:46:213
211899683,13cyclictest23806-21/usr/sbin/munin23:31:093
211899682,17cyclictest394-21runrttasks22:57:083
2094996847,11cyclictest3917-21unixbench_singl21:51:332
2094996837,10cyclictest33-21ksoftirqd/219:16:302
206799684,18cyclictest9844-21modprobe23:05:591
206799683,16cyclictest15712-21latency_hist00:16:001
206799683,16cyclictest15712-21latency_hist00:16:001
2067996824,8cyclictest0-21swapper/119:20:591
206799682,20cyclictest24061-21fschecks_time00:31:241
2037996848,7cyclictest141rcu_preempt21:59:520
2037996845,9cyclictest141rcu_preempt22:56:330
2037996844,14cyclictest0-21swapper/022:50:590
2037996833,21cyclictest26979-21kworker/u8:523:32:180
2037996820,20cyclictest12-21ksoftirqd/019:51:220
2037996818,37cyclictest14630-21apt-get19:41:240
211899673,57cyclictest4083-21fschecks_time19:11:053
211899673,57cyclictest4083-21fschecks_time19:11:053
211899673,55cyclictest32235-21grep20:31:333
211899673,13cyclictest20975-21ssh22:26:003
211899673,11cyclictest6969-21munin-node20:56:103
211899672,58cyclictest4707-21grep20:46:363
211899672,16cyclictest3068-21unixbench_singl20:41:333
211899672,16cyclictest3068-21unixbench_singl20:41:333
211899672,13cyclictest394-21runrttasks00:22:153
211899672,12cyclictest10427-21/usr/sbin/munin00:06:193
211899672,11cyclictest18017-21/usr/sbin/munin19:51:203
211899672,11cyclictest10515-21/usr/sbin/munin19:26:413
2118996717,39cyclictest0-21swapper/320:21:323
2118996714,46cyclictest9044-21basename19:25:593
206799673,18cyclictest16327-21fschecks_time23:16:211
2067996722,8cyclictest0-21swapper/119:41:211
211899669,51cyclictest0-21swapper/300:21:013
211899669,51cyclictest0-21swapper/300:21:013
211899663,15cyclictest6772-21/usr/sbin/munin21:56:343
211899663,12cyclictest31355-21taskset23:45:593
211899662,11cyclictest13767-21cstates21:11:153
206799662,17cyclictest10285-21cron22:05:581
206799662,17cyclictest10285-21cron22:05:581
2067996610,12cyclictest0-21swapper/120:16:431
211899653,16cyclictest394-21runrttasks21:26:023
211899653,10cyclictest8292-21ssh23:01:233
2094996547,12cyclictest13202-21kworker/u8:322:10:582
2094996547,11cyclictest0-21swapper/221:46:012
2094996542,12cyclictest0-21swapper/221:16:202
2094996541,9cyclictest33-21ksoftirqd/220:11:352
2094996541,15cyclictest12517-21sendmail19:35:592
2094996541,15cyclictest12517-21sendmail19:35:592
203799658,45cyclictest9519-21/usr/sbin/munin19:26:190
2037996537,13cyclictest0-21swapper/019:06:150
2037996531,26cyclictest0-21swapper/021:55:580
211899643,11cyclictest9762-21ssh00:05:593
211899642,17cyclictest24444-21fschecks_count22:31:203
206799643,22cyclictest29812-21cstates20:26:231
206799643,22cyclictest29812-21cstates20:26:231
203799648,17cyclictest12-21ksoftirqd/019:41:010
203799647,39cyclictest12-21ksoftirqd/000:26:010
2037996442,14cyclictest2158-21latency_hist23:51:010
2037996442,14cyclictest2158-21latency_hist23:51:010
2037996435,20cyclictest0-21swapper/021:01:000
2037996432,12cyclictest141rcu_preempt23:21:000
2037996419,22cyclictest141rcu_preempt22:04:130
2037996419,22cyclictest141rcu_preempt22:04:130
2037996416,36cyclictest0-21swapper/021:16:200
2037996413,28cyclictest0-21swapper/000:08:070
211899633,13cyclictest31886-21latency_hist22:46:013
211899633,12cyclictest18888-21ssh23:21:023
2094996349,9cyclictest2026-21cyclictest20:46:232
2094996345,11cyclictest26405-21latency_hist00:36:002
206799633,22cyclictest32700-21sendmail-msp20:35:591
203799636,38cyclictest12-21ksoftirqd/020:01:150
2037996342,15cyclictest11083-21cut21:06:250
2037996333,9cyclictest141rcu_preempt22:28:350
2037996333,9cyclictest141rcu_preempt22:28:350
2037996322,25cyclictest141rcu_preempt00:35:470
2094996246,9cyclictest27056-21runrttasks22:36:012
2094996238,8cyclictest33-21ksoftirqd/221:01:012
2037996246,10cyclictest12-21ksoftirqd/021:31:340
2037996246,10cyclictest12-21ksoftirqd/021:31:340
2037996224,26cyclictest0-21swapper/000:26:230
2037996224,26cyclictest0-21swapper/000:26:230
2037996220,24cyclictest52-21kcompactd021:11:250
2037996210,21cyclictest12-21ksoftirqd/019:46:320
321792612,6sleep00-21swapper/020:31:340
203799619,33cyclictest12-21ksoftirqd/019:31:220
203799619,33cyclictest12-21ksoftirqd/019:31:220
2037996139,15cyclictest1-21systemd23:10:590
2037996139,15cyclictest1-21systemd23:10:590
2037996128,9cyclictest0-21swapper/022:16:290
2037996128,12cyclictest141rcu_preempt21:23:250
2094996033,7cyclictest33-21ksoftirqd/222:26:292
2094996033,7cyclictest33-21ksoftirqd/222:26:292
203799604,38cyclictest12-21ksoftirqd/023:26:230
203799604,38cyclictest12-21ksoftirqd/023:26:230
2037996038,12cyclictest0-21swapper/000:11:300
2037996038,12cyclictest0-21swapper/000:11:300
2037996022,12cyclictest0-21swapper/023:44:230
2037996010,11cyclictest141rcu_preempt00:16:200
2037996010,11cyclictest141rcu_preempt00:16:200
2094995934,8cyclictest33-21ksoftirqd/221:56:322
2037995940,10cyclictest0-21swapper/022:11:340
2037995939,8cyclictest141rcu_preempt22:06:060
2037995934,14cyclictest0-21swapper/022:36:300
2094995843,8cyclictest0-21swapper/200:26:312
2094995843,8cyclictest0-21swapper/200:26:312
2094995840,12cyclictest5777-21cron19:15:592
2094995840,12cyclictest5777-21cron19:15:592
2094995840,11cyclictest0-21swapper/220:10:592
2094995834,8cyclictest33-21ksoftirqd/220:31:342
2094995832,12cyclictest33-21ksoftirqd/222:43:032
2094995830,14cyclictest33-21ksoftirqd/222:41:012
2094995820,14cyclictest33-21ksoftirqd/222:56:282
206799582,21cyclictest18514-21apt-get00:21:161
2037995842,10cyclictest31049-21idleruntime-cro20:30:590
2037995842,10cyclictest31049-21idleruntime-cro20:30:590
2037995836,15cyclictest8851-21kworker/u8:021:26:050
2037995835,14cyclictest0-21swapper/022:45:580
2037995824,12cyclictest141rcu_preempt20:37:590
2094995741,8cyclictest33-21ksoftirqd/221:46:552
2037995742,10cyclictest131ktimers/020:21:180
203799573,29cyclictest12-21ksoftirqd/020:11:200
2037995711,18cyclictest12-21ksoftirqd/023:02:010
2094995635,8cyclictest33-21ksoftirqd/221:41:012
2094995632,8cyclictest33-21ksoftirqd/222:11:242
2094995624,11cyclictest179-21pr/ttyS023:11:152
2094995624,11cyclictest179-21pr/ttyS023:11:152
203799565,34cyclictest12-21ksoftirqd/021:42:210
2037995642,10cyclictest5670-21kworker/u8:219:16:240
2037995628,13cyclictest141rcu_preempt20:46:230
225912550,4sleep2311rcuc/220:06:012
209499559,29cyclictest264772sleep220:16:122
2094995540,8cyclictest0-21swapper/222:51:002
2094995539,8cyclictest0-21swapper/220:36:342
2094995538,11cyclictest275-21systemd-journal00:25:502
2094995517,24cyclictest33-21ksoftirqd/223:26:272
2094995517,24cyclictest33-21ksoftirqd/223:26:272
2094995515,24cyclictest33-21ksoftirqd/219:51:342
2037995538,12cyclictest0-21swapper/023:26:010
2037995535,7cyclictest141rcu_preempt00:02:040
203799553,18cyclictest9448-21apt-key19:26:020
2037995525,11cyclictest0-21swapper/022:26:020
2094995441,7cyclictest0-21swapper/223:16:332
2094995438,9cyclictest20634-21systemd23:26:002
2094995438,9cyclictest0-21swapper/223:31:192
2094995435,11cyclictest627-21in:imklog22:06:012
2094995435,11cyclictest627-21in:imklog22:06:012
2094995432,11cyclictest0-21swapper/223:36:192
2094995419,13cyclictest141rcu_preempt19:46:322
2094995418,19cyclictest141rcu_preempt23:44:542
2094995410,13cyclictest141rcu_preempt21:05:032
206799542,12cyclictest31689-21ssh22:45:591
2094995338,9cyclictest5766-21kworker/u8:020:31:012
2094995338,9cyclictest5766-21kworker/u8:020:31:012
2094995338,8cyclictest0-21swapper/221:25:592
2094995337,10cyclictest33-21ksoftirqd/221:31:262
2094995337,10cyclictest33-21ksoftirqd/221:31:262
2094995333,13cyclictest0-21swapper/219:26:012
2094995331,13cyclictest33-21ksoftirqd/223:06:162
2094995331,13cyclictest33-21ksoftirqd/223:06:162
2094995329,7cyclictest33-21ksoftirqd/223:56:022
2094995329,7cyclictest33-21ksoftirqd/223:56:022
203799535,29cyclictest12-21ksoftirqd/019:56:310
2037995334,13cyclictest0-21swapper/020:16:030
209499522,23cyclictest20469-21grep19:56:342
2094995140,6cyclictest0-21swapper/219:06:152
209499512,29cyclictest394-21runrttasks19:26:252
209499494,13cyclictest0-21swapper/222:22:342
2094994923,8cyclictest33-21ksoftirqd/220:45:592
2094994923,8cyclictest33-21ksoftirqd/220:45:592
2094994831,9cyclictest0-21swapper/222:16:332
2094994731,8cyclictest0-21swapper/223:56:352
2094994731,8cyclictest0-21swapper/223:56:352
2094994730,8cyclictest0-21swapper/200:06:172
2094994627,11cyclictest0-21swapper/200:01:442
209499462,23cyclictest275-21systemd-journal23:05:592
2094994428,10cyclictest0-21swapper/220:21:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional