You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-03 - 11:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 03, 2025 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
800699736,4cyclictest131rcu_preempt22:40:023
8006997216,14cyclictest14383-21latency_hist21:15:013
8006997013,18cyclictest21975-21cat23:40:033
8006996812,16cyclictest10778-21systemd-run20:05:003
8006996716,13cyclictest21730-21cat23:35:033
7996996615,28cyclictest12-21ksoftirqd/019:25:010
7996996615,28cyclictest12-21ksoftirqd/019:25:010
8006996513,15cyclictest22646-21sed23:55:023
8006996211,16cyclictest19697-21sed22:55:013
8006996210,16cyclictest9214-21latency_hist19:35:023
8006996115,15cyclictest15648-21/usr/sbin/munin21:40:023
8006995911,17cyclictest21233-21sed23:25:023
800099593,40cyclictest221rcuc/100:15:031
8006995814,14cyclictest24527-21wc00:30:023
8006995814,14cyclictest24527-21wc00:30:013
8006995811,14cyclictest0-21swapper/319:10:023
8000995820,23cyclictest0-21swapper/120:55:021
7996995848,7cyclictest12-21ksoftirqd/023:10:010
800699575,14cyclictest1-21systemd22:50:013
8000995636,15cyclictest1-21systemd22:25:021
8006995512,13cyclictest20294-21cut23:10:013
8006995510,16cyclictest22038-21modprobe23:40:023
8000995437,13cyclictest1-21systemd19:50:001
799699546,10cyclictest12-21ksoftirqd/021:55:020
799699546,10cyclictest12-21ksoftirqd/021:55:010
7996995413,36cyclictest12-21ksoftirqd/019:50:010
8006995336,13cyclictest1-21systemd00:40:013
8006995331,17cyclictest16379-21sshd21:50:353
8006995331,17cyclictest16379-21sshd21:50:343
8006995236,12cyclictest1-21systemd00:05:003
8006995235,13cyclictest15357-21sshd21:30:343
8003995233,14cyclictest1-21systemd23:00:002
8000995234,13cyclictest1-21systemd21:10:011
8000995234,13cyclictest1-21systemd21:10:001
8006995143,4cyclictest37-21ksoftirqd/322:15:023
8006995137,11cyclictest12806-21sshd20:44:273
8006995134,13cyclictest21296-21munin-run23:30:003
8006995132,13cyclictest24679-21munin-run00:34:593
799699519,8cyclictest12-21ksoftirqd/021:35:000
799699518,6cyclictest12-21ksoftirqd/022:25:000
799699516,10cyclictest12-21ksoftirqd/000:25:010
799699512,9cyclictest12-21ksoftirqd/022:20:010
7996995113,8cyclictest12-21ksoftirqd/021:30:020
8006995032,15cyclictest17154-21sshd22:05:343
8006995023,23cyclictest15923-21true21:45:013
799699505,9cyclictest12-21ksoftirqd/020:55:020
799699505,9cyclictest12-21ksoftirqd/020:55:010
7996995041,6cyclictest12-21ksoftirqd/000:35:000
7996995033,12cyclictest1-21systemd20:05:000
799699501,7cyclictest0-21swapper/021:10:010
799699501,7cyclictest0-21swapper/021:10:000
799699501,13cyclictest12-21ksoftirqd/022:54:590
800699494,11cyclictest20483-21sshd23:11:353
8006994935,11cyclictest8803-21cat19:25:013
8006994935,11cyclictest8803-21cat19:25:013
8006994933,12cyclictest18433-21sshd22:30:363
8006994933,12cyclictest18433-21sshd22:30:353
8006994933,12cyclictest12030-21sshd20:28:283
8006994933,12cyclictest12030-21sshd20:28:273
8006994932,14cyclictest10986-21sshd20:06:273
8003994938,8cyclictest30-21ksoftirqd/219:50:012
799699496,8cyclictest12-21ksoftirqd/022:15:010
799699494,10cyclictest12-21ksoftirqd/020:00:010
7996994939,6cyclictest16662-21sendmail21:59:590
799699491,6cyclictest0-21swapper/023:25:020
8006994835,10cyclictest16144-21sshd21:48:353
8006994834,11cyclictest23629-21sshd00:13:343
8006994834,10cyclictest10469-21sshd19:58:263
8006994832,13cyclictest8157-21sshd19:12:263
8006994832,12cyclictest10240-21munin-run19:55:013
8000994831,13cyclictest1-21systemd20:20:011
799699481,11cyclictest12-21ksoftirqd/020:30:010
799699481,11cyclictest12-21ksoftirqd/020:30:010
800699477,13cyclictest20028-21systemctl23:05:013
8006994735,8cyclictest22798-21sshd23:58:353
8006994735,8cyclictest22798-21sshd23:58:343
8006994734,10cyclictest23890-21sshd00:19:363
8006994733,11cyclictest17655-21sshd22:16:343
8006994732,11cyclictest11814-21munin-run20:25:013
8006994731,13cyclictest9131-21cat19:30:033
8006994731,13cyclictest9131-21cat19:30:023
8006994729,13cyclictest13613-21munin-run21:00:003
8003994736,7cyclictest30-21ksoftirqd/223:59:592
8003994736,7cyclictest30-21ksoftirqd/200:00:002
8000994731,11cyclictest235-21systemd-journal20:20:011
799699479,10cyclictest12-21ksoftirqd/023:00:020
799699477,9cyclictest151rcuc/020:50:000
799699477,7cyclictest12-21ksoftirqd/020:59:590
799699476,10cyclictest12-21ksoftirqd/022:30:030
799699471,13cyclictest12-21ksoftirqd/021:25:020
7996994711,14cyclictest12-21ksoftirqd/020:05:030
8006994632,11cyclictest14626-21munin-run21:20:003
8006994631,12cyclictest13826-21sshd21:03:353
8006994631,11cyclictest22335-21munin-run23:50:013
8006994630,12cyclictest17971-21munin-run22:25:003
8006994630,11cyclictest48-21kcompactd021:57:563
8003994635,9cyclictest30-21ksoftirqd/219:15:022
8000994631,9cyclictest235-21systemd-journal22:20:001
799699464,11cyclictest12-21ksoftirqd/021:15:010
799699464,11cyclictest12-21ksoftirqd/020:15:000
7996994635,8cyclictest12-21ksoftirqd/021:20:000
7996994629,14cyclictest272-21dbus-daemon00:00:010
7996994629,14cyclictest272-21dbus-daemon00:00:000
8006994533,8cyclictest24161-21sshd00:23:353
8006994533,8cyclictest18190-21sshd22:26:353
8006994531,11cyclictest9959-21sshd19:45:263
8006994531,11cyclictest13289-21sshd20:50:283
8006994531,11cyclictest13289-21sshd20:50:273
8006994530,12cyclictest11479-21sshd20:15:273
8006994529,12cyclictest363-21snmpd19:40:313
8006994528,12cyclictest363-21snmpd20:48:353
8006994527,13cyclictest272-21dbus-daemon23:00:003
8006994512,10cyclictest9504-21systemctl19:40:013
8006994512,10cyclictest9504-21systemctl19:40:013
8003994530,12cyclictest24194-21systemd-run00:25:012
8000994534,7cyclictest19725-21cron23:00:001
800099451,39cyclictest235-21systemd-journal00:05:001
799699456,6cyclictest12-21ksoftirqd/000:30:030
799699456,6cyclictest12-21ksoftirqd/000:30:020
799699455,10cyclictest12-21ksoftirqd/022:35:020
799699455,10cyclictest12-21ksoftirqd/022:35:010
799699454,9cyclictest12-21ksoftirqd/020:17:000
7996994537,5cyclictest151rcuc/023:50:010
799699452,10cyclictest12-21ksoftirqd/023:13:350
799699452,10cyclictest12-21ksoftirqd/019:10:020
8006994431,10cyclictest14873-21sshd21:23:353
8006994431,10cyclictest12265-21sshd20:30:273
8006994430,11cyclictest16914-21sshd22:02:353
8006994430,10cyclictest23400-21munin-run00:10:013
8003994433,7cyclictest10492-21sendmail20:00:002
800399443,33cyclictest24938-21munin-run00:40:002
8003994429,11cyclictest235-21systemd-journal20:35:012
8003994426,14cyclictest30-21ksoftirqd/200:30:032
8003994426,14cyclictest30-21ksoftirqd/200:30:022
8003994425,14cyclictest0-21swapper/222:10:012
8000994432,8cyclictest23-21ksoftirqd/100:00:001
8000994432,8cyclictest23-21ksoftirqd/100:00:001
8000994418,18cyclictest221rcuc/121:25:021
8000994412,9cyclictest131rcu_preempt00:30:031
8000994412,9cyclictest131rcu_preempt00:30:021
799699445,8cyclictest12-21ksoftirqd/000:36:350
799699445,10cyclictest12-21ksoftirqd/019:12:260
799699444,6cyclictest12-21ksoftirqd/020:35:020
799699444,10cyclictest12-21ksoftirqd/022:05:010
7996994432,8cyclictest12-21ksoftirqd/020:45:010
7996994432,7cyclictest19229-21cron22:50:010
799699443,11cyclictest12-21ksoftirqd/021:45:020
799699443,11cyclictest12-21ksoftirqd/000:05:010
7996994430,8cyclictest12-21ksoftirqd/019:55:030
799699442,8cyclictest12-21ksoftirqd/019:39:270
799699442,8cyclictest12-21ksoftirqd/019:39:270
8006994335,4cyclictest18976-21sshd22:44:353
8006994332,8cyclictest8397-21sshd19:15:283
8006994332,8cyclictest8397-21sshd19:15:273
8006994329,10cyclictest48-21kcompactd020:10:073
8006994324,15cyclictest363-21snmpd21:07:333
8006994324,15cyclictest363-21snmpd21:07:323
8003994327,12cyclictest235-21systemd-journal23:30:002
8003994317,17cyclictest30-21ksoftirqd/221:35:012
8000994328,12cyclictest23-21ksoftirqd/100:37:351
799699436,9cyclictest12-21ksoftirqd/023:55:020
799699435,11cyclictest12-21ksoftirqd/020:25:020
799699434,11cyclictest12-21ksoftirqd/023:35:000
7996994330,10cyclictest12-21ksoftirqd/021:50:020
8006994225,13cyclictest363-21snmpd23:16:433
8006994225,13cyclictest363-21snmpd23:16:423
8003994230,8cyclictest30-21ksoftirqd/222:40:012
8003994229,9cyclictest235-21systemd-journal21:40:002
8003994228,11cyclictest235-21systemd-journal21:55:352
8003994225,14cyclictest235-21systemd-journal21:10:012
8003994225,14cyclictest235-21systemd-journal21:10:012
8000994222,15cyclictest235-21systemd-journal22:15:001
7996994227,12cyclictest12-21ksoftirqd/021:05:010
8006994125,12cyclictest363-21snmpd21:27:343
8006994114,22cyclictest12735-21latency_hist20:40:023
8003994133,5cyclictest10765-21munin-plugin-st20:05:002
8003994131,6cyclictest30-21ksoftirqd/222:05:012
8003994123,13cyclictest272-21dbus-daemon23:20:012
8003994123,13cyclictest272-21dbus-daemon23:20:012
8000994126,11cyclictest235-21systemd-journal20:30:021
8000994126,11cyclictest235-21systemd-journal20:30:011
799699417,5cyclictest131rcu_preempt19:15:280
799699417,5cyclictest131rcu_preempt19:15:270
799699415,6cyclictest12-21ksoftirqd/023:40:030
7996994128,10cyclictest23385-21cron00:09:590
8003994031,6cyclictest30-21ksoftirqd/223:55:012
8003994029,8cyclictest30-21ksoftirqd/223:50:012
8003994027,9cyclictest30-21ksoftirqd/223:35:012
8003994027,8cyclictest30-21ksoftirqd/219:25:002
8003994027,8cyclictest30-21ksoftirqd/219:24:592
8003994026,9cyclictest30-21ksoftirqd/219:10:012
8003994023,12cyclictest272-21dbus-daemon00:35:012
8003994011,23cyclictest30-21ksoftirqd/220:20:012
8000994029,8cyclictest12533-21sshd20:39:281
8000994021,15cyclictest272-21dbus-daemon23:25:011
799699406,7cyclictest12-21ksoftirqd/019:30:010
799699406,7cyclictest12-21ksoftirqd/019:30:000
799699404,9cyclictest12-21ksoftirqd/023:30:000
799699403,9cyclictest12-21ksoftirqd/000:19:360
7996994033,4cyclictest16950irq/224-800000023:45:000
799699402,34cyclictest18684-21sshd22:35:350
7996994015,5cyclictest131rcu_preempt19:45:020
800399394,12cyclictest30-21ksoftirqd/220:10:002
8003993928,7cyclictest19230-21cron22:50:002
8003993927,9cyclictest30-21ksoftirqd/219:25:282
8003993927,9cyclictest30-21ksoftirqd/219:25:282
800399391,8cyclictest0-21swapper/223:42:362
8000993927,9cyclictest19311-21sed22:50:011
8000993926,10cyclictest23-21ksoftirqd/121:50:011
8000993924,11cyclictest18721-21cat22:39:591
7996993933,4cyclictest0-21swapper/000:15:010
7996993927,9cyclictest12-21ksoftirqd/020:40:000
799699391,6cyclictest0-21swapper/021:35:520
8003993829,6cyclictest30-21ksoftirqd/221:55:012
8003993829,6cyclictest30-21ksoftirqd/221:55:002
8003993825,10cyclictest15225-21sed21:30:022
8003993815,9cyclictest131rcu_preempt20:15:012
800099384,7cyclictest131rcu_preempt21:15:011
8000993829,5cyclictest12295-21sh20:34:591
8000993828,7cyclictest19983-21sshd23:00:341
8000993826,9cyclictest23-21ksoftirqd/121:20:001
8000993826,5cyclictest23-21ksoftirqd/123:20:011
8000993826,5cyclictest23-21ksoftirqd/123:20:001
8000993825,9cyclictest23-21ksoftirqd/119:59:591
8000993825,11cyclictest23-21ksoftirqd/119:35:011
7996993832,3cyclictest16950irq/224-800000023:02:090
7996993825,9cyclictest235-21systemd-journal22:44:590
8003993728,6cyclictest8466-21sendmail-msp19:20:002
8003993728,6cyclictest8466-21sendmail-msp19:20:002
8003993728,6cyclictest30-21ksoftirqd/200:00:352
8003993725,10cyclictest30-21ksoftirqd/220:57:292
8003993718,14cyclictest272-21dbus-daemon19:45:002
800099376,3cyclictest131rcu_preempt22:30:011
8000993726,9cyclictest23-21ksoftirqd/119:25:001
8000993726,9cyclictest23-21ksoftirqd/119:24:591
8000993724,10cyclictest23-21ksoftirqd/123:25:011
8000993724,10cyclictest16850-21sed22:00:021
799699375,10cyclictest12-21ksoftirqd/019:35:020
7996993726,8cyclictest17172-21sshd22:09:350
7996993723,11cyclictest12-21ksoftirqd/023:20:020
7996993723,11cyclictest12-21ksoftirqd/023:20:010
8003993626,6cyclictest306-21in:imuxsock21:50:012
8003993624,9cyclictest30-21ksoftirqd/223:15:012
8003993623,9cyclictest9269-21cut19:35:012
8003993619,13cyclictest1-21systemd20:25:002
800399361,31cyclictest14788-21sort21:20:022
8003993613,14cyclictest30-21ksoftirqd/222:55:012
8000993626,6cyclictest11001-21cron20:09:591
8000993625,8cyclictest18464-21sh22:35:011
8000993625,8cyclictest18464-21sh22:35:011
8000993623,8cyclictest15390-21modprobe21:35:011
8000993616,16cyclictest272-21dbus-daemon23:35:011
8003993526,7cyclictest30-21ksoftirqd/222:15:002
8003993526,6cyclictest13273-21sed20:50:022
8003993519,11cyclictest272-21dbus-daemon20:40:012
800399351,5cyclictest0-21swapper/223:37:352
800099354,3cyclictest131rcu_preempt23:50:011
800099353,6cyclictest131rcu_preempt19:40:011
800099353,6cyclictest131rcu_preempt19:40:001
8000993524,8cyclictest23-21ksoftirqd/119:15:001
8000993523,8cyclictest235-21systemd-journal22:04:591
8000993522,10cyclictest23-21ksoftirqd/121:25:011
8000993521,10cyclictest23-21ksoftirqd/121:40:011
8000993511,5cyclictest131rcu_preempt20:03:281
800399343,6cyclictest131rcu_preempt00:19:592
8003993426,6cyclictest30-21ksoftirqd/221:25:012
8003993425,7cyclictest30-21ksoftirqd/223:10:032
8003993423,8cyclictest30-21ksoftirqd/222:25:012
8003993423,8cyclictest272-21dbus-daemon00:15:022
8003993420,9cyclictest235-21systemd-journal19:40:022
8003993420,9cyclictest235-21systemd-journal19:40:012
800399341,8cyclictest0-21swapper/222:30:012
8003993418,12cyclictest0-21swapper/222:20:012
800399341,5cyclictest131rcu_preempt20:45:002
800399341,5cyclictest0-21swapper/220:26:282
800399341,5cyclictest0-21swapper/220:26:272
8000993427,4cyclictest412-21sshd00:23:341
800099342,6cyclictest131rcu_preempt22:45:001
800099342,3cyclictest131rcu_preempt20:15:011
800099342,3cyclictest131rcu_preempt19:55:021
8000993420,9cyclictest272-21dbus-daemon00:10:011
8000993417,13cyclictest235-21systemd-journal22:54:591
8003993324,5cyclictest21046-21munin-plugin-st23:24:592
8003993321,9cyclictest10361-21sed19:55:022
8003993321,8cyclictest20216-21sed23:05:032
8003993318,12cyclictest272-21dbus-daemon21:45:012
8003993318,11cyclictest235-21systemd-journal22:45:012
800099333,7cyclictest131rcu_preempt21:05:021
800099332,8cyclictest12938-21latency_hist20:45:021
8000993325,5cyclictest9883-21latency_hist19:45:011
8003993220,8cyclictest306-21in:imuxsock20:55:002
8003993220,8cyclictest306-21in:imuxsock20:55:002
800099323,14cyclictest221rcuc/123:45:021
8000993224,5cyclictest13070-21latency_hist20:50:011
8000993223,5cyclictest24665-21cron00:35:001
8000993221,8cyclictest23-21ksoftirqd/123:50:341
8000993212,15cyclictest20507-21modprobe23:15:001
8003993120,8cyclictest23379-21sshd00:08:352
8003993119,10cyclictest30-21ksoftirqd/222:31:362
8003993119,10cyclictest30-21ksoftirqd/222:31:352
8003993112,6cyclictest131rcu_preempt21:13:002
800099315,3cyclictest131rcu_preempt23:38:341
8000993123,4cyclictest23825-21latency_hist00:15:021
8000993120,8cyclictest23-21ksoftirqd/122:05:351
8000993119,10cyclictest23-21ksoftirqd/119:30:011
8000993119,10cyclictest23-21ksoftirqd/119:30:001
8000993118,10cyclictest8579-21cat19:20:031
8000993118,10cyclictest8579-21cat19:20:021
8000993112,14cyclictest235-21systemd-journal23:10:011
8003993015,12cyclictest13914-21sort21:05:012
800099304,4cyclictest131rcu_preempt21:55:011
800099304,4cyclictest131rcu_preempt21:55:001
800099303,3cyclictest131rcu_preempt20:53:281
800099303,3cyclictest131rcu_preempt20:53:271
800099301,3cyclictest131rcu_preempt21:45:001
8000992815,10cyclictest8084-21cat19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional