You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-01 - 01:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot4s.osadl.org (updated Sat Aug 31, 2024 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
609999351,16cyclictest1415-21snmpd22:55:123
609999250,21cyclictest13237-21apt-get21:05:133
609999240,22cyclictest24905-21munin-run23:19:593
6097992415,6cyclictest14661-21sort21:10:001
6095992415,6cyclictest20262-21cat19:50:000
609999234,17cyclictest0-21swapper/322:40:013
609999234,17cyclictest0-21swapper/322:40:013
6099992316,4cyclictest12988-21python319:29:593
609999231,5cyclictest2864-21munin-run20:35:003
609899231,2cyclictest111rcu_preempt00:40:002
6097992317,4cyclictest21601-21latency_hist23:10:011
6097992315,5cyclictest11340-21latency_hist19:25:001
6097992314,6cyclictest26408-21grep23:20:241
609599232,18cyclictest31359-21grep22:00:130
6095992313,7cyclictest19673-21grep23:05:000
6099992214,6cyclictest25334-21df_inode23:20:133
6099992214,6cyclictest19992-21sed19:45:243
6099992214,6cyclictest18521-21turbostat00:40:003
6099992214,5cyclictest12639-21python321:05:003
609999221,17cyclictest21296-21python323:10:013
609999220,19cyclictest17885-21perf21:20:003
609999220,19cyclictest17885-21perf21:20:003
609999220,19cyclictest13483-40apt-check19:30:053
609899226,14cyclictest31294-21strings20:20:212
609899222,14cyclictest8996-21apt-get00:10:122
6098992210,8cyclictest26221-21apt-get21:45:122
6097992215,5cyclictest14981-21awk19:35:001
6097992214,6cyclictest6728-21sed19:10:161
6097992214,6cyclictest20087-21awk19:45:241
6095992214,6cyclictest7776-21cat20:50:010
6095992213,6cyclictest16367-21basename22:55:010
609599220,19cyclictest6948-21cat00:00:230
609999212,6cyclictest6838-21apt-get22:25:123
609999212,14cyclictest12257-21apt-get00:20:103
609999212,13cyclictest13873-21apt-get00:25:123
609999211,6cyclictest16607-21idleruntime-cro19:40:003
6099992116,3cyclictest4989-21lsb_release22:20:003
609999211,3cyclictest23475-21munin-run20:00:003
6099992113,6cyclictest32164-21sed22:00:223
6099992113,6cyclictest28510-21expr23:30:123
6099992113,6cyclictest27345-21cat23:25:163
6099992112,7cyclictest22336-21snmp_o5powerctl21:30:213
609999211,17cyclictest9370-21lsb_release20:55:003
609999211,17cyclictest18257-21python319:45:013
609999210,19cyclictest16347-21latency_hist22:55:003
609899212,14cyclictest16570-21apt-get22:55:122
6098992114,5cyclictest29257-21basename21:54:592
6098992114,5cyclictest11160-21date21:00:002
6098992113,6cyclictest6241-21cat00:00:152
609899211,14cyclictest11717-21apt-get22:40:122
609899210,5cyclictest0-21swapper/220:40:092
609899210,18cyclictest6828-21ntp_kernel_err19:10:182
609799218,3cyclictest111rcu_preempt20:20:001
6097992114,5cyclictest29006-21sed23:30:151
6097992114,5cyclictest16408-21date22:55:011
6097992114,5cyclictest15897-21grep21:10:221
6097992113,6cyclictest26926-21sed20:10:001
6095992114,5cyclictest10588-21ls19:20:170
6095992113,6cyclictest7957-21memory19:15:020
6095992113,6cyclictest1079-21awk20:25:240
6095992113,5cyclictest5510-21pluginstate20:40:170
609999209,9cyclictest31-21ksoftirqd/320:10:013
609999203,15cyclictest6614-21latency_hist22:25:003
609999203,15cyclictest6112-21latency_hist20:45:003
609999202,16cyclictest7735-21latency_hist20:50:003
6099992013,5cyclictest4709-21sed20:40:013
6099992013,5cyclictest27672-21date21:50:013
6099992012,5cyclictest3328-21cron22:15:003
609999201,17cyclictest30116-21find20:20:003
6099992011,6cyclictest16901-21awk00:35:013
609999201,14cyclictest6103-21apt-get19:10:123
609999201,13cyclictest10614-21apt-get00:15:103
609999200,3cyclictest31406-21awk23:40:013
609999200,3cyclictest19661-21idleruntime-cro23:05:013
609999200,2cyclictest28359-21munin-run20:15:013
609999200,18cyclictest30430-21readlink20:20:013
609999200,17cyclictest29251-21lsb_release21:55:003
609999200,17cyclictest10402-21perf00:15:003
6098992013,5cyclictest28285-21cut23:30:012
6098992013,5cyclictest19816-21cut23:05:012
6098992013,5cyclictest14995-21sed00:25:262
6098992012,5cyclictest10898-21mailstats19:20:192
609799202,15cyclictest24954-21latency_hist23:20:001
609799202,13cyclictest21267-21apt-get23:10:131
609799202,13cyclictest19578-21apt-get23:02:341
6097992015,2cyclictest18501-21latency_hist19:45:001
6097992013,5cyclictest39-21kauditd21:00:001
6097992013,5cyclictest22415-21grep19:55:151
6097992013,5cyclictest18547-21basename00:40:001
6097992013,5cyclictest18011-21sed21:20:001
6097992013,5cyclictest18011-21sed21:20:001
6097992013,5cyclictest13023-21awk22:40:251
6097992013,5cyclictest12769-21dpkg21:05:001
6097992012,5cyclictest629-21cut23:45:011
609599202,12cyclictest26904-21kworker/u8:120:20:150
6095992013,5cyclictest7436-21load22:25:180
6095992013,5cyclictest29526-21wc21:55:110
6095992013,5cyclictest28703-21expr20:15:150
6095992013,5cyclictest28528-21expr23:30:110
6095992013,5cyclictest28006-21cat21:50:120
6095992013,5cyclictest2786-21perl23:50:150
6095992013,5cyclictest26414-21ls20:05:230
6095992013,5cyclictest26156-21wc23:20:220
6095992013,5cyclictest17306-21ntpq22:55:170
6095992013,5cyclictest16383-21cat21:15:000
6095992013,5cyclictest13523-21df_inode22:45:130
6095992013,5cyclictest12226-21cut00:20:000
6095992012,6cyclictest16635-21wc00:30:270
6095992010,7cyclictest16457-21grep19:35:250
609999192,15cyclictest9704-21latency_hist19:20:003
6099991912,5cyclictest475-21awk22:05:153
6099991911,6cyclictest9908-21date22:35:003
6099991911,6cyclictest26072-21cat21:45:013
6099991911,5cyclictest30857-21munin-run22:00:003
609999191,15cyclictest24354-21sendmail21:40:003
609999191,13cyclictest7184-21latency_hist00:05:003
609999190,3cyclictest11001-21tr21:00:003
6098991915,2cyclictest17546-21timerandwakeup21:15:232
6098991915,2cyclictest17546-21timerandwakeup21:15:232
6098991915,2cyclictest15560-21grep19:35:172
6098991913,4cyclictest10542-21awk22:35:162
6098991913,4cyclictest10542-21awk22:35:162
6098991912,5cyclictest9877-21cat19:19:592
6098991912,5cyclictest7387-21cat22:25:172
6098991912,5cyclictest29159-21load20:15:172
6098991912,5cyclictest28615-21grep23:30:132
6098991912,5cyclictest11269-21awk21:00:122
6098991911,5cyclictest25-21ksoftirqd/200:30:222
609899191,14cyclictest2941-21snmp_o5powerctl22:10:202
609899191,13cyclictest13564-40apt-check19:30:062
609899191,13cyclictest11603-21snmp_o5powerctl00:15:212
609899191,12cyclictest18111-21apt-get21:20:132
6098991910,6cyclictest8543-21cat20:50:152
6097991912,5cyclictest7198-21grep22:25:161
6097991912,5cyclictest31335-21ntpq20:20:231
6097991912,5cyclictest24382-21basename21:40:001
6097991912,5cyclictest1624-21grep22:05:251
6097991911,6cyclictest9031-21sed22:30:151
6097991911,6cyclictest27195-21tr21:45:201
609799191,13cyclictest6342-21apt-get20:45:111
609799191,13cyclictest3568-21apt-get22:15:121
609599194,3cyclictest0-21swapper/023:42:270
6095991915,2cyclictest24249-21ntp_offset23:15:170
6095991915,2cyclictest2361-21processes20:30:190
6095991915,2cyclictest12622-21rt-features22:40:210
6095991913,4cyclictest14059-21df00:25:130
6095991912,5cyclictest6242-21expr19:10:120
6095991912,5cyclictest3439-21expr20:35:140
6095991912,5cyclictest27782-21date21:50:010
6095991912,5cyclictest23267-21unixbench-2d19:55:230
6095991912,5cyclictest14193-21expr19:30:200
6095991912,5cyclictest1159-21proc_pri22:05:200
6095991912,5cyclictest11371-21cat21:00:120
609599191,14cyclictest24707-21snmp_o5powerctl20:00:220
609599191,14cyclictest24707-21snmp_o5powerctl20:00:220
609599191,14cyclictest18103-21snmp_o5powerctl00:35:230
609599191,13cyclictest21889-21apt-get23:10:140
609999182,13cyclictest12442-21ntpq22:40:183
6099991814,2cyclictest14112-21open_files22:45:173
6099991812,4cyclictest5734-21sed00:00:013
609999181,15cyclictest25159-21cat20:05:003
609999181,15cyclictest25159-21cat20:05:003
6099991811,4cyclictest2752-21fschecks_time23:50:163
609999181,12cyclictest32226-21apt-get20:25:123
6099991810,6cyclictest1901-21cat23:45:223
6099991810,6cyclictest1901-21cat23:45:223
6099991810,6cyclictest18295-21df21:20:133
609999180,2cyclictest21876-1kworker/3:2H00:10:003
609999180,2cyclictest16263-1kworker/3:0H21:15:003
609999180,16cyclictest590-21latency_hist23:45:003
609899188,8cyclictest31374-21cron23:40:002
609899187,2cyclictest111rcu_preempt23:10:082
609899183,12cyclictest18497-21cut19:45:012
609899182,12cyclictest4560-21snmp_o5powerctl22:15:212
6098991814,2cyclictest25142-21/usr/sbin/munin23:20:112
6098991814,2cyclictest23741-21open_files21:35:182
6098991814,2cyclictest1957-21interrupts20:30:162
6098991814,2cyclictest12281-21netstat19:25:162
6098991812,4cyclictest32124-21interrupts23:40:142
6098991812,4cyclictest23836-21expr23:15:122
6098991811,5cyclictest9033-21cat22:30:152
6098991811,5cyclictest805-21dbus-daemon20:00:002
6098991811,5cyclictest24603-21apt21:40:102
6098991811,5cyclictest20176-21basename21:25:122
6098991811,5cyclictest15136-21df22:50:122
609899181,13cyclictest20452-21apt-get19:50:102
609899181,12cyclictest838-21apt-get23:45:122
609899181,12cyclictest838-21apt-get23:45:112
609899181,12cyclictest29458-21apt-get21:55:102
609899181,12cyclictest1415-21snmpd22:45:122
609899181,12cyclictest13236-21apt-get21:05:122
6098991810,6cyclictest24990-21sed20:00:252
6098991810,6cyclictest24990-21sed20:00:252
609899180,4cyclictest0-21swapper/200:05:082
609799188,7cyclictest23713-21apt-get20:00:131
609799188,7cyclictest23713-21apt-get20:00:131
609799182,13cyclictest4468-21sh20:39:591
6097991814,2cyclictest15905-21hwlatdetect00:30:141
6097991811,5cyclictest7876-21irqstats00:05:161
609799181,13cyclictest26948-21apt-get20:10:101
609799181,13cyclictest11563-21apt-get19:25:111
609799181,12cyclictest5749-21apt-get00:00:111
609799181,12cyclictest4109-21apt-get23:55:121
609799181,12cyclictest1952-21apt-get22:10:121
6097991810,6cyclictest30997-21cut22:00:001
609599189,6cyclictest20938-21cat19:50:140
609599188,8cyclictest4425-21tune2fs23:55:140
609599182,10cyclictest10977-21munin-run21:00:000
6095991814,2cyclictest8098-21nfs4_client00:05:190
6095991814,2cyclictest27865-21sendmail_mailqu20:10:170
6095991814,2cyclictest10890-21pmu-power22:35:200
6095991814,2cyclictest10890-21pmu-power22:35:190
6095991813,3cyclictest9058-21missed_timers22:30:160
6095991813,3cyclictest27678-21sendmail_mailqu23:25:190
6095991813,3cyclictest2740-21open_inodes22:10:190
6095991813,3cyclictest18673-21latency21:20:160
6095991812,4cyclictest3927-21grep22:15:150
6095991812,4cyclictest16890-21hostname21:15:140
6095991812,4cyclictest16890-21hostname21:15:140
6095991812,3cyclictest1599-21ntp_kernel_pll_23:45:170
6095991812,3cyclictest1599-21ntp_kernel_pll_23:45:160
6095991811,5cyclictest23285-21fw_packets21:35:140
609599181,13cyclictest21344-21/usr/sbin/munin21:30:260
6095991810,6cyclictest9836-21awk00:10:200
6095991810,6cyclictest12988-21sed21:05:010
6095991810,5cyclictest17006-21cat19:40:120
6099991715,1cyclictest21649-21apt-config23:10:023
609999171,14cyclictest20528-21tr19:50:103
6099991711,4cyclictest19961-21tune2fs21:25:113
6099991711,4cyclictest10354-21grep19:20:143
609899172,13cyclictest26752-21grep20:10:012
6098991713,2cyclictest32712-21/usr/sbin/munin22:05:242
6098991713,2cyclictest32222-21/usr/sbin/munin20:25:202
6098991713,2cyclictest2488-21/usr/sbin/munin23:50:182
6098991713,2cyclictest12753-21interrupts00:20:152
6098991711,4cyclictest20028-21expr23:05:122
609899171,11cyclictest3076-21apt-get20:35:122
6098991710,5cyclictest7154-21cut20:45:182
6098991710,5cyclictest5656-21sed00:00:002
6098991710,5cyclictest27466-21sed20:10:142
6098991710,5cyclictest18624-21mktemp19:45:002
609799179,6cyclictest14276-21tr22:45:201
609799173,12cyclictest25715-21wc21:40:231
609799171,2cyclictest208-1kworker/1:1H00:30:001
6097991712,3cyclictest11641-21grep22:40:011
6097991712,3cyclictest11641-21grep22:40:001
6097991711,4cyclictest2581-21tr23:50:141
6097991711,3cyclictest31497-21cut22:00:131
609799171,12cyclictest791-21snmp_o5powerctl20:25:241
609799171,12cyclictest6218-21snmp_o5powerctl22:20:221
609799171,12cyclictest19104-21snmp_o5powerctl21:20:211
609799171,12cyclictest16655-21latency_hist19:40:001
6097991710,5cyclictest30439-21basename23:35:151
6097991710,5cyclictest10166-21grep00:10:261
609799170,5cyclictest0-21swapper/121:50:121
609799170,15cyclictest0-21swapper/100:20:161
609599178,7cyclictest20353-21cat23:05:160
6095991711,4cyclictest30129-21expr23:35:130
609599171,12cyclictest25603-21snmp_o5powerctl21:40:220
609899169,5cyclictest31567-21cat22:00:142
609899169,5cyclictest21478-21expr21:30:132
6098991612,2cyclictest6139-21sendmail_mailqu22:20:192
6098991610,4cyclictest16153-21grep21:10:232
609799169,5cyclictest5687-21tr20:40:181
609799169,5cyclictest16830-21fschecks_time22:55:141
609799168,5cyclictest26946-21df23:25:121
6097991614,1cyclictest0-21swapper/120:31:551
6097991612,2cyclictest8741-21idleruntime19:15:151
609799161,11cyclictest19730-21apt-get21:25:091
6097991610,4cyclictest12062-21sort00:20:001
6095991613,2cyclictest11669-21cstates19:25:110
6095991610,4cyclictest8249-21tune2fs20:50:140
6095991610,4cyclictest19939-21forks21:25:110
609799157,6cyclictest21867-21basename19:55:001
609799154,9cyclictest8065-21cstates20:50:131
6097991513,1cyclictest0-21swapper/121:34:231
6097991510,3cyclictest1831-21mailstats23:45:201
6097991510,3cyclictest1831-21mailstats23:45:201
6095991512,2cyclictest5209-21/usr/sbin/munin22:20:250
6095991511,2cyclictest12256-21apt00:20:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional