Skip to main content

Linux Systems Performance/Observability (BPF (bpfcc-tools), BCC Tools

Linux System Performance/Observability Tools

BPF (bpfcc-tools), BCC Tools, and Complete Observability Toolkit

Assuming you have Linux Server in place and have the required BPF aka BCC related packages installed on the system(s) for Linux distribution.

BPF(eBPF) aka BCC Tools (bpfcc-tools):

  1. BPF, which originally stood for Berkley Packet Filter is the dynamic tracing tools for Linux Systems.
  2. BPF initially used for the speeding up for the tcpdump expressions and since then it has been know as the extended Berkley packet Filter (eBPF).
  3. Its new uses are Tracing Tools where it provides programmability for the BPF Compiler Collection (BCC) and bpftrace front ends.
  4. Example: execsnoop, biosnoop etc is a BCC Tool.
  5. When facing production performance crisis these such list of tools comes handy to trace and fix the issue. However, it requires certain KERNEL level config options to be enabled such as CONFIG_FTRACE, CONFIG_BPF.
  6. Profiling tools typically required complied version of all packages to run properly on the systems.
Linux bcc/BPF Tracing Tools

Credits: Brendan Gregg

When facing production performance crisis these such list of tools comes handy to trace and fix the issue.

Here are the list of tools that will be handy when you wanted to fix issues for Prod or any other environments.

🚨 Production CRISIS Tools

#. Name - Provides

  1. procps - ps, vmstat, uptime, top
  2. util-linux - dmesg, lsblk, lscpu
  3. sysstat - iostat, mpstat, pidstat, sar
  4. iproute2 - ip, ss, nstat, tc
  5. numactl - numastat
  6. linux-tools-common - perf, turbostat
    linux-tools-$(uname -r)
  7. bcc-tools (aka bpfcc-tools) - opensnoop, execsnoop, runqlat, runqlen, softirqs, hardirqs, ext4slower, tcptop, ext4dist, biotop, biosnoop, biolatency, tcplife, trace, argdist, funccount, stackcount, profile etc..
  8. bpftrace - bpftrace, etc..
  9. perf-tools-unstable - ftrace version of opensnoop, execsnoop, iolatency, iosnoop, bitesize, kprobe, funccount
  10. trace-cmd - trace-cmd
  11. nicstat - nicstat
  12. ethtool - ethtool
  13. tiptop - tiptop (# apt install tiptop)
  14. msr-tools - rdmsr, wrmsr

🔧 Linux Application Debugging/Observability Tools

#. Tool Name - Description

  1. perf - CPU (Profiling | Flame Graphs), syscall tracing
  2. profile - CPU Profiling using timed sampling
  3. offcputime - Off-CPU profiling using Scheduler Tracing
  4. strace - Syscall Tracing
  5. execsnoop - New Process Tracing
  6. syscount - Syscall Counting
  7. bpftrace - Signal tracing, I/O profiling, Lock analysis

⚙️ Linux CPU Performance Debugging/Observability Tools

#. Tool Name - Description

  1. uptime - Load Averages (# cat /proc/pressure/cpu (10s, 60s & 300s))
  2. vmstat - Includes system-wide CPU Averages
  3. mpstat - Per-CPU Statistics
  4. sar - Historical Statistics
  5. ps - Process Status
  6. top - Monitor per-process/thread CPU usage
  7. pidstat - Per-process/thread CPU breakdowns
  8. time, ptime - time a command, with CPU breakdowns
  9. turboboost - Show CPU clock rate and other states
  10. showboot - Show CPU clock rate and turbo boost
  11. pmcarch - show high-level CPU cycle usage
  12. tlbstat - Summarize TLB Cycles
  13. perf - CPU profiling & PMC Analysis
  14. profile - Sample CPU Stack traces
  15. cpudist - Summarize on cpu-time
  16. runqlat - Summarize CPU run queue latency
  17. runqlen - Summarize CPU run queue length
  18. softirqs - Summarize soft Interrupt time
  19. hardirqs - Summarize hard Interrupt time
  20. bpftrace - Tracing programs for CPU analysis

💾 Linux MEMORY Performance Debugging/Observability Tools

#. Tool Name - Description

  1. vmstat - Virtual and Physical Memory statistics

    Ex: root@ip-172-31-21-94:~# vmstat -Sm 1

    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0    271     29    511    0    0    67   223   54   42  1  0 89  0 10
 0  0      0    271     29    511    0    0     0     0   40   27  0  0 99  0  1
 1  0      0    271     29    511    0    0     0     0   32   28  0  0 81  0 19
  2. PSI - Memory pressure stall information

    Ex: root@ip-172-31-21-94:~# cat /proc/pressure/memory

    some avg10=0.00 avg60=0.00 avg300=0.00 total=730880
full avg10=0.00 avg60=0.00 avg300=0.00 total=649756
  3. swapon - Swap Device Usage

    Ex: swapon

  4. sar - Historical Statistics
  5. slabtop - Kernel Slab Allocator Statistics

    Ex: root@ip-172-31-21-94:~# slabtop -sc

    Active / Total Objects (% used)    : 245106 / 253954 (96.5%)
Active / Total Slabs (% used)      : 6869 / 6869 (100.0%)
Active / Total Caches (% used)     : 312 / 370 (84.3%)
Active / Total Size (% used)       : 61201.99K / 63941.81K (95.7%)
Minimum / Average / Maximum Object : 0.01K / 0.25K / 10.12K

 OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
12339  12188  98%    1.16K    457       27     14624K ext4_inode_cache
 7975   7886  98%    0.62K    319       25      5104K inode_cache
30772  30772 100%    0.14K   1099       28      4396K kernfs_node_cache
22953  21881  95%    0.19K   1093       21      4372K dentry
  6. numastat - NUMA Statistics

    Ex: root@ip-172-31-21-94:~# numastat

                               node0
numa_hit                 1673622
numa_miss                      0
numa_foreign                   0
interleave_hit                77
local_node               1673622
other_node                     0
  7. ps - Process Status

    Ex:

    # ps aux
# ps -eo pid,pmem,vsz,rss,comm
  8. top - Monitor Per Process memory usage

    Ex: # top -o %MEM

  9. pmap - Process address space statistics

    Ex: pmap -x <pid>

  10. perf - Memory PMC and tracepoint analysis

    Ex: Sample page faults (RSS Growth) with stack traces system wide, until Ctrl-C

    root@ip-172-31-21-94:~# perf record -e page-faults -a -g
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.135 MB perf.data (3 samples) ]
  11. drsnoop - Direct reclaim tracing (BCC tool)
  12. wss - Working set size estimation (shows how a working set size be measured via PTE page table entry)
  13. bpftrace - Tracing Programs for memory analysis (BPF based tracer)

📁 File System Performance Debugging/Observability Tools

#. Tool Name - Description

  1. mount - List file system and their mount flags
  2. free - Cache capacity statistics
  3. top - Includes memory usage summary
  4. vmstat - Virtual memory statistics
  5. sar - Various statistics including historic
  6. slabtop - Kernel slab allocator statistics
  7. strace - System call tracing
  8. fatrace - Trace file system operations using fanotify
  9. latencytop - Show system-wide latency sources
  10. opensnoop - Traces file opened
  11. filetop - Top files in by IOPS
  12. cachestat - Page Cache Statistics
  13. ext4dist (xfs,zfs,btrfs,nfs) - Show ext4 operation latency distribution
  14. ext4slower (xfs,zfs,btrfs,nfs) - Show slow ext4 operations
  15. bpftrace - Custom file system tracing

💿 Disk Performance Debugging/Observability Tools

#. Tool Name - Description

  1. iostat - Various per-disk statistics
  2. sar - Historical disk statistics
  3. PSI - Disk Pressure stall information
  4. pidstat - Disk I/O usage by process
  5. perf - Record Block I/O tracepoints

    Ex:

    # perf list 'block:*'
# perf record -e block:block_rq_issue -a -g sleep 10
# perf script --header
  6. biolatency - Summarize Disk I/O latency as histogram
  7. biosnoop - Trace disk I/O with PID and latency
  8. iotop, biotop - Top for disks: summarize disk I/O by process
  9. biostacks - Show disk I/O with Initialization Stacks
  10. blktrace - Disk I/O event tracing
  11. bpftrace - Custom Disk Tracing

    Ex: Count block I/O tracepoint events:

    # bpftrace -e 'tracepoint:block:* { @[probe] = count(); }'
  12. smartctl - Disk controller statistics (Self-Monitoring, Analysis & Reporting Technology)

    Ex: Install and use

    # apt install smartmontools
# smartctl --all -d megaraid,0 /dev/xvda15

🌐 Network Performance Debugging/Observability Tools

#. Tool Name - Description

  1. ss - Socket statistics

    Ex: root@ip-172-31-21-94:~# ss -tiepm

    State  Recv-Q  Send-Q  Local Address:Port   Peer Address:Port   Process
ESTAB  0       52      172.31.21.94:ssh    103.252.203.93:7237  users:(("sshd",pid=1251,fd=4))
  timer:(on,217ms,0) ino:7097 sk:5e <->
  skmem:(r0,rb2142943,t0,tb87040,f3148,w948,o0,bl0,d25) ts sack cubic
  rto:223 rtt:22.584/29.792 mss:1448 cwnd:10 bytes_sent:1513930
  2. ip - Network interface & route statistics

    Ex: root@ip-172-31-21-94:~# ip -s link

    1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped  missed   mcast
        31750      264       0        0       0       0
  3. ifconfig - Network interface statistics
  4. nstat - Network stack statistics

    Ex: root@ip-172-31-21-94:~# nstat -s

    #kernel
IpInReceives                    19426              0.0
IpInDelivers                    19426              0.0
IpOutRequests                   15286              0.0
TcpActiveOpens                     27              0.0
TcpInSegs                       18280              0.0
TcpOutSegs                      14237              0.0
  5. netstat - Various network stack & interface statistics
  6. sar - Historical statistics

    Ex: root@ip-172-31-21-94:~# sar -n TCP 1

    Linux 6.8.0-1015-aws (ip-172-31-21-94)  11/18/24  _x86_64_  (1 CPU)

14:47:33  active/s passive/s    iseg/s    oseg/s
14:47:34      0.00      0.00      2.00      0.00
14:47:35      0.00      0.00      1.00      1.00
Average:      0.00      0.00      1.25      0.75
  7. nicstat - Network interface throughput and utilization

    Ex: root@ip-172-31-21-94:~# nicstat -z 1

        Time      Int   rKB/s   wKB/s   rPk/s   wPk/s    rAvs    wAvs %Util    Sat
14:49:35       lo    0.00    0.00    0.02    0.02   122.0   122.0  0.00   0.00
14:49:35     eth0   19.70    0.27   14.33    1.88  1408.4   144.7  0.00   0.00
  8. ethtool - Network interface driver statistics

    Ex: ethtool -i eth0 [-i option shows driver details & -k shows interface Tunables]

    # ethtool -k eth0
  9. tcplife - Trace TCP Session lifespans with connection details

    Ex: # tcplife

  10. tcptop - Show TCP throughput by Host and Process

    Ex: # tcptop

  11. tcpretrans - Trace TCP retransmits with address & TCP state

    Ex: # tcpretrans

  12. bpftrace - TCP/IP Stack Tracing: connections, packets, drops, latency

    Ex: Count socket accepts by PIDs and process name

    # bpftrace -e 't:syscalls:sys_enter_accept* { @[pid, comm] = count(); }'

# bpftrace -l 't:tcp:*'
tracepoint:tcp:tcp_bad_csum
tracepoint:tcp:tcp_cong_state_set
tracepoint:tcp:tcp_destroy_sock
tracepoint:tcp:tcp_probe
tracepoint:tcp:tcp_retransmit_skb
tracepoint:tcp:tcp_send_reset
  13. tcpdump - Network packet sniffer
  14. wireshark - Graphical network packet inspection

Benefits of knowing & using the BCC aka eBPF (bpfcc-tools) Tools:

  1. Can debug, identify and fix the issue within stipulated timelines.
  2. Provides dynamic tracing capabilities using BPF Tools
  3. Can use specific tools for the right system resource(s).
  4. And many more..

Ref: Brendan Gregg Online resources and books (BPF Performance Tools, Systems Performance).

Labels:
Location: Singapore

Comments

Post a Comment

Popular posts from this blog

Hacking via Cloning Site Using Kali Linux

Hacking via Cloning Site Using Kali Linux Hacking via Cloning Site Using Kali Linux  SET Attack Method : SET stands for Social Engineering Toolkist , primarily written by  David Kennedy . The Social-Engineer Toolkit (SET) is specifically designed to perform advanced attacks against the human element. SET was designed to be released with the  http://www.social-engineer.org  launch and has quickly became a standard tool in a penetration testers arsenal. The attacks built into the toolkit are designed to be targeted and focused attacks against a person or organization used during a penetration test. Actually this hacking method will works perfectly with DNS spoofing or Man in the Middle Attack method. Here in this tutorial I’m only writing how-to and step-by-step to perform the basic attack , but for the rest you can modified it with your own imagination. In this tutorial we will see how this attack methods can owned your com...
Post a Comment
Read more

Defacing Sites via HTML Injections (XSS)

Defacing Sites via HTML Injections Defacing Sites via HTML Injections What Is HTML Injection: "HTML Injection" is called as the Virtual Defacement Technique and also known as the "XSS" Cross Site Scripting. It is a very common vulnerability found when searched for most of the domains. This kind of a Vulnerability allows an "Attacker" to Inject some code into the applications affected in order to bypass access to the "Website" or to Infect any particular Page in that "Website". HTML injections = Cross Site Scripting, It is a Security Vulnerability in most of the sites, that allows an Attacker to Inject HTML Code into the Web Pages that are viewed by other users. XSS Attacks are essentially code injection attacks into the various interpreters in the browser. These attacks can be carried out using HTML, JavaScript, VBScript, ActiveX, Flash and other clinet side Languages. Well crafted Malicious Code can even hep the ...
Post a Comment
Read more

Hacking DNN Based Web Sites

Hacking DNN Based Web Sites Hacking DNN Based Web Sites Hacking DNN (Dot Net Nuke) CMS based websites is based on the Security Loop Hole in the CMS. For using that exploit we will see the few mentioned points which illustrates us on how to hack any live site based on Dot Net Nuke CMS. Vulnerability : This is the know Vulnerability in Dot Net Nuke (DNN) CMS. This allows aone user to Upload a File/Shell Remotely to hack that Site which is running on Dot Net Nuke CMS. The Link's for more Information regarding this Vulnerability is mentioned below -                                  http://www.exploit-db.com/exploits/12700/ Getting Started : Here we will use the Google Dork to trace the sites that are using DNN (Dot Net Nuke) CMS and are vulnerable to Remote File Upload. How To Do It : Here, I an mentioning the few points on how to Search for the existing Vulnerability in DNN. Let'...
Post a Comment
Read more

Excellent tricks and techniques of Google Hacks

Frontpage.. very nice clean search results listing !! I magine with me that you can steal or know the password of any web site designed by "Frontpage". But the file containing the password might be encrypted; to decrypt the file download the program " john the ripper". To see results; just write in the ( http://www.google.com/ ) search engine the code: "# -FrontPage-" inurl:service.pwd ============================================== This searches the password for "Website Access Analyzer", a Japanese software that creates webstatistics. To see results; just write in the ( http://www.google.com/ ) search engine the code: "AutoCreate=TRUE password=*" ============================================== This is a query to get inline passwords from search engines (not just Google), you must type in the query followed with the the domain name without the .com or .net. To see results; just write in the ( http://www.google.co...
Post a Comment
Read more

Hacking via BackTrack using SET Attack Method

Hacking via BackTrack using SET Attack Method Hacking via BackTrack using SET Attack  1. Click on Applications, BackTrack, Exploit Tools, Social Engineering Tools, Social Engineering Toolkit then select set.
Post a Comment
Read more