brendangregg / bpf-perf-tools-book Goto Github PK

Official repository for the BPF Performance Tools book

Python 100.00%

bpf-perf-tools-book's Introduction

BPF Performance Tools

This is the official repository of BPF (eBPF) tools from the book BPF Performance Tools: Linux and Application Observability. The directories are:

originals: The original published version of the tools.
updated: Updated versions of the tools.
exercises: Exercise solutions.

These tools are documented in the book.

Tools:

In red are many of the new tools developed for the book, which are all in this repository. In black are the prior tools from BCC and bpftrace, many of which are also covered in the book.

Updated tools

You may contribute updated versions of the tools to the updated/ directory as you find they need fixes for newer kernels or other distributions (which can change CONFIG options or library paths, changing the probes). Please avoid ifdef's where possible, and create separate tools as they can be tested and maintained independently. Filename examples:

opensnoop_5.4.bt: opensnoop.bt for Linux 5.4 onwards
opensnoop_redfrog.bt: opensnoop for the RedFrog Linux distribution (I made that up).
opensnoop_redfrog11.bt: opensnoop for RedFrog release 11 onwards.
opensnoop_redfrog11_5.4.bt: opensnoop for RedFrog release 11 onwards, with Linux 5.4 onwards.

By contributing updates to this repository, you agree that the publisher has the necessary permissions to include your updates in possible later editions of the book. Attribution will be given: make it clear in the PR what your name is. Note that the tools are deliberately short to serve as textbook examples and to simplify maintenance, and updates should not add functionality. For that, see the next section.

Enhanced tools

A small selection of tools from this book has been added to the bpftrace and BCC repositories. It is expected that the BCC versions will be enhanced with various command line options. We are not planning to add any more of these tools to those repositories: they are limited to ensure that they can be well supported by the repository community.

bpf-perf-tools-book's People

Contributors

Stargazers

Watchers

Forkers

aaron-ai bollwarm missfall nalinikanth sjanulonoks dgq2011 bianyuan1986 gmarler chaitanyaphalak turningme ronaldosaheki hdhoang bogsi madhuakula sasg processren thomaschr zerolugithub xiazemin hanbinglengyue terry-xf fenix0111 fangjiedevp anjali05 prtkgpta guoqingjiang-linux jdel12 ppltools cmvelo memacs netman2k nullllun learn-knowlege netcodergit silence2012 corvinux wjn740 binchenx prashantpcm94 bobbyzhu rhsyang lyx634449800 sthagen lanzafame aebm abhay2101 lirulei reborn2266 f3746307 fly542 dankamongmen binnyrs mckayhou easyfmxu soldierxue rayhassan zhanglei002 coyang leemgs li--paul markhorsfield albertygu jimmu76 melodylail comeback1986 weiwangblog slashtpa louisfeng valer-cara olivierh59500 fengyindiehun gongshijun wubo009 oliveiradan crixalis2013 zayat guanyuding rikeyone x-ops joelone koh1 zhangbinghua ingenicc zenosslk rachelsqh lzqtim nuc630 lic17 lizhanhui yydzhou desmax74 wuyouqian96169 x1b2j yanchdh feiyunwill op-y rajkrishnamurthy hanchengqin tongzhongkai pengdonglin137

bpf-perf-tools-book's Issues

jnistacks.bt

In jnistacks.bt, the call to jmaps throws "Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes". The jmaps by itself works fine. Tested out for multiple jvms. Ubuntu 16.04, kernel 4.15 and bpftrace-0.9.4

iosched.bt: Linux 5.0

bpftrace v0.9-236-g012e

# bpftrace iosched.bt 
Attaching 5 probes...
cannot attach kprobe, probe entry may not exist
Warning: could not attach probe kprobe:blk_start_request, skipping.
cannot attach kprobe, probe entry may not exist
Error attaching probe: 'kprobe:__elv_add_request'

Variable scoping inside if statements

Related to bpftrace/bpftrace#686

There's only one script which defines variables inside an if-block and then uses them outside the block:

bpf-perf-tools-book/Ch10_Networking/tcpretrans.bt

Line 53 in d928209

$daddr = ntop($sk->__sk_common.skc_daddr);

Ideally what this script does should be fine, since $daddr and $saddr and defined in both branches of the if/else, but I think it'd be very hard (impossible?) to confirm this is true in the general case at compile time, so we should require variables to be declared in the scope they are used in, or a parent scope, like C does.

To be compatible with a future bpftrace which has variable scoping, this bit of code:

    if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }

should be changed to:

    $daddr = 0;
    $saddr = 0;
    if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }

In section 18.1.1.2 Action Performed, it is unclear how out.txt was set up

Hi!

In Table 18-2 bpftrace Per-Event Costs, the exact setup of the out.txt file in row 18 (the entry with the "per event" test purpose) is a bit unclear.

As far as I know, > out.txt is not a valid bpftrace syntax. So my guess is that it is just a note to the reader that the output should be redirected. However, I'm not 100% sure so I'd to leave a note about it here in case someone would like to correct me.

Thanks!

minor errata: ch. 9 "strategy" section is misnumbered

From the Safari version:

Scripts throwing LLVM OOM

Found two scripts affected by bpftrace/bpftrace#528:

soaccept
soconnect

There might be other scripts affected by this.

Slow startup of skblife (and possible others)

skblife is taking >3 seconds to start up. Using perf it shows that most of the time is spent on resolve_kname. I opened a PR to fix that (bpftrace/bpftrace#765).

Update tools using ntop to support IPv6 addresses

These are the tools that need updating:

soconnect.bt
soaccept.bt
udpconnect.bt

Ch07_Memory/mmapsnoop.py: typo in comment

I think a comment in that file should point to Chapter 7, not Chapter 11.

I'd be willing to submit a PR that fixes this. Should I provide a new updated/Ch07_Memory/mmapsnoop.py or a changed originals/Ch07_Memory/mmapsnoop.py, if applicable?

5.9.2 bpftrace -l 'usdt:*' doesnt run on new version.

book section 5.9.2, page 150,
quote"""

bpftrace -l 'usdt:/usr/local/cpython/python'

usdt:/usr/local/cpython/python:line
usdt:/usr/local/cpython/python:function__entry
usdt:/usr/local/cpython/python:function__return
......
"""

on my env:
Oracle Linux 7.9
kernel: 5.4.17-2136.304.1.el7uek.x86_64

bpftrace-0.13.0-2.el7.x86_64
$bpftrace --version
bpftrace v0.13.0

[root@EL7 ~]$bpftrace -l 'usdt:/usr/bin/python'
stdin:1:1-21: ERROR: usdt probe type requires 2 or 3 arguments
usdt:/usr/bin/python


seems bpftrace -l parameter changes in newer versions.

blostacks.bt kprobe:blk_start_request

This function was removed in 5.0 (as 5.0 went multi-queue only).

Warning: could not attach probe kprobe:blk_start_request, skipping.

The tool works anyway, since it traces both old and new functions. It just prints a warning on 5.0+, which is ok. I'm going to assume that this behavior -- printing a warning -- is not going to change, and document it as such.

tcplife: ICmp Assertion abort

This tool worked not that long ago, but now hits an Assertion:

# tcplife.bt 
bpftrace: /usr/include/llvm-7/llvm/IR/Instructions.h:1117: void llvm::ICmpInst::AssertOK(): Assertion `getOperand(0)->getType() == getOperand(1)->getType() && "Both operands to ICmp instruction are not of the same type!"' failed.
Aborted (core dumped)

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fffef922801 in __GI_abort () at abort.c:79
#2  0x00007fffef91239a in __assert_fail_base (fmt=0x7fffefa997d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x5555557ebbc8 "getOperand(0)->getType() == getOperand(1)->getType() && \"Both operands to ICmp instruction are not of the same type!\"", 
    file=file@entry=0x5555557eba78 "/usr/include/llvm-7/llvm/IR/Instructions.h", line=line@entry=1117, function=function@entry=0x5555557ed820 <llvm::ICmpInst::AssertOK()::__PRETTY_FUNCTION__> "void llvm::ICmpInst::AssertOK()")
    at assert.c:92
#3  0x00007fffef912412 in __GI___assert_fail (assertion=0x5555557ebbc8 "getOperand(0)->getType() == getOperand(1)->getType() && \"Both operands to ICmp instruction are not of the same type!\"", 
    file=0x5555557eba78 "/usr/include/llvm-7/llvm/IR/Instructions.h", line=1117, function=0x5555557ed820 <llvm::ICmpInst::AssertOK()::__PRETTY_FUNCTION__> "void llvm::ICmpInst::AssertOK()") at assert.c:101
#4  0x000055555578d2a6 in llvm::ICmpInst::AssertOK (this=0x5555562f1810) at /usr/include/llvm-7/llvm/IR/Instructions.h:1116
#5  0x000055555578d3a2 in llvm::ICmpInst::ICmpInst (this=0x5555562f1810, pred=llvm::CmpInst::ICMP_NE, LHS=0x555556157e10, RHS=0x55555604b8c0, NameStr=...) at /usr/include/llvm-7/llvm/IR/Instructions.h:1171
#6  0x00005555557922d1 in llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateICmp (this=0x7fffffffe618, P=llvm::CmpInst::ICMP_NE, LHS=0x555556157e10, RHS=0x55555604b8c0, Name=...)
    at /usr/include/llvm-7/llvm/IR/IRBuilder.h:1851
#7  0x00005555557924b2 in llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateICmpNE (this=0x7fffffffe618, LHS=0x555556157e10, RHS=0x55555604b8c0, Name=...) at /usr/include/llvm-7/llvm/IR/IRBuilder.h:1741
#8  0x0000555555782956 in bpftrace::ast::CodegenLLVM::createLogicalAnd (this=0x7fffffffe5f0, binop=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1480
#9  0x000055555577d09c in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, binop=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:793
#10 0x0000555555771bae in bpftrace::ast::Binop::accept (this=0x7fffe851c180, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:45
#11 0x000055555577fc19 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, if_block=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1159
#12 0x0000555555771de0 in bpftrace::ast::If::accept (this=0x7fffe8a35eb0, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:89
#13 0x0000555555780be4 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, probe=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1290
#14 0x0000555555771e48 in bpftrace::ast::Probe::accept (this=0x555556384480, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:97
#15 0x00005555557817e5 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, program=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1377
#16 0x0000555555771e7c in bpftrace::ast::Program::accept (this=0x7fffe8011da0, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:101
#17 0x0000555555785037 in bpftrace::ast::CodegenLLVM::compile (this=0x7fffffffe5f0, debug=<incomplete type>, out=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1678
#18 0x000055555574ccf1 in main (argc=2, argv=0x7fffffffea38) at /home/bgregg/Build/bpftrace/src/main.cpp:417

bpftrace executable may not reside in /usr/local/bin/bpftrace

apt-get install bpftrace on Debian installs the executable /usr/bin/bpftrace whereas various scripts *.py assume /usr/local/bin/bpftrace.

The discrepancy should be solved by employing the generic shebang #!/usr/bin/env bpftrace instead of #!/usr/local/bin/bpftrace througout. Please let me know if I should submit a corresponding PR (following PR #26).

Solutions of optionnal exercices

Hi,

Is there any possibilty to have the answers/solutions for the optionnal exercices at the end of some chapters ?
There are really helpful to know if the usage of bcc tools/bpftrace has been well understand but without the solution i dont know exactly if im right ..

Ch07_Memory/vmscan.bt: tracepoint:vmscan:mm_vmscan_writepage no longer available

mm_vmscan_writepage is no longer available in recent kernels; vmscan.bt fails with "ERROR: tracepoint not found: vmscan:mm_vmscan_writepage" e.g. with Kernel 6.1.0-16-amd64.

Hence presumably Ch07_Memory/vmscan.bt should be refactored to make use of mm_vmscan_write_folio instead.

ttysnoop.bt does not run properly

When run as bpftrace ttysnoop.bt pts2 it returns

$1 used numerically, but given "pts2". Try using str($1).

I have tried to look into str($1 + 3) with printf("arg+3 is %s\n", str($1 + 3)); with an exit right after. I wasn't able unfortunately to get my argument string as expected, or the argument from the 3rd char on, I would only get arg+3 is .
Interestingly, the BCC counterpart works

scread.bt doesn't name files

Having copied the source from this repo just to ensure I wasn't fat-fingering it, the output of scread appears as:

[root@aws-instance build]# scread.bt
Attaching 1 probe...
^C

@filename[]: 20

On bpftrace v0.9.2-247-g4fe9, built about 15 mins ago.
Kernel 4.18.0-147.3.1.el8_1.x86_64
How might I approach troubleshooting this?

tcpconnect-tp.bt doesn't work in Ubuntu 18.04 w/ 4.19 kernel

ubuntu@ip-172-31-18-109:~$ sudo bpftrace tcpconnect-tp.bt 
tcpconnect-tp.bt:30:38-69: ERROR: ntop() expects an integer or array argument, got ctx
        printf("%-15s %-15s %-5d\n", ntop(args->family, args->saddr),
                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tcpconnect-tp.bt:31:13-44: ERROR: ntop() expects an integer or array argument, got ctx
            ntop(args->family, args->daddr), args->dport)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

environment

ubuntu@ip-172-31-18-109:~$ uname -sr
Linux 4.19.0-041900-generic

ubuntu@ip-172-31-18-109:~$ bpftrace --version
bpftrace v0.9.4-97-gef39

Ch07_Memory/mmapsnoop.py: __close_fd() has been renamed into close_fd() in recent kernels

applies e.g. to kernel 6.1.0-16-amd64

updated Links giving 404

kvmexits.bt

When running kvmexits.bt on Ubuntu 20.04/5.4.0-31-generic, I'm not getting correct exit reason info from kvm_exit calls. Any idea what could be going wrong?

Example output below:

$ sudo bpftrace kvmexits.bt
Attaching 4 probes...
Tracing KVM exits. Ctrl-C to end

@exit_ns[97, ]:
[256, 512)             1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

@exit_ns[119, ]:
[512, 1K)              1 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |
[1K, 2K)               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2K, 4K)               1 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |

@exit_ns[4, ]:
[256, 512)            28 |@@@@@@@                                             |
[512, 1K)            188 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

@exit_ns[114, ]:
[256, 512)           205 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)             37 |@@@@@@@@@                                           |
[1K, 2K)               6 |@                                                   |

@exit_ns[20, VMLAUNCH]:
[512, 1K)             33 |@@@@@@@@@                                           |
[1K, 2K)              54 |@@@@@@@@@@@@@@@                                     |
[2K, 4K)             116 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[4K, 8K)             187 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)             42 |@@@@@@@@@@@                                         |

@exit_ns[100, ]:
[256, 512)           178 |@@@@@@@@@                                           |
[512, 1K)            986 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)              10 |                                                    |

@exit_ns[1024, ]:
[512, 1K)              1 |                                                    |
[1K, 2K)              35 |@                                                   |
[2K, 4K)             139 |@@@@@                                               |
[4K, 8K)            1234 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)            297 |@@@@@@@@@@@@                                        |
[16K, 32K)            27 |@                                                   |
[32K, 64K)             1 |                                                    |

@exit_ns[96, ]:
[256, 512)           932 |@@@@@@@@@@@@@@                                      |
[512, 1K)           3458 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)            1866 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@                        |
[2K, 4K)             150 |@@                                                  |
[4K, 8K)               5 |                                                    |
[8K, 16K)              1 |                                                    |

@exit_ns[24, VMRESUME]:
[256, 512)         13683 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)           1612 |@@@@@@                                              |
[1K, 2K)              68 |                                                    |
[2K, 4K)               1 |                                                    |

@exit_ns[124, ]:
[256, 512)          6768 |@@@@@                                               |
[512, 1K)          59247 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)           36592 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[2K, 4K)           11950 |@@@@@@@@@@                                          |
[4K, 8K)            1000 |                                                    |
[8K, 16K)           5791 |@@@@@                                               |
[16K, 32K)          6892 |@@@@@@                                              |
[32K, 64K)             3 |                                                    |

@exit_ns[123, ]:
[256, 512)        111656 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)          10590 |@@@@                                                |
[1K, 2K)             243 |                                                    |
[2K, 4K)           85229 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@             |
[4K, 8K)           35191 |@@@@@@@@@@@@@@@@                                    |
[8K, 16K)           2102 |                                                    |
[16K, 32K)             1 |                                                    |

@exit_ns[120, ]:
[256, 512)            89 |                                                    |
[512, 1K)           3561 |                                                    |
[1K, 2K)            7425 |@                                                   |
[2K, 4K)            9633 |@@                                                  |
[4K, 8K)           13830 |@@@                                                 |
[8K, 16K)           9828 |@@                                                  |
[16K, 32K)         14671 |@@@                                                 |
[32K, 64K)          9344 |@@                                                  |
[64K, 128K)         7860 |@@                                                  |
[128K, 256K)        9456 |@@                                                  |
[256K, 512K)       45928 |@@@@@@@@@@@@                                        |
[512K, 1M)        193559 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1M, 2M)              53 |                                                    |

6.3.10 syscount

The -P option can be used to count by process ID instead:

should be lower case p, i.e.

The -p option can be used to count by process ID instead:

Also includes the command line sample:

#syscount -Pi 1