'\" t .\" Title: bpftrace .\" Author: [see the "AUTHOR(S)" section] .\" Generator: Asciidoctor 2.0.20 .\" Date: 2023-09-19 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" .TH "BPFTRACE" "8" "2023-09-19" "\ \&" "\ \&" .ie \n(.g .ds Aq \(aq .el .ds Aq ' .ss \n[.ss] 0 .nh .ad l .de URL \fI\\$2\fP <\\$1>\\$3 .. .als MTO URL .if \n[.g] \{\ . mso www.tmac . am URL . ad l . . . am MTO . ad l . . . LINKSTYLE blue R < > .\} .SH "NAME" bpftrace \- a high\-level tracing language .SH "SYNOPSIS" .sp \fBbpftrace\fP [\fIOPTIONS\fP] \fIFILENAME\fP .br \fBbpftrace\fP [\fIOPTIONS\fP] \-e \*(Aqprogram code\*(Aq .SH "DESCRIPTION" .sp bpftrace is a high\-level tracing language and runtime for Linux based on BPF. It supports static and dynamic tracing for both the kernel and user\-space. .sp When \fIFILENAME\fP is "\fI\-\fP", read from stdin. .SH "EXAMPLES" .sp List all probes with "sleep" in their name .RS 4 .RE .sp .if n .RS 4 .nf .fam C # bpftrace \-l \*(Aq*sleep*\*(Aq .fam .fi .if n .RE .sp Trace processes calling sleep .RS 4 .RE .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aqkprobe:do_nanosleep { printf("%d sleeping\(rsn", pid); }\*(Aq .fam .fi .if n .RE .sp Trace processes calling sleep while spawning \f(CRsleep 5\fP as a child process .RS 4 .RE .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aqkprobe:do_nanosleep { printf("%d sleeping\(rsn", pid); }\*(Aq \-c \*(Aqsleep 5\*(Aq .fam .fi .if n .RE .SH "SUPPORTED ARCHITECTURES" .sp x86_64, arm64 and s390x .SH "OPTIONS" .SS "Output format" .sp \fB\-B\fP \fIMODE\fP, Set the buffer mode for stdout. Valid values are .RS 4 \fBnone\fP No buffering. Each I/O is written as soon as possible .br \fBline\fP Data is written on the first newline or when the buffer is full. This is the default mode. .br \fBfull\fP Data is written once the buffer is full. .RE .sp \fB\-f\fP \fIFORMAT\fP, Set the output format. Valid values are .RS 4 \fBjson\fP .br \fBtext\fP .RE .sp \fB\-o\fP \fIFILENAME\fP .RS 4 Write bpftrace tracing output to \fIFILENAME\fP instead of stdout. This doesn\(cqt include child process (\fB\-c\fP option) output. Errors are still written to stderr. .RE .sp \fB\-\-no\-warnings\fP .RS 4 Suppress all warning messages created by bpftrace. .RE .SS "Tracing" .sp \fB\-e\fP \fIPROGRAM\fP .RS 4 Execute \fIPROGRAM\fP instead of reading the program from a file .RE .sp \fB\-I\fP \fIDIR\fP .RS 4 Add the directory \fIDIR\fP to the search path for C headers. This option can be used multiple times. .RE .sp \fB\-\-include\fP \fIFILENAME\fP .RS 4 Add \fIFILENAME\fP as an include for the pre\-processor. This is equal to adding \*(Aq#include \fIFILENAME\fP\*(Aq to the start bpftrace program. This option can be used multiple times. .RE .sp \fB\-l\fP [\fISEARCH\fP] .RS 4 List all probes that match the \fISEARCH\fP pattern. If the pattern is omitted all probes will be listed. This pattern supports wildcards in the same way that probes do. E.g. \*(Aq\-l kprobe:*file*\*(Aq to list all \*(Aqkprobes\*(Aq with \*(Aqfile\*(Aq in the name. For more details see the LISTING PROBES section. .RE .sp \fB\-\-unsafe\fP .RS 4 Some calls, like \*(Aqsystem\*(Aq, are marked as unsafe as they can have dangerous side effects (\*(Aqsystem("rm \-rf")\*(Aq) and are disabled by default. This flag allows their use. .RE .sp \fB\-k\fP .RS 4 Errors from bpf\-helpers(7) are silently ignored by default which can lead to strange results. This flag enables the detection of errors (except for errors from \*(Aqprobe_read_*\*(Aq). When errors occurs bpftrace will log an error containing the source location and the error code: .RE .sp .if n .RS 4 .nf .fam C stdin:48\-57: WARNING: Failed to probe_read_user_str: Bad address (\-14) u:lib.so:"fn(char const*)" { printf("arg0:%s\(rsn", str(arg0));} ~~~~~~~~~ .fam .fi .if n .RE .sp \fB\-kk\fP .RS 4 Same as \*(Aq\-k\*(Aq but also includes the errors from \*(Aqprobe_read_*\*(Aq helpers. .RE .SS "Process management" .sp \fB\-p\fP \fIPID\fP .RS 4 Attach to the process with \fIPID\fP. If the process terminates, bpftrace will also terminate. When using USDT probes they will be attached to only this process. .RE .sp \fB\-c\fP \fICOMMAND\fP .RS 4 Run \fICOMMAND\fP as a child process. When the child terminates bpftrace stops as well, as if \*(Aqexit()\*(Aq has been called. If bpftrace terminates before the child process does the child process will be terminated with a SIGTERM. If used, \*(AqUSDT\*(Aq probes these will only be attached to the child process. To avoid a race condition when using \*(AqUSDTs\*(Aq the child is stopped after \*(Aqexecve\*(Aq using \*(Aqptrace(2)\*(Aq and continued when all \*(AqUSDT\*(Aq probes are attached. .br The child PID is available to programs as the \*(Aqcpid\*(Aq builtin. .br The child process runs with the same privileges as bpftrace itself (usually root). .RE .sp \fB\-\-usdt\-file\-activation\fP .RS 4 activate usdt semaphores based on file path .RE .SS "Miscellaneous" .sp \fB\-\-info\fP .RS 4 Print detailed information about features supported by the kernel and the bpftrace build. .RE .sp \fB\-h, \-\-help\fP .RS 4 Print the help summary .RE .sp \fB\-V, \-\-version\fP .RS 4 Print bpftrace version information .RE .sp \fB\-v\fP .RS 4 verbose messages .RE .sp \fB\-d\fP .RS 4 debug mode .RE .sp \fB\-dd\fP .RS 4 verbose debug mode .RE .SH "ENVIRONMENT VARIABLES" .sp Some behavior can only be controlled through environment variables. This section lists all those variables. .SS "BPFTRACE_STRLEN" .sp Default: 64 .sp Number of bytes allocated on the BPF stack for the string returned by \f(CRstr()\fP. .sp Make this larger if you wish to read bigger strings with str(). .sp Beware that the BPF stack is small (512 bytes). .sp Support for even larger strings is [being discussed](\c .URL "https://github.com/iovisor/bpftrace/issues/305" "" ")." .SS "BPFTRACE_NO_CPP_DEMANGLE" .sp Default: 0 .sp C++ symbol demangling in user space stack traces is enabled by default. .sp This feature can be turned off by setting the value of this environment variable to \f(CR1\fP. .SS "BPFTRACE_MAP_KEYS_MAX" .sp Default: 4096 .sp This is the maximum number of keys that can be stored in a map. Increasing the value will consume more memory and increase startup times. There are some cases where you will want to: for example, sampling stack traces, recording timestamps for each page, etc. .SS "BPFTRACE_MAX_PROBES" .sp Default: 512 .sp This is the maximum number of probes that bpftrace can attach to. Increasing the value will consume more memory, increase startup times and can incur high performance overhead or even freeze or crash the system. .SS "BPFTRACE_CACHE_USER_SYMBOLS" .sp Default: PER_PROGRAM if ASLR disabled or \f(CR\-c\fP option given, PER_PID otherwise. .sp Caching strategy for user symbols. Valid values are: .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} PER_PROGRAM \- each program has its own cache. If there are more processes with enabled ASLR for a single program, this might produce incorrect results. .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} PER_PID \- each process has its own cache. This is accurate for processes with ASLR enabled, and enables bpftrace to preload caches for processes running at probe attachement time. If there are many processes running, it will consume a lot of a memory. .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} NONE \- caching disabled. This saves the most memory, but at the cost of speed. .RE .SS "BPFTRACE_VMLINUX" .sp Default: None .sp This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. If this value is not given, bpftrace searches vmlinux from pre defined locations. See src/attached_probe.cpp:find_vmlinux() for details. .SS "BPFTRACE_BTF" .sp Default: None .sp The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. See src/btf.cpp for the details. .SS "BPFTRACE_PERF_RB_PAGES" .sp Default: 64 .sp Number of pages to allocate per CPU for perf ring buffer. The value must be a power of 2. .sp If you\(cqre getting a lot of dropped events bpftrace may not be processing events in the ring buffer fast enough. It may be useful to bump the value higher so more events can be queued up. The tradeoff is that bpftrace will use more memory. .SS "BPFTRACE_MAX_BPF_PROGS" .sp Default: 512 .sp This is the maximum number of BPF programs (functions) that bpftrace can generate. The main purpose of this limit is to prevent bpftrace from hanging since generating a lot of probes takes a lot of resources (and it should not happen often). .SS "BPFTRACE_STR_TRUNC_TRAILER" .sp Default: \f(CR..\fP .sp Trailer to add to strings that were truncated. Set to empty string to disable truncation trailers. .SS "BPFTRACE_STACK_MODE" .sp Default: bpftrace .sp Output format for ustack and kstack builtins. Available modes/formats: \f(CRbpftrace\fP, \f(CRperf\fP, and \f(CRraw\fP. This can be overwritten at the call site. .SH "BPFTRACE LANGUAGE" .SS "Overview" .sp The \f(CRbpftrace\fP (\f(CRbt\fP) language is inspired by the D language used by \f(CRdtrace\fP and uses the same program structure. Each script consists of an preamble and one or more action blocks. .sp .if n .RS 4 .nf .fam C preamble actionblock1 actionblock2 .fam .fi .if n .RE .sp Preprocessor and type definitions take place in the preamble: .sp .if n .RS 4 .nf .fam C #include #define RED "\(rs033[31m" struct S { int x; } .fam .fi .if n .RE .sp Each action block consists of three parts: .sp .if n .RS 4 .nf .fam C probe[,probe] /predicate/ { action } .fam .fi .if n .RE .sp Probes .RS 4 A probe specifies the event and event type to attach too. .RE .sp Predicate .RS 4 The predicate is optional condition that must be met for the action to be executed. .RE .sp Action .RS 4 Actions are the programs that run when an event fires (and the predicate is met). An action is a semicolon (\f(CR;\fP) separated list of statements and always enclosed by brackets \f(CR{}\fP .RE .sp A basic script that traces the \f(CRopen(2)\fP and \f(CRopenat(2)\fP system calls can be written as follows: .sp .if n .RS 4 .nf .fam C BEGIN { printf("Tracing open syscalls... Hit Ctrl\-C to end.\(rsn"); } tracepoint:syscalls:sys_enter_open, tracepoint:syscalls:sys_enter_openat { printf("%\-6d %\-16s %s\(rsn", pid, comm, str(args.filename)); } .fam .fi .if n .RE .sp This script has two action blocks and a total of 3 probes. The first action block uses the special \f(CRBEGIN\fP probe, which fires once during \f(CRbpftrace\fP startup. This probe is used to print a header, indicating that the tracing has started. .sp The second action block uses two probes, one for \f(CRopen\fP and one for \f(CRopenat\fP, and defines an action that prints the file being \f(CRopen\fP ed as well as the \f(CRpid\fP and \f(CRcomm\fP of the process that execute the syscall. See the PROBES section for details on the available probe types. .SS "Identifiers" .sp Identifiers must match the following regular expression: \f(CR[_a\-zA\-Z][_a\-zA\-Z0\-9]*\fP .SS "Comments" .sp Both single line and multi line comments are supported. .sp .if n .RS 4 .nf .fam C // A single line comment i:s:1 { // can also be used to comment inline /* a multi line comment */ print(/* inline comment block */ 1); } .fam .fi .if n .RE .SS "Data Types" .sp The following fundamental integer types are provided by the language. .TS allbox tab(:); lt lt. T{ .sp \fBType\fP T}:T{ .sp \fBDescription\fP T} T{ .sp uint8 T}:T{ .sp Unsigned 8 bit integer T} T{ .sp int8 T}:T{ .sp Signed 8 bit integer T} T{ .sp uint16 T}:T{ .sp Unsigned 16 bit integer T} T{ .sp int16 T}:T{ .sp Signed 16 bit integer T} T{ .sp uint32 T}:T{ .sp Unsigned 32 bit integer T} T{ .sp int32 T}:T{ .sp Signed 32 bit integer T} T{ .sp uint64 T}:T{ .sp Unsigned 64 bit integer T} T{ .sp int64 T}:T{ .sp Signed 64 bit integer T} .TE .sp .SS "Floating\-point" .sp Floating\-point numbers are not supported by BPF and therefore not by bpftrace. .SS "Constants" .sp Integers constants can be defined in the following formats: .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} decimal (base 10) .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} octal (base 8) .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} hexadecimal (base 16) .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} scientific (base 10) .RE .sp Octal constants have to be prefixed with a \f(CR0\fP, e.g. \f(CR0123\fP. Hexadecimal constants start with either \f(CR0x\fP or \f(CR0X\fP, e.g. \f(CR0x10\fP. Scientific are written in the \f(CRe\fP format which is a shorthand for \f(CRm*10^n\fP, e.g. \f(CR$i = 2e3;\fP. Note that scientific literals are integer only due to the lack of floating point support, \f(CR1e\-3\fP is not valid. .sp To improve the readability of big literals a underscore \f(CR_\fP can be used as field separator, e.g. 1_000_123_000. .sp Integer suffixes as found in the C language are parsed by bpftrace to ensure compatibility with C headers/definitions but they\(cqre not used as size specifiers. \f(CR123UL\fP, \f(CR123U\fP and \f(CR123LL\fP all result in the same integer type with a value of \f(CR123\fP. .sp Character constants can be defined by enclosing the character in single quotes, e.g. \f(CR$c = \*(Aqc\*(Aq;\fP. .sp String constants can be defined by enclosing the character string in double quotes, e.g. \f(CR$str = "Hello world";\fP. .sp Characters and strings support the following escape sequences: .TS allbox tab(:); lt lt. T{ .sp \(rsn T}:T{ .sp Newline T} T{ .sp \(rst T}:T{ .sp Tab T} T{ .sp \(rs0nn T}:T{ .sp Octal value nn T} T{ .sp \(rsxnn T}:T{ .sp Hexadecimal value nn T} .TE .sp .SS "Type conversion" .sp Integer and pointer types can be converted using explicit type conversion with an expression like: .sp .if n .RS 4 .nf .fam C $y = (uint32) $z; $py = (int16 *) $pz; .fam .fi .if n .RE .sp Integer casts to a higher rank are sign extended. Conversion to a lower rank is done by zeroing leading bits. .sp It is also possible to cast between integers and integer arrays using the same syntax: .sp .if n .RS 4 .nf .fam C $a = (uint8[8]) 12345; $x = (uint64) $a; .fam .fi .if n .RE .sp Both the cast and the destination type must have the same size. When casting to an array, it is possible to omit the size which will be determined automatically from the size of the cast value. .SS "Operators and Expressions" .SS "Arithmetic Operators" .sp The following operators are available for integer arithmetic: .TS allbox tab(:); lt lt. T{ .sp + T}:T{ .sp integer addition T} T{ .sp \- T}:T{ .sp integer subtraction T} T{ .sp * T}:T{ .sp integer multiplication T} T{ .sp / T}:T{ .sp integer division T} T{ .sp % T}:T{ .sp integer modulo T} .TE .sp .SS "Logical Operators" .TS allbox tab(:); lt lt. T{ .sp && T}:T{ .sp Logical AND T} T{ .sp || T}:T{ .sp Logical OR T} T{ .sp ! T}:T{ .sp Logical NOT T} .TE .sp .SS "Bitwise Operators" .TS allbox tab(:); lt lt. T{ .sp & T}:T{ .sp AND T} T{ .sp | T}:T{ .sp OR T} T{ .sp ^ T}:T{ .sp XOR T} T{ .sp << T}:T{ .sp Left shift the left\-hand operand by the number of bits specified by the right\-hand expression value T} T{ .sp >> T}:T{ .sp Right shift the left\-hand operand by the number of bits specified by the right\-hand expression value T} .TE .sp .SS "Relational Operators" .sp The following relational operators are defined for integers and pointers. .TS allbox tab(:); lt lt. T{ .sp < T}:T{ .sp left\-hand expression is less than right\-hand T} T{ .sp <= T}:T{ .sp left\-hand expression is less than or equal to right\-hand T} T{ .sp > T}:T{ .sp left\-hand expression is bigger than right\-hand T} T{ .sp >= T}:T{ .sp left\-hand expression is bigger or equal to than right\-hand T} T{ .sp == T}:T{ .sp left\-hand expression equal to right\-hand T} T{ .sp != T}:T{ .sp left\-hand expression not equal to right\-hand T} .TE .sp .sp The following relation operators are available for comparing strings and integer arrays. .TS allbox tab(:); lt lt. T{ .sp == T}:T{ .sp left\-hand string equal to right\-hand T} T{ .sp != T}:T{ .sp left\-hand string not equal to right\-hand T} .TE .sp .SS "Assignment Operators" .sp The following assignment operators can be used on both \f(CRmap\fP and \f(CRscratch\fP variables: .TS allbox tab(:); lt lt. T{ .sp = T}:T{ .sp Assignment, assign the right\-hand expression to the left\-hand variable T} T{ .sp <<= T}:T{ .sp Update the variable with its value left shifted by the number of bits specified by the right\-hand expression value T} T{ .sp >>= T}:T{ .sp Update the variable with its value right shifted by the number of bits specified by the right\-hand expression value T} T{ .sp += T}:T{ .sp Increment the variable by the right\-hand expression value T} T{ .sp \-= T}:T{ .sp Decrement the variable by the right\-hand expression value T} T{ .sp *= T}:T{ .sp Multiple the variable by the right\-hand expression value T} T{ .sp /= T}:T{ .sp Divide the variable by the right\-hand expression value T} T{ .sp %= T}:T{ .sp Modulo the variable by the right\-hand expression value T} T{ .sp &= T}:T{ .sp Bitwise AND the variable by the right\-hand expression value T} T{ .sp |= T}:T{ .sp Bitwise OR the variable by the right\-hand expression value T} T{ .sp ^= T}:T{ .sp Bitwise XOR the variable by the right\-hand expression value T} .TE .sp .sp All these operators are syntactic sugar for combining assignment with the specified operator. \f(CR@ \-= 5\fP is equal to \f(CR@ = @ \- 5\fP. .SS "Increment and Decrement Operators" .sp The increment (\f(CR+`) and decrement (`\-\-`) operators can be used on integer and pointer variables to increment their value by one. They can only be used on variables and can either be applied as prefix or suffix. The difference is that the expression `x+\fP returns the original value of \f(CRx\fP, before it got incremented while \f(CR++x\fP returns the value of \f(CRx\fP post increment. E.g. .sp .if n .RS 4 .nf .fam C $x = 10; $y = $x\-\-; // y = 10; x = 9 $a = 10; $b = \-\-$a; // a = 9; b = 9 .fam .fi .if n .RE .sp Note that maps will be implicitly declared and initialized to 0 if not already declared or defined. Scratch variables must be initialized before using these operators. .SS "Variables and Maps" .sp bpftrace knows two types of variables, \f(CRscratch\fP and \f(CRmap\fP. .sp \*(Aqscratch\*(Aq variables are kept on the BPF stack and only exists during the execution of the action block and cannot be accessed outside of the program. Scratch variable names always start with a \f(CR$\fP, e.g. \f(CR$myvar\fP. .sp \*(Aqmap\*(Aq variables use BPF \*(Aqmaps\*(Aq. These exist for the lifetime of \f(CRbpftrace\fP itself and can be accessed from all action blocks and user\-space. Map names always start with a \f(CR@\fP, e.g. \f(CR@mymap\fP. .sp All valid identifiers can be used as \f(CRname\fP. .sp The data type of a variable is automatically determined during first assignment and cannot be changed afterwards. .SS "Associative Arrays" .sp Associative arrays are a collection of elements indexed by a key, similar to the hash tables found in languages like C++ (\f(CRstd::map\fP) and Python (\f(CRdict\fP). They\(cqre a variant of \*(Aqmap\*(Aq variables. .sp .if n .RS 4 .nf .fam C @name[key] = expression @name[key1,key2] = expression .fam .fi .if n .RE .sp Just like with any variable the type is determined on first use and cannot be modified afterwards. This applies to both the key(s) and the value type. .sp The following snippet creates a map with key signature \f(CR[int64, string[16]]\fP and a value type of \f(CRint64\fP: .sp .if n .RS 4 .nf .fam C @[pid, comm]++ .fam .fi .if n .RE .SS "Variable scoping" .SS "Pointers" .sp Pointers in bpftrace are similar to those found in \f(CRC\fP. .SS "Tuples" .sp bpftrace has support for immutable N\-tuples (\f(CRn > 1\fP). A tuple is a sequence type (like an array) where, unlike an array, every element can have a different type. .sp Tuples are a comma separated list of expressions, enclosed in brackets, \f(CR(1,2)\fP Individual fields can be accessed with the \f(CR.\fP operator. Tuples are zero indexed like arrays are. .sp .if n .RS 4 .nf .fam C i:s:1 { $a = (1,2); $b = (3,4, $a); print($a); print($b); print($b.0); } .fam .fi .if n .RE .sp Prints: .sp .if n .RS 4 .nf .fam C (1, 2) (3, 4, (1, 2)) 3 .fam .fi .if n .RE .SS "Arrays" .sp bpftrace supports accessing one\-dimensional arrays like those found in \f(CRC\fP. .sp Constructing arrays from scratch, like \f(CRint a[] = {1,2,3}\fP in \f(CRC\fP, is not supported. They can only be read into a variable from a pointer. .sp The \f(CR[]\fP operator is used to access elements. .sp .if n .RS 4 .nf .fam C struct MyStruct { int y[4]; } kprobe:dummy { $s = (struct MyStruct *) arg0; print($s\->y[0]); } .fam .fi .if n .RE .SS "Structs" .sp \f(CRC\fP like structs are supported by bpftrace. Fields are accessed with the \f(CR.\fP operator. Fields of a pointer to a struct can be accessed with the \f(CR\->\fP operator. .sp Custom struct can be defined in the preamble .sp Constructing structs from scratch, like \f(CRstruct X var = {.f1 = 1}\fP in \f(CRC\fP, is not supported. They can only be read into a variable from a pointer. .sp .if n .RS 4 .nf .fam C struct MyStruct { int a; } kprobe:dummy { $ptr = (struct MyStruct *) arg0; $st = *$ptr; print($st.a); print($ptr\->a); } .fam .fi .if n .RE .SS "Conditionals" .sp Conditional expressions are supported in the form of if/else statements and the ternary operator. .sp The ternary operator consists of three operands: a condition followed by a \f(CR?\fP, the expression to execute when the condition is true followed by a \f(CR:\fP and the expression to execute if the condition is false. .sp .if n .RS 4 .nf .fam C condition ? ifTrue : ifFalse .fam .fi .if n .RE .sp Both the \f(CRifTrue\fP and \f(CRifFalse\fP expressions must be of the same type, mixing types is not allowed. .sp The ternary operator can be used as part of an assignment. .sp .if n .RS 4 .nf .fam C $a == 1 ? print("true") : print("false"); $b = $a > 0 ? $a : \-1; .fam .fi .if n .RE .sp If/else statements, like the one in \f(CRC\fP, are supported. .sp .if n .RS 4 .nf .fam C if (condition) { ifblock } else if (condition) { if2block } else { elseblock } .fam .fi .if n .RE .SS "Loops" .sp Since kernel 5.3 BPF supports loops as long as the verifier can prove they\(cqre bounded and fit within the instruction limit. .sp In bpftrace loops are available through the \f(CRwhile\fP statement. .sp .if n .RS 4 .nf .fam C while (condition) { block; } .fam .fi .if n .RE .sp Within a while\-loop the following control flow statements can be used: .TS allbox tab(:); lt lt. T{ .sp continue T}:T{ .sp skip processing of the rest of the block and jump back to the evaluation of the conditional T} T{ .sp break T}:T{ .sp Terminate the loop T} .TE .sp .sp .if n .RS 4 .nf .fam C i:s:1 { $i = 0; while ($i <= 100) { printf("%d ", $i); if ($i > 5) { break; } $i++ } printf("\(rsn"); } .fam .fi .if n .RE .sp Loop unrolling is also supported with the \f(CRunroll\fP statement. .sp .if n .RS 4 .nf .fam C unroll(n) { block; } .fam .fi .if n .RE .sp The compiler will evaluate the block \f(CRn\fP times and generate the BPF code for the block \f(CRn\fP times. As this happens at compile time \f(CRn\fP must be a constant greater than 0 (\f(CRn > 0\fP). .sp The following two probes compile into the same code: .sp .if n .RS 4 .nf .fam C i:s:1 { unroll(3) { print("Unrolled") } } i:s:1 { print("Unrolled") print("Unrolled") print("Unrolled") } .fam .fi .if n .RE .SH "INVOCATION MODE" .sp There are three invocation modes for bpftrace built\-in functions. .TS allbox tab(:); lt lt lt. T{ .sp Mode T}:T{ .sp Description T}:T{ .sp Example functions T} T{ .sp Synchronous T}:T{ .sp The value/effect of the built\-in function is determined/handled right away by the bpf program in the kernel space. T}:T{ .sp \f(CRreg(), str(), ntop()\fP T} T{ .sp Asynchronous T}:T{ .sp The value/effect of the built\-in function is determined/handled later by the bpftrace process in the user space. T}:T{ .sp \f(CRprintf(), clear(), exit()\fP T} T{ .sp Compile\-time T}:T{ .sp The value of the built\-in function is determined before bpf programs are running. T}:T{ .sp \f(CRkaddr(), cgroupid(), offsetof()\fP T} .TE .sp .sp While BPF in the kernel can do a lot there are still things that can only be done from user space, like the outputting (printing) of data. The way bpftrace handles this is by sending events from the BPF program which user\-space will pick up some time in the future (usually in milliseconds). Operations that happen in the kernel are \*(Aqsynchronous\*(Aq (\*(Aqsync\*(Aq) and those that are handled in user space are \*(Aqasynchronous\*(Aq (\*(Aqasync\*(Aq) .sp The asynchronous behaviour can lead to some unexpected behavior as updates can happen before user space had time to process the event. The following situations may occur: .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} event loss: when using printf(), the amount of data printed may be less than the actual number of events generated by the kernel during BPF program\(cqs execution. .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} delayed exit: when using the exit() to terminate the program, bpftrace needs to handle the exit signal asynchronously casuing the BPF program may continue to run for some additional time. .RE .sp One example is updating a map value in a tight loop: .sp .if n .RS 4 .nf .fam C BEGIN { @=0; unroll(10) { print(@); @++; } exit() } .fam .fi .if n .RE .sp Maps are printed by reference not by value and as the value gets updated right after the print user\-space will likely only see the final value once it processes the event: .sp .if n .RS 4 .nf .fam C @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 .fam .fi .if n .RE .sp Therefore, when you need precise event statistics, it is recommended to use synchronous functions (e.g. count() and hist()) to ensure more reliable and accurate results. .SH "ADDRESS\-SPACES" .sp Kernel and user pointers live in different address spaces which, depending on the CPU architecture, might overlap. Trying to read a pointer that is in the wrong address space results in a runtime error. This error is hidden by default but can be enabled with the \f(CR\-kk\fP flag: .sp .if n .RS 4 .nf .fam C stdin:1:9\-12: WARNING: Failed to probe_read_user: Bad address (\-14) BEGIN { @=*uptr(kaddr("do_poweroff")) } ~~~ .fam .fi .if n .RE .sp bpftrace tries to automatically set the correct address space for a pointer based on the probe type, but might fail in cases where it is unclear. The address space can be changed with the \f(CRkptr()\fP and \f(CRuptr()\fP functions. .SH "BUILTINS" .sp Builtins are special variables built into the language. Unlike the scratch and map variable they don\(cqt need a \f(CR$\fP or \f(CR@\fP as prefix (except for the positional parameters). .TS allbox tab(:); lt lt lt lt lt. T{ .sp Variable T}:T{ .sp Type T}:T{ .sp Kernel T}:T{ .sp BPF Helper T}:T{ .sp Description T} T{ .sp \f(CR$1\fP, \f(CR$2\fP, \f(CR...$n\fP T}:T{ .sp int64 T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp The nth positional parameter passed to the bpftrace program. If less than n parameters are passed this evaluates to \f(CR0\fP. For string arguments use the \f(CRstr()\fP call to retrieve the value. T} T{ .sp \f(CR$#\fP T}:T{ .sp int64 T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp Total amount of positional parameters passed. T} T{ .sp \f(CRarg0\fP, \f(CRarg1\fP, \f(CR...argn\fP T}:T{ .sp int64 T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp nth argument passed to the function being traced. These are extracted from the CPU registers. The amount of args passed in registers depends on the CPU architecture. (kprobes, uprobes, usdt). T} T{ .sp \f(CRargs\fP T}:T{ .sp struct args T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp The struct of all arguments of the traced function. Available in \f(CRtracepoint\fP, \f(CRkfunc\fP, and \f(CRuprobe\fP (with DWARF) probes. Use \f(CRargs.x\fP to access argument \f(CRx\fP or \f(CRargs\fP to get a record with all arguments. T} T{ .sp cgroup T}:T{ .sp uint64 T}:T{ .sp 4.18 T}:T{ .sp get_current_cgroup_id T}:T{ .sp ID of the cgroup the current task is in. Only works with cgroupv2. T} T{ .sp comm T}:T{ .sp string[16] T}:T{ .sp 4.2 T}:T{ .sp get_current_com T}:T{ .sp \f(CRcomm\fP of the current task. Equal to the value in \f(CR/proc//comm\fP T} T{ .sp cpid T}:T{ .sp uint32 T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp PID of the child process T} T{ .sp numaid T}:T{ .sp uint32 T}:T{ .sp 5.8 T}:T{ .sp numa_node_id T}:T{ .sp ID of the NUMA node executing the BPF program T} T{ .sp cpu T}:T{ .sp uint32 T}:T{ .sp 4.1 T}:T{ .sp raw_smp_processor_id T}:T{ .sp ID of the processor executing the BPF program T} T{ .sp curtask T}:T{ .sp uint64 T}:T{ .sp 4.8 T}:T{ .sp get_current_task T}:T{ .sp Pointer to \f(CRstruct task_struct\fP of the current task T} T{ .sp elapsed T}:T{ .sp uint64 T}:T{ .sp (see nsec) T}:T{ .sp ktime_get_ns / ktime_get_boot_ns T}:T{ .sp Nanoseconds elapsed since bpftrace initialization, based on \f(CRnsecs\fP T} T{ .sp func T}:T{ .sp string T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp Name of the current function being traced (kprobes,uprobes) T} T{ .sp gid T}:T{ .sp uint64 T}:T{ .sp 4.2 T}:T{ .sp get_current_uid_gid T}:T{ .sp GID of current task T} T{ .sp kstack T}:T{ .sp kstack T}:T{ .sp T}:T{ .sp get_stackid T}:T{ .sp Kernel stack trace T} T{ .sp nsecs T}:T{ .sp uint64 T}:T{ .sp 4.1 / 5.7 T}:T{ .sp ktime_get_ns / ktime_get_boot_ns T}:T{ .sp nanoseconds since kernel boot. On kernels that support \f(CRktime_get_boot_ns\fP this includes the time spent suspended, on older kernels it does not. T} T{ .sp pid T}:T{ .sp uint64 T}:T{ .sp 4.2 T}:T{ .sp get_current_pid_tgid T}:T{ .sp Process ID (or thread group ID) of the current task. T} T{ .sp probe T}:T{ .sp string T}:T{ .sp n/na T}:T{ .sp n/a T}:T{ .sp Name of the current probe T} T{ .sp rand T}:T{ .sp uint32 T}:T{ .sp 4.1 T}:T{ .sp get_prandom_u32 T}:T{ .sp Random number T} T{ .sp retval T}:T{ .sp int64 T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp Value returned by the function being traced (kretprobe, uretprobe, kretfunc) T} T{ .sp \f(CRsarg0\fP, \f(CRsarg1\fP, \f(CR...sargn\fP T}:T{ .sp int64 T}:T{ .sp n/a T}:T{ .sp n/a T}:T{ .sp nth stack value of the function being traced. (kprobes, uprobes). T} T{ .sp tid T}:T{ .sp uint64 T}:T{ .sp 4.2 T}:T{ .sp get_current_pid_tgid T}:T{ .sp Thread ID of the current task. T} T{ .sp uid T}:T{ .sp uint64 T}:T{ .sp 4.2 T}:T{ .sp get_current_uid_gid T}:T{ .sp UID of current task T} T{ .sp ustack T}:T{ .sp ustack T}:T{ .sp 4.6 T}:T{ .sp get_stackid T}:T{ .sp Userspace stack trace T} .TE .sp .SH "MAP FUNCTIONS" .sp Map functions are built\-in functions who\(cqs return value can only be assigned to maps. The data type associated with these functions are only for internal use and are not compatible with the (integer) operators. .sp Functions that are marked \fBasync\fP are asynchronous which can lead to unexpected behavior, see the [Sync and Async] section for more information. .SS "avg" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRavg(int64 n)\fP .RE .sp Calculate the running average of \f(CRn\fP between consecutive calls. .sp .if n .RS 4 .nf .fam C i:s:1 { @x++; @y = avg(@x); print(@x); print(@y); } .fam .fi .if n .RE .sp Internally this keeps two values in the map: value count and running total. The average is computed in user\-space when printing by dividing the total by the count. .SS "clear" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRclear(map m)\fP .RE .sp \fBasync\fP .sp Clear all keys/values from map \f(CRm\fP. .sp .if n .RS 4 .nf .fam C i:ms:100 { @[rand % 10] = count(); } i:s:10 { print(@); clear(@); } .fam .fi .if n .RE .SS "count" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcount()\fP .RE .sp Count how often this function is called. .sp Using \f(CR@=count()\fP is conceptually similar to \f(CR@++\fP. The difference is that the \f(CRcount()\fP function uses a map type optimized for this (PER_CPU), increasing performance. Due to this the map cannot be accessed as a regular integer. .sp .if n .RS 4 .nf .fam C i:ms:100 { @ = count(); } i:s:10 { print(@); clear(@); } .fam .fi .if n .RE .SS "delete" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRdelete(mapkey k)\fP .RE .sp Delete a single key from a map. For a single value map this deletes the only element. For an associative\-array the key to delete has to be specified. .sp .if n .RS 4 .nf .fam C k:dummy { @scalar = 1; @associative[1,2] = 1; delete(@scalar); delete(@associative[1,2]); delete(@associative); // error } .fam .fi .if n .RE .SS "hist" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRhist(int64 n)\fP .RE .sp Create a log2 histogram of \f(CRn\fP. .sp .if n .RS 4 .nf .fam C kretprobe:vfs_read { @bytes = hist(retval); } .fam .fi .if n .RE .sp Results in: .sp .if n .RS 4 .nf .fam C @: [1M, 2M) 3 | | [2M, 4M) 2 | | [4M, 8M) 2 | | [8M, 16M) 6 | | [16M, 32M) 16 | | [32M, 64M) 27 | | [64M, 128M) 48 |@ | [128M, 256M) 98 |@@@ | [256M, 512M) 191 |@@@@@@ | [512M, 1G) 394 |@@@@@@@@@@@@@ | [1G, 2G) 820 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ | .fam .fi .if n .RE .SS "lhist" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRlhist(int64 n, int64 min, int64 max, int64 step)\fP .RE .sp Create a linear histogram of \f(CRn\fP. \f(CRlhist\fP creates \f(CRM\fP (\f(CR(max \- min) / step\fP) buckets in the range \f(CR[min,max)\fP where each bucket is \f(CRstep\fP in size. Values in the range \f(CR(\-inf, min)\fP and \f(CR(max, inf)\fP get their get their own bucket too, bringing the total amount of buckets created to \f(CRM+2\fP. .sp .if n .RS 4 .nf .fam C i:ms:1 { @ = lhist(rand %10, 0, 10, 1); } i:s:5 { exit(); } .fam .fi .if n .RE .sp Prints: .sp .if n .RS 4 .nf .fam C @: [0, 1) 306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [1, 2) 284 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [2, 3) 294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [3, 4) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [4, 5) 311 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [5, 6) 362 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [6, 7) 336 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [7, 8) 326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 9) 328 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [9, 10) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | .fam .fi .if n .RE .SS "max" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRmax(int64 n)\fP .RE .sp Update the map with \f(CRn\fP if \f(CRn\fP is bigger than the current value held. .SS "min" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRmin(int64 n)\fP .RE .sp Update the map with \f(CRn\fP if \f(CRn\fP is smaller than the current value held. .SS "stats" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRstats(int64 n)\fP .RE .sp \f(CRstats\fP combines the \f(CRcount\fP, \f(CRavg\fP and \f(CRsum\fP calls into one. .sp .if n .RS 4 .nf .fam C kprobe:vfs_read { @bytes[comm] = stats(arg2); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C @bytes[bash]: count 7, average 1, total 7 @bytes[sleep]: count 5, average 832, total 4160 @bytes[ls]: count 7, average 886, total 6208 @ .fam .fi .if n .RE .SS "sum" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRsum(int64 n)\fP .RE .sp Calculate the sum of all \f(CRn\fP passed. .SS "zero" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRzero(map m)\fP .RE .sp \fBasync\fP .sp Set all values for all keys to zero. .SH "FUNCTIONS" .sp Functions that are marked \fBasync\fP are asynchronous which can lead to unexpected behaviour, see the [sync and async] section for more information. .sp \fBcompile time\fP functions are evaluated at compile time, a static value will be compiled into the program. .sp \fBunsafe\fP functions can have dangerous side effects and should be used with care, the \f(CR\-\-unsafe\fP flag is required for use. .SS "bswap" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuint8 bswap(uint8 n)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuint16 bswap(uint16 n)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuint32 bswap(uint32 n)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuint64 bswap(uint64 n)\fP .RE .sp \f(CRbswap\fP reverses the order of the bytes in integer \f(CRn\fP. In case of 8 bit integers, \f(CRn\fP is returned without being modified. The return type is an unsigned integer of the same width as \f(CRn\fP. .SS "buf" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRbuf_t buf(void * data, [int64 length])\fP .RE .sp \f(CRbuf\fP reads \f(CRlength\fP amount of bytes from address \f(CRdata\fP. The maximum value of \f(CRlength\fP is limited to the \f(CRBPFTRACE_STRLEN\fP variable. For arrays the \f(CRlength\fP is optional, it is automatically inferred from the signature. .sp \f(CRbuf\fP is address space aware and will call the correct helper based on the address space associated with \f(CRdata\fP. .sp The \f(CRbuf_t\fP object returned by \f(CRbuf\fP can safely be printed as a hex encoded string with the \f(CR%r\fP format specifier. .sp Bytes with values >=32 and <=126 are printed using their ASCII character, other bytes are printed in hex form (e.g. \f(CR\(rsx00\fP). The \f(CR%rx\fP format specifier can be used to print everything in hex form, including ASCII characters. The similar \f(CR%rh\fP format specifier prints everything in hex form without \f(CR\(rsx\fP and with spaces between bytes (e.g. \f(CR0a fe\fP). .sp .if n .RS 4 .nf .fam C i:s:1 { printf("%r\(rsn", buf(kaddr("avenrun"), 8)); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C \(rsx00\(rsx03\(rsx00\(rsx00\(rsx00\(rsx00\(rsx00\(rsx00 \(rsxc2\(rsx02\(rsx00\(rsx00\(rsx00\(rsx00\(rsx00\(rsx00 .fam .fi .if n .RE .SS "cat" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid cat(string namefmt, [...args])\fP .RE .sp \fBasync\fP .sp Dump the contents of the named file to stdout. \f(CRcat\fP supports the same format string and arguments that \f(CRprintf\fP does. If the file cannot be opened or read an error is printed to stderr. .sp .if n .RS 4 .nf .fam C t:syscalls:sys_enter_execve { cat("/proc/%d/maps", pid); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C 55f683ebd000\-55f683ec1000 r\-\-p 00000000 08:01 1843399 /usr/bin/ls 55f683ec1000\-55f683ed6000 r\-xp 00004000 08:01 1843399 /usr/bin/ls 55f683ed6000\-55f683edf000 r\-\-p 00019000 08:01 1843399 /usr/bin/ls 55f683edf000\-55f683ee2000 rw\-p 00021000 08:01 1843399 /usr/bin/ls 55f683ee2000\-55f683ee3000 rw\-p 00000000 00:00 0 .fam .fi .if n .RE .SS "cgroup_path" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcgroup_path cgroup_path(int cgroupid, string filter)\fP .RE .sp Convert cgroup id to cgroup path. This is done asynchronously in userspace when the cgroup_path value is printed, therefore it can resolve to a different value if the cgroup id gets reassigned. This also means that the returned value can only be used for printing. .sp A string literal may be passed as an optional second argument to filter cgroup hierarchies in which the cgroup id is looked up by a wildcard expression (cgroup2 is always represented by "unified", regardless of where it is mounted). .sp The currently mounted hierarchy at /sys/fs/cgroup is used to do the lookup. If the cgroup with the given id isn\(cqt present here (e.g. when running in a Docker container), the cgroup path won\(cqt be found (unlike when looking up the cgroup path of a process via /proc/.../cgroup). .sp .if n .RS 4 .nf .fam C BEGIN { $cgroup_path = cgroup_path(3436); print($cgroup_path); print($cgroup_path); /* This may print a different path */ printf("%s %s", $cgroup_path, $cgroup_path); /* This may print two different paths */ } .fam .fi .if n .RE .SS "cgroupid" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuint64 cgroupid(const string path)\fP .RE .sp \fBcompile time\fP .sp \f(CRcgroupid\fP retrieves the cgroupv2 ID of the cgroup available at \f(CRpath\fP. .sp .if n .RS 4 .nf .fam C BEGIN { print(cgroupid("/sys/fs/cgroup/system.slice")); } .fam .fi .if n .RE .SS "exit" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid exit()\fP .RE .sp \fBasync\fP .sp Terminate bpftrace, as if a \f(CRSIGTERM\fP was received. The \f(CREND\fP probe will still trigger (if specified) and maps will be printed. .SS "join" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid join(char *arr[], [char * sep = \*(Aq \*(Aq])\fP .RE .sp \fBasync\fP .sp \f(CRjoin\fP joins all the string array \f(CRarr\fP with \f(CRsep\fP as separator into one string. This string will be printed to stdout directly, it cannot be used as string value. .sp The concatenation of the array members is done in BPF and the printing happens in userspace. .sp .if n .RS 4 .nf .fam C tracepoint:syscalls:sys_enter_execve { join(args.argv); } .fam .fi .if n .RE .SS "kaddr" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuint64 kaddr(const string name)\fP .RE .sp \fBcompile time\fP .sp Get the address of the kernel symbol \f(CRname\fP. .sp The following script: .SS "kptr" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRT * kptr(T * ptr)\fP .RE .sp Marks \f(CRptr\fP as a kernel address space pointer. See the address\-spaces section for more information on address\-spaces. The pointer type is left unchanged. .SS "ksym" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRksym_t ksym(uint64 addr)\fP .RE .sp \fBasync\fP .sp Retrieve the name of the function that contains address \f(CRaddr\fP. The address to name mapping happens in user\-space. .sp The \f(CRksym_t\fP type can be printed with the \f(CR%s\fP format specifier. .sp .if n .RS 4 .nf .fam C kprobe:do_nanosleep { printf("%s\(rsn", ksym(reg("ip"))); } .fam .fi .if n .RE .sp Prints: .sp .if n .RS 4 .nf .fam C do_nanosleep .fam .fi .if n .RE .SS "macaddr" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRmacaddr_t macaddr(char [6] mac)\fP .RE .sp Create a buffer that holds a macaddress as read from \f(CRmac\fP This buffer can be printed in the canonical string format using the \f(CR%s\fP format specifier. .sp .if n .RS 4 .nf .fam C kprobe:arp_create { printf("SRC %s, DST %s\(rsn", macaddr(sarg0), macaddr(sarg1)); } .fam .fi .if n .RE .sp Prints: .sp .if n .RS 4 .nf .fam C SRC 18:C0:4D:08:2E:BB, DST 74:83:C2:7F:8C:FF .fam .fi .if n .RE .SS "ntop" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinet_t ntop([int64 af, ] int addr)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinet_t ntop([int64 af, ] char addr[4])\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinet_t ntop([int64 af, ] char addr[16])\fP .RE .sp \f(CRntop\fP returns the string representation of an IPv4 or IPv6 address. \f(CRntop\fP will infer the address type (IPv4 or IPv6) based on the \f(CRaddr\fP type and size. If an integer or \f(CRchar[4]\fP is given, ntop assumes IPv4, if a \f(CRchar[16]\fP is given, ntop assumes IPv6. You can also pass the address type (e.g. AF_INET) explicitly as the first parameter. .SS "pton" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRchar addr[4] pton(const string *addr_v4)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRchar addr[16] pton(const string *addr_v6)\fP .RE .sp \fBcompile time\fP .sp \f(CRpton\fP converts a text representation of an IPv4 or IPv6 address to byte array. \f(CRpton\fP infers the address family based on \f(CR.\fP or \f(CR:\fP in the given argument. \f(CRpton\fP comes in handy when we need to select packets with certain IP addresses. .SS "override" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRoverride(uint64 rc)\fP .RE .sp \fBunsafe\fP .sp \fBKernel\fP 4.16 .sp \fBHelper\fP \f(CRbpf_override\fP .sp .B Supported probes .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} kprobe .RE .sp When using \f(CRoverride\fP the probed function will not be executed and instead \f(CRrc\fP will be returned. .sp .if n .RS 4 .nf .fam C k:__x64_sys_getuid /comm == "id"/ { override(2<<21); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C uid=4194304 gid=0(root) euid=0(root) groups=0(root) .fam .fi .if n .RE .sp This feature only works on kernels compiled with \f(CRCONFIG_BPF_KPROBE_OVERRIDE\fP and only works on functions tagged \f(CRALLOW_ERROR_INJECTION\fP. .sp bpftrace does not test whether error injection is allowed for the probed function, instead if will fail to load the program into the kernel: .sp .if n .RS 4 .nf .fam C ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument Error attaching probe: \*(Aqkprobe:vfs_read\*(Aq .fam .fi .if n .RE .SS "reg" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRreg(const string name)\fP .RE .sp .B Supported probes .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} kprobe .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} uprobe .RE .sp Get the contents of the register identified by \f(CRname\fP. Valid names depend on the CPU architecture. .SS "signal" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRsignal(const string sig)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRsignal(uint32 signum)\fP .RE .sp \fBunsafe\fP .sp \fBKernel\fP 5.3 .sp \fBHelper\fP \f(CRbpf_send_signal\fP .sp Probe types: k(ret)probe, u(ret)probe, USDT, profile .sp Send a signal to the process being traced. The signal can either be identified by name, e.g. \f(CRSIGSTOP\fP or by ID, e.g. \f(CR19\fP as found in \f(CRkill \-l\fP. .sp .if n .RS 4 .nf .fam C kprobe:__x64_sys_execve /comm == "bash"/ { signal(5); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C $ ls Trace/breakpoint trap (core dumped) .fam .fi .if n .RE .SS "sizeof" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRsizeof(TYPE)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRsizeof(EXPRESSION)\fP .RE .sp \fBcompile time\fP .sp Returns size of the argument in bytes. Similar to C/C++ \f(CRsizeof\fP operator. Note that the expression does not get evaluated. .SS "offsetof" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRoffsetof(STRUCT, FIELD)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRoffsetof(EXPRESSION, FIELD)\fP .RE .sp \fBcompile time\fP .sp Returns offset of the field offset bytes in struct. Similar to kernel \f(CRoffsetof\fP operator. Note that subfields are not yet supported. .SS "str" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRstr(char * data [, uint32 length)\fP .RE .sp \fBHelper\fP \f(CRprobe_read_str, probe_read_{kernel,user}_str\fP .sp \f(CRstr\fP reads a NULL terminated (\f(CR\(rs0\fP) string from \f(CRdata\fP. The maximum string length is limited by the \f(CRBPFTRACE_STR_LEN\fP env variable, unless \f(CRlength\fP is specified and shorter than the maximum. In case the string is longer than the specified length only \f(CRlength \- 1\fP bytes are copied and a NULL byte is appended at the end. .sp When available (starting from kernel 5.5, see the \f(CR\-\-info\fP flag) bpftrace will automatically use the \f(CRkernel\fP or \f(CRuser\fP variant of \f(CRprobe_read_{kernel,user}_str\fP based on the address space of \f(CRdata\fP, see ADDRESS\-SPACES for more information. .SS "strerror" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRstrerror strerror(int error)\fP .RE .sp Convert errno code to string. This is done asynchronously in userspace when the strerror value is printed, hence the returned value can only be used for printing. .sp .if n .RS 4 .nf .fam C #include BEGIN { print(strerror(EPERM)); } .fam .fi .if n .RE .SS "strftime" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRstrtime_t strftime(const string fmt, int64 timestamp_ns)\fP .RE .sp \fBasync\fP .sp Format the nanoseconds since boot timestamp \f(CRtimestamp_ns\fP according to the format specified by \f(CRfmt\fP. The time conversion and formatting happens in user space, therefore the \f(CRtimestr_t\fP value returned can only be used for printing using the \f(CR%s\fP format specifier. .sp bpftrace uses the \f(CRstrftime(3)\fP function for formatting time and supports the same format specifiers. .sp .if n .RS 4 .nf .fam C i:s:1 { printf("%s\(rsn", strftime("%H:%M:%S", nsecs)); } .fam .fi .if n .RE .sp bpftrace also supports the following format string extensions: .TS allbox tab(:); lt lt. T{ .sp Specifier T}:T{ .sp Description T} T{ .sp \f(CR%f\fP T}:T{ .sp Microsecond as a decimal number, zero\-padded on the left T} .TE .sp .SS "strncmp" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRint64 strncmp(char * s1, char * s2, int64 n)\fP .RE .sp \f(CRstrncmp\fP compares up to \f(CRn\fP characters string \f(CRs1\fP and string \f(CRs2\fP. If they\(cqre equal \f(CR0\fP is returned, else a non\-zero value is returned. .sp bpftrace doesn\(cqt read past the length of the shortest string. .sp The use of the \f(CR==\fP and \f(CR!=\fP operators is recommended over calling \f(CRstrncmp\fP directly. .SS "strcontains" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRint64 strcontains(const char *haystack, const char *needle)\fP .RE .sp \f(CRstrcontains\fP compares whether the string haystack contains the string needle. If needle is contained \f(CR1\fP is returned, else zero is returned. .sp bpftrace doesn\(cqt read past the length of the shortest string. .SS "system" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid system(string namefmt [, ...args])\fP .RE .sp \fBunsafe\fP \fBasync\fP .sp \f(CRsystem\fP lets bpftrace run the specified command (\f(CRfork\fP and \f(CRexec\fP) until it completes and print its stdout. The \f(CRcommand\fP is run with the same privileges as bpftrace and it blocks execution of the processing threads which can lead to missed events and delays processing of async events. .sp .if n .RS 4 .nf .fam C i:s:1 { time("%H:%M:%S: "); printf("%d\(rsn", @++); } i:s:10 { system("/bin/sleep 10"); } i:s:30 { exit(); } .fam .fi .if n .RE .sp Note how the async \f(CRtime\fP and \f(CRprintf\fP first print every second until the \f(CRi:s:10\fP probe hits, then they print every 10 seconds due to bpftrace blocking on \f(CRsleep\fP. .sp .if n .RS 4 .nf .fam C Attaching 3 probes... 08:50:37: 0 08:50:38: 1 08:50:39: 2 08:50:40: 3 08:50:41: 4 08:50:42: 5 08:50:43: 6 08:50:44: 7 08:50:45: 8 08:50:46: 9 08:50:56: 10 08:50:56: 11 08:50:56: 12 08:50:56: 13 08:50:56: 14 08:50:56: 15 08:50:56: 16 08:50:56: 17 08:50:56: 18 08:50:56: 19 .fam .fi .if n .RE .sp \f(CRsystem\fP supports the same format string and arguments that \f(CRprintf\fP does. .sp .if n .RS 4 .nf .fam C t:syscalls:sys_enter_execve { system("/bin/grep %s /proc/%d/status", "vmswap", pid); } .fam .fi .if n .RE .SS "time" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid time(const string fmt)\fP .RE .sp \fBasync\fP .sp Format the current wall time according to the format specifier \f(CRfmt\fP and print it to stdout. Unlike \f(CRstrftime()\fP \f(CRtime()\fP doesn\(cqt send a timestamp from the probe, instead it is the time at which user\-space processes the event. .sp bpftrace uses the \f(CRstrftime(3)\fP function for formatting time and supports the same format specifiers. .SS "uaddr" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRT * uaddr(const string sym)\fP .RE .sp .B Supported probes .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} uprobes .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} uretprobes .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} USDT .RE .sp \fBDoes not work with ASLR, see issue \c .URL "https://github.com/iovisor/bpftrace/issues/75" "#75" "\fP" .sp The \f(CRuaddr\fP function returns the address of the specified symbol. This lookup happens during program compilation and cannot be used dynamically. .sp The default return type is \f(CRuint64*\fP. If the ELF object size matches a known integer size (1, 2, 4 or 8 bytes) the return type is modified to match the width (\f(CRuint8*\fP, \f(CRuint16*\fP, \f(CRuint32*\fP or \f(CRuint64*\fP resp.). As ELF does not contain type info the type is always assumed to be unsigned. .sp .if n .RS 4 .nf .fam C uprobe:/bin/bash:readline { printf("PS1: %s\(rsn", str(*uaddr("ps1_prompt"))); } .fam .fi .if n .RE .SS "uptr" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRT * uptr(T * ptr)\fP .RE .sp Marks \f(CRptr\fP as a user address space pointer. See the address\-spaces section for more information on address\-spaces. The pointer type is left unchanged. .SS "usym" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRusym_t usym(uint64 * addr)\fP .RE .sp \fBasync\fP .sp .B Supported probes .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} uprobes .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} uretprobes .RE .sp Equal to ksym but resolves user space symbols. .sp If ASLR is enabled, user space symbolication only works when the process is running at either the time of the symbol resolution or the time of the probe attachment. The latter requires \f(CRBPFTRACE_CACHE_USER_SYMBOLS\fP to be set to \f(CRPER_PID\fP, and might not work with older versions of BCC. A similar limitation also applies to dynamically loaded symbols. .sp .if n .RS 4 .nf .fam C uprobe:/bin/bash:readline { printf("%s\(rsn", usym(reg("ip"))); } .fam .fi .if n .RE .sp Prints: .sp .if n .RS 4 .nf .fam C readline .fam .fi .if n .RE .SS "path" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRchar * path(struct path * path)\fP .RE .sp \fBKernel\fP 5.10 .sp \fBHelper\fP bpf_d_path .sp Return full path referenced by struct path pointer in argument. .sp This function can only be used by functions that are allowed to, these functions are contained in the \f(CRbtf_allowlist_d_path\fP set in the kernel. .SS "unwatch" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid unwatch(void * addr)\fP .RE .sp \fBasync\fP .sp Removes a watchpoint .SS "skboutput" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuint32 skboutput(const string path, struct sk_buff *skb, uint64 length, const uint64 offset)\fP .RE .sp \fBKernel\fP 5.5 .sp \fBHelper\fP bpf_skb_output .sp Write sk_buff \f(CRskb\fP \*(Aqs data section to a PCAP file in the \f(CRpath\fP, starting from \f(CRoffset\fP to \f(CRoffset\fP + \f(CRlength\fP. .sp The PCAP file is encapsulated in RAW IP, so no ethernet header is included. The \f(CRdata\fP section in the struct \f(CRskb\fP may contain ethernet header in some kernel contexts, you may set \f(CRoffset\fP to 14 bytes to exclude ethernet header. .sp Each packet\(cqs timestamp is determined by adding \f(CRnsecs\fP and boot time, the accuracy varies on different kernels, see \f(CRnsecs\fP. .sp This function returns 0 on success, or a negative error in case of failure. .sp Environment variable \f(CRBPFTRACE_PERF_RB_PAGES\fP should be increased in order to capture large packets, or else these packets will be dropped. .sp Usage .sp .if n .RS 4 .nf .fam C # cat dump.bt kfunc:napi_gro_receive { $ret = skboutput("receive.pcap", args.skb, args.skb\->len, 0); } kfunc:dev_queue_xmit { // setting offset to 14, to exclude ethernet header $ret = skboutput("output.pcap", args.skb, args.skb\->len, 14); printf("skboutput returns %d\(rsn", $ret); } # export BPFTRACE_PERF_RB_PAGES=1024 # bpftrace dump.bt \&... # tcpdump \-n \-r ./receive.pcap | head \-3 reading from file ./receive.pcap, link\-type RAW (Raw IP) dropped privs to tcpdump 10:23:44.674087 IP 22.128.74.231.63175 > 192.168.0.23.22: Flags [.], ack 3513221061, win 14009, options [nop,nop,TS val 721277750 ecr 3115333619], length 0 10:23:45.823194 IP 100.101.2.146.53 > 192.168.0.23.46619: 17273 0/1/0 (130) 10:23:45.823229 IP 100.101.2.146.53 > 192.168.0.23.46158: 45799 1/0/0 A 100.100.45.106 (60) .fam .fi .if n .RE .SH "OUTPUT FORMATTING" .SS "print" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid print(T val)\fP .RE .sp \fBasync\fP .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid print(T val)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid print(@map)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid print(@map, uint64 top)\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid print(@map, uint64 top, uint64 div)\fP .RE .sp \f(CRprint\fP prints a the value, which can be a map or a scalar value, with the default formatting for the type. .sp .if n .RS 4 .nf .fam C i:ms:10 { @=hist(rand); } i:s:1 { print(@); print(123); print("abc"); exit(); } .fam .fi .if n .RE .sp Prints: .sp .if n .RS 4 .nf .fam C @: [16M, 32M) 3 |@@@ | [32M, 64M) 2 |@@ | [64M, 128M) 1 |@ | [128M, 256M) 4 |@@@@ | [256M, 512M) 3 |@@@ | [512M, 1G) 14 |@@@@@@@@@@@@@@ | [1G, 2G) 22 |@@@@@@@@@@@@@@@@@@@@@@ | [2G, 4G) 51 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| 123 abc .fam .fi .if n .RE .sp Note that maps are printed by reference while scalar values are copied. This means that updating and printing maps in a fast loop will likely result in bogus map values as the map will be updated before userspace gets the time to dump and print it. .sp The printing of maps supports the optional \f(CRtop\fP and \f(CRdiv\fP arguments. \f(CRtop\fP limits the printing to the top N entries with the highest integer values .sp .if n .RS 4 .nf .fam C BEGIN { $i = 11; while($i) { @[$i] = \-\-$i; } print(@, 2); clear(@); exit() } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C @[9]: 9 @[10]: 10 .fam .fi .if n .RE .sp The \f(CRdiv\fP argument scales the values prior to printing them. Scaling values before storing them can result in rounding errors. Consider the following program: .sp .if n .RS 4 .nf .fam C k:f { @[func] += arg0/10; } .fam .fi .if n .RE .sp With the following sequence as numbers for arg0: \f(CR134, 377, 111, 99\fP. The total is \f(CR721\fP which rounds to \f(CR72\fP when scaled by 10 but the program would print \f(CR70\fP due to the rounding of individual values. .sp Changing the print call to \f(CRprint(@, 5, 2)\fP will take the top 5 values and scale them by 2: .sp .if n .RS 4 .nf .fam C @[6]: 3 @[7]: 3 @[8]: 4 @[9]: 4 @[10]: 5 .fam .fi .if n .RE .SS "printf" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRvoid printf(const string fmt, args...)\fP .RE .sp \fBasync\fP .sp \f(CRprintf()\fP formats and prints data. It behaves similar to \f(CRprintf()\fP found in \f(CRC\fP and many other languages. .sp The format string has to be a constant, it cannot be modified at runtime. The formatting of the string happens in user space. Values are copied and passed by value. .sp bpftrace supports all the typical format specifiers like \f(CR%llx\fP and \f(CR%hhu\fP. The non\-standard ones can be found in the table below: .TS allbox tab(:); lt lt lt. T{ .sp Specifier T}:T{ .sp Type T}:T{ .sp Description T} T{ .sp r T}:T{ .sp buffer T}:T{ .sp Hex\-formatted string to print arbitrary binary content returned by the buf (buf) function. T} .TE .sp .sp Supported escape sequences .sp Colors are supported too, using standard terminal escape sequences: .sp .if n .RS 4 .nf .fam C print("\(rs033[31mRed\(rst\(rs033[33mYellow\(rs033[0m\(rsn") .fam .fi .if n .RE .SH "PROBES" .sp bpftrace supports various probe types which allow the user to attach BPF programs to different types of events. Each probe starts with a provider (e.g. \f(CRkprobe\fP) followed by a colon (\f(CR:\fP) separated list of options. The amount of options and their meaning depend on the provider and are detailed below. The valid values for options can depend on the system or binary being traced, e.g. for uprobes it depends on the binary. Also see LISTING PROBES .sp It is possible to associate multiple probes with a single action as long as the action is valid for all specified probes. Multiple probes can be specified as a comma (\f(CR,\fP) separated list: .sp .if n .RS 4 .nf .fam C kprobe:tcp_reset,kprobe:tcp_v4_rcv { printf("Entered: %s\(rsn", probe); } .fam .fi .if n .RE .sp Wildcards are supported too: .sp .if n .RS 4 .nf .fam C kprobe:tcp_* { printf("Entered: %s\(rsn", probe); } .fam .fi .if n .RE .sp Both can be combined: .sp .if n .RS 4 .nf .fam C kprobe:tcp_reset,kprobe:*socket* { printf("Entered: %s\(rsn", probe); } .fam .fi .if n .RE .sp Most providers also support a short name which can be used instead of the full name, e.g. \f(CRkprobe:f\fP and \f(CRk:f\fP are identical. .SS "BEGIN and END" .sp These are special built\-in events provided by the bpftrace runtime. \f(CRBEGIN\fP is triggered before all other probes are attached. \f(CREND\fP is triggered after all other probes are detached. .sp Note that specifying an \f(CREND\fP probe doesn\(cqt override the printing of \*(Aqnon\-empty\*(Aq maps at exit. To prevent the printing all used maps need be cleared, which can be done in the \f(CREND\fP probe: .sp .if n .RS 4 .nf .fam C END { clear(@map1); clear(@map2); } .fam .fi .if n .RE .SS "hardware" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRhardware:event_name:\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRhardware:event_name:count\fP .RE .sp .B shortname .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRh\fP .RE .sp The \f(CRhardware\fP probe attaches to pre\-defined hardware events provided by the kernel. .sp They are implemented using performance monitoring counters (PMCs): hardware resources on the processor. There are about ten of these, and they are documented in the \f(CRperf_event_open(2)\fP man page. The event names are: .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcpu\-cycles\fP or \f(CRcycles\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinstructions\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcache\-references\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcache\-misses\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRbranch\-instructions\fP or \f(CRbranches\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRbranch\-misses\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRbus\-cycles\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRfrontend\-stalls\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRbackend\-stalls\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRref\-cycles\fP .RE .sp The \f(CRcount\fP option specifies how many events must happen before the probe fires. If \f(CRcount\fP is left unspecified a default value is used. .sp .if n .RS 4 .nf .fam C hardware:cache\-misses:1e6 { @[pid] = count(); } .fam .fi .if n .RE .SS "interval" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinterval:us:count\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinterval:ms:count\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinterval:s:count\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRinterval:hz:rate\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRi\fP .RE .sp The interval probe fires at a fixed interval as specified by its time spec. Interval fire on one CPU at the time, unlike [profile] probes. .SS "iterator" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRiter:task\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRiter:task:pin\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRiter:task_file\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRiter:task_file:pin\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRiter:task_vma\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRiter:task_vma:pin\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRit\fP .RE .sp These are eBPF iterator probes, that allow iteration over kernel objects. .sp Iterator probe can\(cqt be mixed with any other probe, not even other iterator. .sp Each iterator probe provides set of fields that could be accessed with ctx pointer. User can display set of available fields for iterator via \-lv options as described below. .sp Examples: .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aqiter:task { printf("%s:%d\(rsn", ctx\->task\->comm, ctx\->task\->pid); }\*(Aq Attaching 1 probe... systemd:1 kthreadd:2 rcu_gp:3 rcu_par_gp:4 kworker/0:0H:6 mm_percpu_wq:8 \&... # bpftrace \-e \*(Aqiter:task_file { printf("%s:%d %d:%s\(rsn", ctx\->task\->comm, ctx\->task\->pid, ctx\->fd, path(ctx\->file\->f_path)); }\*(Aq Attaching 1 probe... systemd:1 1:/dev/null systemd:1 2:/dev/null systemd:1 3:/dev/kmsg \&... su:1622 1:/dev/pts/1 su:1622 2:/dev/pts/1 su:1622 3:/var/lib/sss/mc/passwd \&... bpftrace:1892 1:pipe:[35124] bpftrace:1892 2:/dev/pts/1 bpftrace:1892 3:anon_inode:bpf\-map bpftrace:1892 4:anon_inode:bpf\-map bpftrace:1892 5:anon_inode:bpf_link bpftrace:1892 6:anon_inode:bpf\-prog bpftrace:1892 7:anon_inode:bpf_iter # bpftrace \-e \*(Aqiter:task_vma {printf("%s %d %lx\-%lx\(rsn", comm, pid, ctx\->vma\->vm_start, ctx\->vma\->vm_end);}\*(Aq Attaching 1 probe... bpftrace 119480 55b92c380000\-55b92c386000 bpftrace 119480 55b92c386000\-55b92c391000 bpftrace 119480 55b92c391000\-55b92c397000 bpftrace 119480 55b92c398000\-55b92c399000 bpftrace 119480 55b92c399000\-55b92c39a000 bpftrace 119480 55b92cce3000\-55b92d010000 \&... bpftrace 119480 7ffd55dde000\-7ffd55de2000 bpftrace 119480 7ffd55de2000\-7ffd55de4000 .fam .fi .if n .RE .sp It\(cqs possible to pin iterator with specifying optional probe \*(Aq:pin\*(Aq part, that defines the pin file. It can be specified as absolute path or relative to /sys/fs/bpf. .sp .B relative pin .br .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aqiter:task:list { printf("%s:%d\(rsn", ctx\->task\->comm, ctx\->task\->pid); }\*(Aq Program pinned to /sys/fs/bpf/list .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C # cat /sys/fs/bpf/list systemd:1 kthreadd:2 rcu_gp:3 rcu_par_gp:4 kworker/0:0H:6 mm_percpu_wq:8 rcu_tasks_kthre:9 \&... .fam .fi .if n .RE .sp Examples with absolute pin file: .sp .B absolute pin .br .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aq iter:task_file:/sys/fs/bpf/files { printf("%s:%d %s\(rsn", ctx\->task\->comm, ctx\->task\->pid, path(ctx\->file\->f_path)); }\*(Aq Program pinned to /sys/fs/bpf/files .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C # cat /sys/fs/bpf/files systemd:1 anon_inode:inotify systemd:1 anon_inode:[timerfd] \&... systemd\-journal:849 /dev/kmsg systemd\-journal:849 anon_inode:[eventpoll] \&... sssd:1146 /var/log/sssd/sssd.log sssd:1146 anon_inode:[eventpoll] \&... NetworkManager:1155 anon_inode:[eventfd] NetworkManager:1155 /var/lib/sss/mc/passwd (deleted) .fam .fi .if n .RE .SS "kfunc and kretfunc" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRkfunc[:mod]:fn\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRkretfunc[:mod]:fn\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRf\fP (\f(CRkfunc\fP) .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRfr\fP (\f(CRkretfunc\fP) .RE .sp .B requires (\f(CR\-\-info\fP) .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} Kernel features:BTF .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} Probe types:kfunc .RE .sp \f(CRkfunc\fPs attach to kernel function similar to kprobe and kretprobe. They make use of eBPF trampolines which allows kernel code to call into BPF programs with near zero overhead. .sp \f(CRkfunc\fP s make use of BTF type information to derive the type of function arguments at compile time. This removes the need for manual type casting and makes the code more resilient against small signature changes in the kernel. The function arguments are available in the \f(CRargs\fP struct which can be inspected by doing verbose listing (see LISTING PROBES). These arguments are also available in the return probe (\f(CRkretfunc\fP). .sp .if n .RS 4 .nf .fam C # bpftrace \-lv \*(Aqkfunc:tcp_reset\*(Aq kfunc:tcp_reset struct sock * sk struct sk_buff * skb .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C kfunc:x86_pmu_stop { printf("pmu %s stop\(rsn", str(args.event\->pmu\->name)); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C kretfunc:fget { printf("fd %d name %s\(rsn", args.fd, str(retval\->f_path.dentry\->d_name.name)); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C fd 3 name ld.so.cache fd 3 name libselinux.so.1 fd 3 name libselinux.so.1 \&... .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C kfunc:kvm:x86_emulate_insn { @ = count(); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C @ = 347603 .fam .fi .if n .RE .SS "kprobe and kretprobe" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRkprobe:fn\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRkprobe:fn+offset\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRkretprobe:fn\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRk\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRkr\fP .RE .sp \f(CRkprobe\fP s allow for dynamic instrumentation of kernel functions. Each time the specified kernel function is executed the attached BPF programs are ran. .sp .if n .RS 4 .nf .fam C kprobe:tcp_reset { @tcp_resets = count() } .fam .fi .if n .RE .sp Function arguments are available through the \f(CRargX\fP and \f(CRsargX\fP builtins, for register args and stack args respectively. Whether arguments passed on stack or in a register depends on the architecture and the number or arguments in used, e.g. on x86_64 the first non\-floating point 6 arguments are passed in registers, all following arguments are passed on the stack. Note that floating point arguments are typically passed in special registers which don\(cqt count as \f(CRargX\fP arguments which can cause confusion. Consider a function with the following signature: .sp .if n .RS 4 .nf .fam C void func(int a, double d, int x) .fam .fi .if n .RE .sp Due to \f(CRd\fP being a floating point \f(CRx\fP is accessed through \f(CRarg1\fP where one might expect \f(CRarg2\fP. .sp bpftrace does not detect the function signature so it is not aware of the argument count or their type. It is up to the user to perform Type conversion when needed, e.g. .sp .if n .RS 4 .nf .fam C kprobe:tcp_connect { $sk = ((struct sock *) arg0); ... } .fam .fi .if n .RE .sp \f(CRkprobe\fP s are not limited to function entry, they can be attached to any instruction in a function by specifying an offset from the start of the function. .sp \f(CRkretprobe\fP s trigger on the return from a kernel function. Return probes do not have access to the function (input) arguments, only to the return value (through \f(CRretval\fP). A common pattern to work around this is by storing the arguments in a map on function entry and retrieving in the return probe: .sp .if n .RS 4 .nf .fam C kprobe:d_lookup { $name = (struct qstr *)arg1; @fname[tid] = $name\->name; } kretprobe:d_lookup /@fname[tid]/ { printf("%\-8d %\-6d %\-16s M %s\(rsn", elapsed / 1e6, pid, comm, str(@fname[tid])); } .fam .fi .if n .RE .SS "profile" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRprofile:us:count\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRprofile:ms:count\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRprofile:s:count\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRprofile:hz:rate\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRp\fP .RE .sp Profile probes fire on each CPU on the specified interval. .SS "software" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRsoftware:event:\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRsoftware:event:count\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRs\fP .RE .sp The \f(CRsoftware\fP probe attaches to pre\-defined software events provided by the kernel. Event details can be found in the \f(CRperf_event_open(2)\fP man page. .sp The event names are: .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcpu\-clock\fP or \f(CRcpu\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRtask\-clock\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRpage\-faults\fP or \f(CRfaults\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcontext\-switches\fP or \f(CRcs\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRcpu\-migrations\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRminor\-faults\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRmajor\-faults\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRalignment\-faults\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRemulation\-faults\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRdummy\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRbpf\-output\fP .RE .SS "tracepoint" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRtracepoint:subsys:event\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRt\fP .RE .sp Tracepoints are hooks into events in the kernel. Tracepoints are defined in the kernel source and compiled into the kernel binary which makes them a form of static tracing. Which means that unlike \f(CRkprobe\fP s new tracepoints cannot be added without modifying the kernel. .sp The advantage of tracepoints is that they generally provide a more stable interface than \f(CRkprobe\fP s do, they do not depend on the existence of a kernel function. .sp Tracepoint arguments are available in the \f(CRargs\fP struct which can be inspected with verbose listing, see the LISTING PROBES section for more details. .sp .if n .RS 4 .nf .fam C tracepoint:syscalls:sys_enter_openat { printf("%s %s\(rsn", comm, str(args.filename)); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C irqbalance /proc/interrupts irqbalance /proc/stat snmpd /proc/diskstats snmpd /proc/stat snmpd /proc/vmstat snmpd /proc/net/dev [...] .fam .fi .if n .RE .sp .B Additional information .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} .URL "https://www.kernel.org/doc/html/latest/trace/tracepoints.html" "" "" .RE .SS "rawtracepoint" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRrawtracepoint:event\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRrt\fP .RE .sp The hook point triggered by \f(CRtracepoint\fP and \f(CRrawtracepoint\fP is the same. \f(CRtracepoint\fP and \f(CRrawtracepoint\fP are nearly identical in terms of functionality. The only difference is in the program context. \f(CRrawtracepoint\fP offers raw arguments to the tracepoint while \f(CRtracepoint\fP applies further processing to the raw arguments. The additional processing is defined inside the kernel. .sp Tracepoint arguments are available via the \f(CRargN\fP builtins. The available arguments can be found in the relative path of the kernel source code \f(CRinclude/trace/events/\fP. Each arg is a 64\-bit integer. .sp .if n .RS 4 .nf .fam C rawtracepoint:block_rq_insert { printf("%llx %llx\(rsn", arg0, arg1); } .fam .fi .if n .RE .sp .if n .RS 4 .nf .fam C ffff88810977d6f8 ffff8881097e8e80 [...] .fam .fi .if n .RE .SS "uprobe, uretprobe" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuprobe:binary:func\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuprobe:binary:func+offset\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuprobe:binary:offset\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRuretprobe:binary:func\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRu\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRur\fP .RE .sp \f(CRuprobe\fP s or user\-space probes are the user\-space equivalent of \f(CRkprobe\fP s. The same limitations that apply kprobe and kretprobe also apply to \f(CRuprobe\fP s and \f(CRuretprobe\fP s. .sp When tracing libraries, it is sufficient to specify the library name instead of a full path. The path will be then automatically resolved using \f(CR/etc/ld.so.cache\fP: .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aquprobe:libc:malloc { printf("Allocated %d bytes\(rsn", arg0); }\*(Aq Allocated 4 bytes \&... .fam .fi .if n .RE .sp If the traced binary has DWARF included, function arguments are available in the \f(CRargs\fP struct which can be inspected with verbose listing, see the LISTING PROBES section for more details. .sp When tracing C++ programs, it is possible to turn on automatic symbol demangling by using the \f(CR:cpp\fP prefix: .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aqu:src/bpftrace:cpp:"bpftrace::BPFtrace::add_probe" { print("adding probe\(rsn"); }\*(Aq Attaching 1 probe... adding probe .fam .fi .if n .RE .sp It is important to note that for \f(CRuretprobe\fP s to work the kernel runs a special helper on user\-space function entry which overrides the return address on the stack. This can cause issues with languages that have their own runtime like Golang: .sp .B example.go .br .sp .if n .RS 4 .nf .fam C func myprint(s string) { fmt.Printf("Input: %s\(rsn", s) } func main() { ss := []string{"a", "b", "c"} for _, s := range ss { go myprint(s) } time.Sleep(1*time.Second) } .fam .fi .if n .RE .sp .B bpftrace .br .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aquretprobe:./test:main.myprint { @=count(); }\*(Aq \-c ./test runtime: unexpected return pc for main.myprint called from 0x7fffffffe000 stack: frame={sp:0xc00008cf60, fp:0xc00008cfd0} stack=[0xc00008c000,0xc00008d000) fatal error: unknown caller pc .fam .fi .if n .RE .SS "usdt" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRusdt:binary_path:probe_name\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRusdt:binary_path:[probe_namespace]:probe_name\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRusdt:library_path:probe_name\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRusdt:library_path:[probe_namespace]:probe_name\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRU\fP .RE .sp You can target the entire host (or an entire process\(cqs address space by using the \f(CR\-p\fP arg) by using a single wildcard in place of the binary_path/library_path e.g. bpftrace \-e \*(Aqusdt:*:loop { printf("hi\(rsn"); }. Please note that if you use wildcards for the probe_name or probe_namespace and end up targeting multiple USDTs for the same probe you might get errors if you also utilize the USDT argument builtins (e.g. arg0) as they could be of different types. .SS "watchpoint and asyncwatchpoint" .sp .B variants .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRwatchpoint:absolute_address:length:mode\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRwatchpoint:function+argN:length:mode\fP .RE .sp .B shortnames .br .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRw\fP .RE .sp .RS 4 .ie n \{\ \h'-04'\(bu\h'+03'\c .\} .el \{\ . sp -1 . IP \(bu 2.3 .\} \f(CRaw\fP .RE .sp These are memory watchpoints provided by the kernel. Whenever a memory address is written to (\f(CRw\fP), read from (\f(CRr\fP), or executed (\f(CRx\fP), the kernel can generate an event. .sp In the first form, an absolute address is monitored. If a pid (\f(CR\-p\fP) or a command (\f(CR\-c\fP) is provided, bpftrace takes the address as a userspace address and monitors the appropriate process. If not, bpftrace takes the address as a kernel space address. .sp In the second form, the address present in \f(CRargN\fP when \f(CRfunction\fP is entered is monitored. A pid or command must be provided for this form. If synchronous (\f(CRwatchpoint\fP), a \f(CRSIGSTOP\fP is sent to the tracee upon function entry. The tracee will be \f(CRSIGCONT\fPed after the watchpoint is attached. This is to ensure events are not missed. If you want to avoid the \f(CRSIGSTOP\fP + \f(CRSIGCONT\fP use \f(CRasyncwatchpoint\fP. .sp Note that on most architectures you may not monitor for execution while monitoring read or write. .sp Examples .sp Print \f(CRhit\fP when a read from or write to \f(CR0x10000000\fP happens: .sp .if n .RS 4 .nf .fam C # bpftrace \-e \*(Aqwatchpoint:0x10000000:8:rw { printf("hit!\(rsn"); exit(); }\*(Aq \-c ./testprogs/watchpoint .fam .fi .if n .RE .sp Print the call stack every time the \f(CRjiffies\fP variable is updated: .sp .if n .RS 4 .nf .fam C # bpftrace \-e "watchpoint:0x$(awk \*(Aq$3 == "jiffies" {print $1}\*(Aq /proc/kallsyms):8:w { @[kstack] = count(); } i:s:1 { exit(); }" \&...... @[ do_timer+12 tick_do_update_jiffies64.part.22+89 tick_sched_do_timer+103 tick_sched_timer+39 __hrtimer_run_queues+256 hrtimer_interrupt+256 smp_apic_timer_interrupt+106 apic_timer_interrupt+15 cpuidle_enter_state+188 cpuidle_enter+41 do_idle+536 cpu_startup_entry+25 start_secondary+355 secondary_startup_64+164 ]: 319 .fam .fi .if n .RE .sp "hit" and exit when the memory pointed to by \f(CRarg1\fP of \f(CRincrement\fP is written to. .sp .if n .RS 4 .nf .fam C # cat wpfunc.c #include #include #include __attribute__((noinline)) void increment(__attribute__((unused)) int _, int *i) { (*i)++; } int main() { int *i = malloc(sizeof(int)); while (1) { increment(0, i); (*i)++; usleep(1000); } } # bpftrace \-e \*(Aqwatchpoint:increment+arg1:4:w { printf("hit!\(rsn"); exit() }\*(Aq \-c ./wpfunc .fam .fi .if n .RE .SH "LISTING PROBES" .sp Probe listing is the method to discover which probes are supported by the current system. Listing supports the same syntax as normal attachment does: .sp .if n .RS 4 .nf .fam C # bpftrace \-l \*(Aqkprobe:*\*(Aq # bpftrace \-l \*(Aqt:syscalls:*openat* # bpftrace \-l \*(Aqkprobe:tcp*,trace # bpftrace \-l \*(Aqk:*socket*,tracepoint:syscalls:*tcp*\*(Aq .fam .fi .if n .RE .sp The verbose flag (\f(CR\-v\fP) can be specified to inspect arguments (\f(CRargs\fP) for providers that support it: .sp .if n .RS 4 .nf .fam C # bpftrace \-l \*(Aqfr:tcp_reset,t:syscalls:sys_enter_openat\*(Aq \-v kretfunc:tcp_reset struct sock * sk struct sk_buff * skb tracepoint:syscalls:sys_enter_openat int __syscall_nr int dfd const char * filename int flags umode_t mode # bpftrace \-l \*(Aquprobe:/bin/bash:rl_set_prompt\*(Aq \-v # works only if /bin/bash has DWARF uprobe:/bin/bash:rl_set_prompt const char *prompt .fam .fi .if n .RE