NAME¶

SYNOPSIS¶

DESCRIPTION¶

WARNING: This uses dynamic tracing of user-level functions, using some relatively new kernel code. I have seen this cause target processes to fail, either entering endless spin loops or crashing on illegal instructions. I believe newer kernels (post 4.0) are relatively safer, but use caution. Test in a lab environment, and know what you are doing, before use. Also consider other (more developed) user-level tracers (perf_events, LTTng, etc.).

Use extreme caution with the raw address mode: eg, "p:libc:0xbf130". uprobe does not check for instruction alignment, so tracing the wrong address (eg, mid-way through a multi-byte instruction) will corrupt the target's memory. Other tracers (eg, perf_events with debuginfo) check alignment.

Also beware of widespread tracing that interferes with the operation of the system, eg, tracing libc:malloc, which by-default will trace _all_ processes.

I wrote this because I kept testing different custom uprobes at the command line, and wanted a way to automate the steps. For generic user-level tracing, use perf_events directly.

Since this uses ftrace, only the root user can use this tool.

REQUIREMENTS¶

OPTIONS¶

-F

Force. Trace despite kernel version warnings. Use on older kernels may expose you to (since fixed) bugs, which can lock up or crash target processes, which could also lock up the entire system. Test in a lab environment before use, and consider other more developed user-level tracers (perf_events, LTTng, etc.).

-d seconds

Set the duration of tracing, in seconds. Trace output will be buffered and printed at the end. This also reduces overheads by buffering in-kernel, instead of printing events as they occur.

The ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing, try increasing that size.

-h

Print usage message.

-H

Print column headers.

-s

Print user-level stack traces after each event. These are currently printed in hex, and need post-processing to see user-level symbols (eg, addr2line; I should automate that).

-v

Show the uprobe format file only (do not trace), identifying possible variables for use in a custom filter.

-p PID

Only trace user-level functions when this process ID is on-CPU.

uprobe_definition

A full uprobe definition, as documented by Documentation/trace/uprobetracer.txt in the Linux kernel source. Note that the probe alias name is optional with uprobe(8), and if not specified, it will default to something meaningful. See the EXAMPLES section.

filter

An ftrace filter definition.

EXAMPLES¶

trace readline() calls in all running "bash" executables:

# uprobe p:bash:readline

trace readline() with explicit executable path:

# uprobe p:/bin/bash:readline

trace the return of readline() with return value as a string:

# uprobe 'r:bash:readline +0($retval):string'

trace sleep() calls in all running libc shared libraries:

# uprobe p:libc:sleep

trace sleep() with register %di (x86):

# uprobe 'p:libc:sleep %di'

trace this address (use caution: must be instruction aligned):

# uprobe p:libc:0xbf130

trace gettimeofday() for PID 1182 only:

# uprobe -p 1182 p:libc:gettimeofday

trace the return of fopen() only when it returns NULL:

# uprobe 'r:libc:fopen file=$retval' 'file == 0'

FIELDS¶

Typical fields are:

TASK-PID

The process name (which could include dashes), a dash, and the process ID.

CPU#

The CPU ID, in brackets.

||||

Kernel state flags. For example, on Linux 3.16 these are for irqs-off, need-resched, hardirq/softirq, and preempt-depth.

TIMESTAMP

Time of event, in seconds.

FUNCTION

User-level function name.

OVERHEAD¶

If you find a use for uprobe(8) where the overhead is prohibitive, consider the same enabling using perf_events where overhead should be reduced.

SOURCE¶

https://github.com/brendangregg/perf-tools

Also look under the examples directory for a text file containing example usage, output, and commentary for this tool.

OS¶

STABILITY¶

AUTHOR¶

SEE ALSO¶