NAME¶
iosnoop - trace block I/O events as they occur. Uses Linux ftrace.
SYNOPSIS¶
iosnoop [-hQst] [-d device] [-i iotype] [-p pid] [-n name] [duration]
DESCRIPTION¶
iosnoop prints block device I/O events as they happen, with useful details such
as PID, device, I/O type, block number, I/O size, and latency.
This traces disk I/O at the block device interface, using the block:
tracepoints. This can help characterize the I/O requested for the storage
devices and their resulting performance. I/O completions can also be studied
event-by-event for debugging disk and controller I/O scheduling issues.
NOTE: Use of a duration buffers I/O, which reduces overheads, but this also
introduces a limit to the number of I/O that will be captured. See the
duration section in OPTIONS.
Since this uses ftrace, only the root user can use this tool.
REQUIREMENTS¶
FTRACE CONFIG, and the tracepoints block:block_rq_insert, block:block_rq_issue,
and block:block_rq_complete, which you may already have enabled and available
on recent Linux kernels. And awk.
OPTIONS¶
- -d device
- Only show I/O issued by this device. (eg, "202,1"). This matches
the DEV column in the iosnoop output, and is filtered in-kernel.
- -i iotype
- Only show I/O issued that matches this I/O type. This matches the TYPE
column in the iosnoop output, and wildcards ("*") can be used at
the beginning or end (only). Eg, "*R*" matches all reads. This
is filtered in-kernel.
- -p PID
- Only show I/O issued by this PID. This filters in-kernel. Note that I/O
may be issued indirectly; for example, as the result of a memory
allocation, causing dirty buffers (maybe from another PID) to be written
to storage.
- -n name
- Only show I/O issued by processes with this name. Partial strings and
regular expressions are allowed. This is a post-filter, so all I/O is
traced and then filtered in user space. As with PID, this includes
indirectly issued I/O.
- -h
- Print usage message.
- -Q
- Use the queue insertion as the I/O start time (block:block_rq_insert),
instead of the issue time (block:block_rq_issue). This affects the STARTs
and LATms columns.
- -s
- Include a column for the start time (issue time) of the I/O, in seconds.
If the -Q option is used, this is the time the I/O is inserted on the
block I/O queue.
- -t
- Include a column for the completion time of the I/O, in seconds.
- duration
- Set the duration of tracing, in seconds. Trace output will be buffered and
printed at the end. This also reduces overheads by buffering in-kernel,
instead of printing events as they occur.
The ftrace buffer has a fixed size per-CPU (see
/sys/kernel/debug/tracing/buffer_size_kb). If you think events are
missing, try increasing that size (the bufsize_kb setting in iosnoop).
With the default setting (4 Mbytes), I'd expect this to happen around 50k
I/O.
EXAMPLES¶
- Default output, print I/O activity as it occurs:
- # iosnoop
- Buffer for 5 seconds (lower overhead) and write to a file:
- # iosnoop 5 > outfile
- Trace reads only:
- # iosnoop -i '*R*'
- Trace I/O issued to device 202,1 only:
- # iosnoop -d 202,1
- Include I/O start and completion timestamps:
- # iosnoop -ts
- Include block I/O queueing time:
- # iosnoop -Q Include I/O queueing and completion timestamps: #
iosnop -Qts
- Trace I/O issued when PID 181 was on-CPU only:
- # iosnoop -p 181
FIELDS¶
- COMM
- Process name (command) for the PID that was on-CPU when the I/O was
issued. See PID.
- PID
- Process ID which was on-CPU when the I/O was issued. This will usually be
the process directly requesting I/O, however, it may also include indirect
I/O. For example, a memory allocation by this PID which causes dirty
memory from another PID to be flushed to disk.
- TYPE
- Type of I/O. R=read, W=write, M=metadata, S=sync, A=readahead, F=flush or
FUA (force unit access), D=discard, E=secure, N=null (not RWFD).
- DEV
- Storage device ID.
- BLOCK
- Disk block for the operation (location, relative to this device).
- BYTES
- Size of the I/O, in bytes.
- LATms
- Latency (time) for the I/O, in milliseconds.
OVERHEAD¶
By default, iosnoop works without buffering, printing I/O events as they happen
(uses trace_pipe), context switching and consuming CPU to do so. This has a
limit of about 10,000 IOPS (depending on your platform), at which point
iosnoop will be consuming 1 CPU. The duration mode uses buffering, and can
handle much higher IOPS rates, however, the buffer has a limit of about 50,000
I/O, after which events will be dropped. You can tune this with bufsize_kb,
which is per-CPU. Also note that the "-n" option is currently
post-filtered, so all events are traced.
The overhead may be acceptable in many situations. If it isn't, this tool can be
reimplemented in C, or using a different tracer (eg, perf_events, SystemTap,
ktap.)
SOURCE¶
This is from the perf-tools collection.
- https://github.com/brendangregg/perf-tools
Also look under the examples directory for a text file containing example usage,
output, and commentary for this tool.
Linux
STABILITY¶
Unstable - in development.
AUTHOR¶
Brendan Gregg
SEE ALSO¶
iolatency(8),
iostat(1)
