Scroll to navigation

DRGN(1) drgn DRGN(1)

NAME

drgn - drgn 0.0.20

drgn (pronounced "dragon") is a debugger with an emphasis on programmability. drgn exposes the types and variables in a program for easy, expressive scripting in Python. For example, you can debug the Linux kernel:

>>> from drgn.helpers.linux import list_for_each_entry
>>> for mod in list_for_each_entry('struct module',
...                                prog['modules'].address_of_(),
...                                'list'):
...    if mod.refcnt.counter > 10:
...        print(mod.name)
...
(char [56])"snd"
(char [56])"evdev"
(char [56])"i915"


Although other debuggers like GDB have scripting support, drgn aims to make scripting as natural as possible so that debugging feels like coding. This makes it well-suited for introspecting the complex, inter-connected state in large programs. It is also designed as a library that can be used to build debugging and introspection tools; see the official tools.

drgn was developed at Meta for debugging the Linux kernel (as an alternative to the crash utility), but it can also debug userspace programs written in C. C++ support is in progress.

In addition to the main Python API, an experimental C library, libdrgn, is also available.

See the Installation instructions. Then, start with the User Guide.

LICENSE

Copyright (c) Meta Platforms, Inc. and affiliates.

drgn is licensed under the GPLv3 or later.

ACKNOWLEDGEMENTS

drgn is named after this because dragons eat dwarves.

TABLE OF CONTENTS

Installation

There are several options for installing drgn.

Dependencies

drgn depends on:

  • Python 3.6 or newer
  • elfutils 0.165 or newer

It optionally depends on:

libkdumpfile for makedumpfile compressed kernel core dump format support

The build requires:

  • GCC
  • GNU Make
  • pkgconf
  • setuptools

Building from the Git repository (rather than a release tarball) additionally requires:

  • autoconf
  • automake
  • libtool

Installation

Package Manager

drgn can be installed using the package manager on some Linux distributions.

Fedora >= 32

$ sudo dnf install drgn


RHEL/CentOS >= 8

Enable EPEL. Then:

$ sudo dnf install drgn


Arch Linux

Install the drgn package from the AUR.


pip

If your Linux distribution doesn't package the latest release of drgn, you can install it with pip.

First, install pip. Then, run:

$ sudo pip3 install drgn


This will install a binary wheel by default. If you get a build error, then pip wasn't able to use the binary wheel. Install the dependencies listed below and try again.

Note that RHEL/CentOS 6, Debian Stretch, Ubuntu Trusty, and Ubuntu Xenial (and older) ship Python versions which are too old. Python 3.6 or newer must be installed.

From Source

To get the development version of drgn, you will need to build it from source. First, install dependencies:

Fedora/RHEL/CentOS

$ sudo dnf install autoconf automake elfutils-devel gcc git libtool make pkgconf python3 python3-devel python3-pip python3-setuptools


Replace dnf with yum for RHEL/CentOS < 8.

Debian/Ubuntu

$ sudo apt-get install autoconf automake gcc git liblzma-dev libelf-dev libdw-dev libtool make pkgconf python3 python3-dev python3-pip python3-setuptools zlib1g-dev


Arch Linux

$ sudo pacman -S --needed autoconf automake gcc git libelf libtool make pkgconf python python-pip python-setuptools



Optionally, install libkdumpfile if you want support for the makedumpfile compressed kernel core dump format. libkdumpfile is currently only packaged on Fedora and EPEL. For other distributions, you must install it manually.

Then, run:

$ git clone https://github.com/osandov/drgn.git
$ cd drgn
$ python3 setup.py build
$ sudo python3 setup.py install


Virtual Environment

The above options all install drgn globally. You can also install drgn in a virtual environment, either with pip:

$ python3 -m venv drgnenv
$ source drgnenv/bin/activate
(drgnenv) $ pip3 install drgn
(drgnenv) $ drgn --help


Or from source:

$ python3 -m venv drgnenv
$ source drgnenv/bin/activate
(drgnenv) $ python3 setup.py install
(drgnenv) $ drgn --help


Running Locally

If you build drgn from source, you can also run it without installing it:

$ python3 setup.py build_ext -i
$ python3 -m drgn --help


User Guide

Quick Start

drgn debugs the running kernel by default; run sudo drgn. To debug a running program, run sudo drgn -p $PID. To debug a core dump (either a kernel vmcore or a userspace core dump), run drgn -c $PATH. Make sure to install debugging symbols for whatever you are debugging.

Then, you can access variables in the program with prog['name'] and access structure members with .:

$ sudo drgn
>>> prog['init_task'].comm
(char [16])"swapper/0"


You can use various predefined helpers:

>>> len(list(bpf_prog_for_each(prog)))
11
>>> task = find_task(prog, 115)
>>> cmdline(task)
[b'findmnt', b'-p']


You can get stack traces with prog.stack_trace() and access parameters or local variables with stack_trace['name']:

>>> trace = prog.stack_trace(task)
>>> trace[5]
#5 at 0xffffffff8a5a32d0 (do_sys_poll+0x400/0x578) in do_poll at ./fs/select.c:961:8 (inlined)
>>> poll_list = trace[5]['list']
>>> file = fget(task, poll_list.entries[0].fd)
>>> d_path(file.f_path.address_of_())
b'/proc/115/mountinfo'


Core Concepts

The most important interfaces in drgn are programs, objects, and helpers.

Programs

A program being debugged is represented by an instance of the drgn.Program class. The drgn CLI is initialized with a Program named prog; unless you are using the drgn library directly, this is usually the only Program you will need.

A Program is used to look up type definitions, access variables, and read arbitrary memory:

>>> prog.type('unsigned long')
prog.int_type(name='unsigned long', size=8, is_signed=False)
>>> prog['jiffies']
Object(prog, 'volatile unsigned long', address=0xffffffffbe405000)
>>> prog.read(0xffffffffbe411e10, 16)
b'swapper/0\x00\x00\x00\x00\x00\x00\x00'


The drgn.Program.type(), drgn.Program.variable(), drgn.Program.constant(), and drgn.Program.function() methods look up those various things in a program. drgn.Program.read() reads memory from the program's address space. The [] operator looks up a variable, constant, or function:

>>> prog['jiffies'] == prog.variable('jiffies')
True


It is usually more convenient to use the [] operator rather than the variable(), constant(), or function() methods unless the program has multiple objects with the same name, in which case the methods provide more control.

Objects

Variables, constants, functions, and computed values are all called objects in drgn. Objects are represented by the drgn.Object class. An object may exist in the memory of the program (a reference):

>>> Object(prog, 'int', address=0xffffffffc09031a0)


Or, an object may be a constant or temporary computed value (a value):

>>> Object(prog, 'int', value=4)


What makes drgn scripts expressive is that objects can be used almost exactly like they would be in the program's own source code. For example, structure members can be accessed with the dot (.) operator, arrays can be subscripted with [], arithmetic can be performed, and objects can be compared:

>>> print(prog['init_task'].comm[0])
(char)115
>>> print(repr(prog['init_task'].nsproxy.mnt_ns.mounts + 1))
Object(prog, 'unsigned int', value=34)
>>> prog['init_task'].nsproxy.mnt_ns.pending_mounts > 0
False


A common use case is converting a drgn.Object to a Python value so it can be used by a standard Python library. There are a few ways to do this:

  • The drgn.Object.value_() method gets the value of the object with the directly corresponding Python type (i.e., integers and pointers become int, floating-point types become float, booleans become bool, arrays become list, structures and unions become dict).
  • The drgn.Object.string_() method gets a null-terminated string as bytes from an array or pointer.
  • The int(), float(), and bool() functions do an explicit conversion to that Python type.

Objects have several attributes; the most important are drgn.Object.prog_ and drgn.Object.type_. The former is the drgn.Program that the object is from, and the latter is the drgn.Type of the object.

Note that all attributes and methods of the Object class end with an underscore (_) in order to avoid conflicting with structure or union members. The Object attributes and methods always take precedence; use drgn.Object.member_() if there is a conflict.

References vs. Values

The main difference between reference objects and value objects is how they are evaluated. References are read from the program's memory every time they are evaluated; values simply return the stored value (drgn.Object.read_() reads a reference object and returns it as a value object):

>>> import time
>>> jiffies = prog['jiffies']
>>> jiffies.value_()
4391639989
>>> time.sleep(1)
>>> jiffies.value_()
4391640290
>>> jiffies2 = jiffies.read_()
>>> jiffies2.value_()
4391640291
>>> time.sleep(1)
>>> jiffies2.value_()
4391640291
>>> jiffies.value_()
4391640593


References have a drgn.Object.address_ attribute, which is the object's address as a Python int. This is slightly different from the drgn.Object.address_of_() method, which returns the address as a drgn.Object. Of course, both references and values can have a pointer type; address_ refers to the address of the pointer object itself, and drgn.Object.value_() refers to the value of the pointer (i.e., the address it points to):

>>> address = prog['jiffies'].address_
>>> type(address)
<class 'int'>
>>> print(hex(address))
0xffffffffbe405000
>>> jiffiesp = prog['jiffies'].address_of_()
>>> jiffiesp
Object(prog, 'volatile unsigned long *', value=0xffffffffbe405000)
>>> print(hex(jiffiesp.value_()))
0xffffffffbe405000


Absent Objects

In addition to reference objects and value objects, objects may also be absent.

>>> Object(prog, "int").value_()
Traceback (most recent call last):

File "<console>", line 1, in <module> _drgn.ObjectAbsentError: object absent

This represents an object whose value or address is not known. For example, this can happen if the object was optimized out of the program by the compiler.

Any attempt to operate on an absent object results in a drgn.ObjectAbsentError exception, although basic information including its type may still be accessed.

Helpers

Some programs have common data structures that you may want to examine. For example, consider linked lists in the Linux kernel:

struct list_head {

struct list_head *next, *prev; }; #define list_for_each(pos, head) \
for (pos = (head)->next; pos != (head); pos = pos->next)


When working with these lists, you'd probably want to define a function:

def list_for_each(head):

pos = head.next
while pos != head:
yield pos
pos = pos.next


Then, you could use it like so for any list you need to look at:

>>> for pos in list_for_each(head):
...     do_something_with(pos)


Of course, it would be a waste of time and effort for everyone to have to define these helpers for themselves, so drgn includes a collection of helpers for many use cases. See Helpers.

Validators

Validators are a special category of helpers that check the consistency of a data structure. In general, helpers assume that the data structures that they examine are valid. Validators do not make this assumption and do additional (potentially expensive) checks to detect broken invariants, corruption, etc.

Validators raise drgn.helpers.ValidationError if the data structure is not valid or drgn.FaultError if the data structure is invalid in a way that causes a bad memory access. They have names prefixed with validate_.

For example, drgn.helpers.linux.list.validate_list() checks the consistency of a linked list in the Linux kernel (in particular, the consistency of the next and prev pointers):

>>> validate_list(prog["my_list"].address_of_())
drgn.helpers.ValidationError: (struct list_head *)0xffffffffc029e460 next 0xffffffffc029e000 has prev 0xffffffffc029e450


drgn.helpers.linux.list.validate_list_for_each_entry() does the same checks while also returning the entries in the list for further validation:

def validate_my_list(prog):

for entry in validate_list_for_each_entry(
"struct my_entry",
prog["my_list"].address_of_(),
"list",
):
if entry.value < 0:
raise ValidationError("list contains negative entry")


Other Concepts

In addition to the core concepts above, drgn provides a few additional abstractions.

Threads

The drgn.Thread class represents a thread. drgn.Program.threads(), drgn.Program.thread(), drgn.Program.main_thread(), and drgn.Program.crashed_thread() can be used to find threads:

>>> for thread in prog.threads():
...     print(thread.tid)
...
39143
39144
>>> print(prog.main_thread().tid)
39143
>>> print(prog.crashed_thread().tid)
39144


Stack Traces

drgn represents stack traces with the drgn.StackTrace and drgn.StackFrame classes. drgn.Thread.stack_trace() and drgn.Program.stack_trace() return the call stack for a thread. The [] operator looks up an object in the scope of a StackFrame:

>>> trace = prog.stack_trace(115)
>>> trace
#0  context_switch (./kernel/sched/core.c:4683:2)
#1  __schedule (./kernel/sched/core.c:5940:8)
#2  schedule (./kernel/sched/core.c:6019:3)
#3  schedule_hrtimeout_range_clock (./kernel/time/hrtimer.c:2148:3)
#4  poll_schedule_timeout (./fs/select.c:243:8)
#5  do_poll (./fs/select.c:961:8)
#6  do_sys_poll (./fs/select.c:1011:12)
#7  __do_sys_poll (./fs/select.c:1076:8)
#8  __se_sys_poll (./fs/select.c:1064:1)
#9  __x64_sys_poll (./fs/select.c:1064:1)
#10 do_syscall_x64 (./arch/x86/entry/common.c:50:14)
#11 do_syscall_64 (./arch/x86/entry/common.c:80:7)
#12 entry_SYSCALL_64+0x7c/0x15b (./arch/x86/entry/entry_64.S:113)
#13 0x7f3344072af7
>>> trace[5]
#5 at 0xffffffff8a5a32d0 (do_sys_poll+0x400/0x578) in do_poll at ./fs/select.c:961:8 (inlined)
>>> prog['do_poll']
(int (struct poll_list *list, struct poll_wqueues *wait, struct timespec64 *end_time))<absent>
>>> trace[5]['list']
*(struct poll_list *)0xffffacca402e3b50 = {

.next = (struct poll_list *)0x0,
.len = (int)1,
.entries = (struct pollfd []){}, }


Symbols

The symbol table of a program is a list of identifiers along with their address and size. drgn represents symbols with the drgn.Symbol class, which is returned by drgn.Program.symbol().

Types

drgn automatically obtains type definitions from the program. Types are represented by the drgn.Type class and created by various factory functions like drgn.Program.int_type():

>>> prog.type('int')
prog.int_type(name='int', size=4, is_signed=True)


You won't usually need to work with types directly, but see Types if you do.

Platforms

Certain operations and objects in a program are platform-dependent; drgn allows accessing the platform that a program runs with the drgn.Platform class.

Command Line Interface

The drgn CLI is basically a wrapper around the drgn library which automatically creates a drgn.Program. The CLI can be run in interactive mode or script mode.

Script Mode

Script mode is useful for reusable scripts. Simply pass the path to the script along with any arguments:

$ cat script.py
import sys
from drgn.helpers.linux import find_task
pid = int(sys.argv[1])
uid = find_task(prog, pid).cred.uid.val.value_()
print(f'PID {pid} is being run by UID {uid}')
$ sudo drgn script.py 601
PID 601 is being run by UID 1000


It's even possible to run drgn scripts directly with the proper shebang:

$ cat script2.py
#!/usr/bin/env drgn
mounts = prog['init_task'].nsproxy.mnt_ns.mounts.value_()
print(f'You have {mounts} filesystems mounted')
$ sudo ./script2.py
You have 36 filesystems mounted


Interactive Mode

Interactive mode uses the Python interpreter's interactive mode and adds a few nice features, including:

  • History
  • Tab completion
  • Automatic import of relevant helpers
  • Pretty printing of objects and types

The default behavior of the Python REPL is to print the output of repr(). For drgn.Object and drgn.Type, this is a raw representation:

>>> print(repr(prog['jiffies']))
Object(prog, 'volatile unsigned long', address=0xffffffffbe405000)
>>> print(repr(prog.type('atomic_t')))
prog.typedef_type(name='atomic_t', type=prog.struct_type(tag=None, size=4, members=(TypeMember(prog.type('int'), name='counter', bit_offset=0),)))


The standard print() function uses the output of str(). For drgn objects and types, this is a representation in programming language syntax:

>>> print(prog['jiffies'])
(volatile unsigned long)4395387628
>>> print(prog.type('atomic_t'))
typedef struct {

int counter; } atomic_t


In interactive mode, the drgn CLI automatically uses str() instead of repr() for objects and types, so you don't need to call print() explicitly:

$ sudo drgn
>>> prog['jiffies']
(volatile unsigned long)4395387628
>>> prog.type('atomic_t')
typedef struct {

int counter; } atomic_t


Next Steps

Refer to the API Reference. Look through the Helpers. Read some Case Studies. Browse through the official examples and tools.

Advanced Usage

The User Guide covers basic usage of drgn, but drgn also supports more advanced use cases which are covered here.

Loading Debugging Symbols

drgn will automatically load debugging information based on the debugged program (e.g., from loaded kernel modules or loaded shared libraries). drgn.Program.load_debug_info() can be used to load additional debugging information:

>>> prog.load_debug_info(['./libfoo.so', '/usr/lib/libbar.so'])


Library

In addition to the CLI, drgn is also available as a library. drgn.program_from_core_dump(), drgn.program_from_kernel(), and drgn.program_from_pid() correspond to the -c, -k, and -p command line options, respectively; they return a drgn.Program that can be used just like the one initialized by the CLI:

>>> import drgn
>>> prog = drgn.program_from_kernel()


C Library

The core functionality of drgn is implemented in C and is available as a C library, libdrgn. See drgn.h.

Full documentation can be generated by running doxygen in the libdrgn directory of the source code. Note that the API and ABI are not yet stable.

Custom Programs

The main components of a drgn.Program are the program memory, types, and symbols. The CLI and equivalent library interfaces automatically determine these. However, it is also possible to create a "blank" Program and plug in the main components.

drgn.Program.add_memory_segment() defines a range of memory and how to read that memory. The following example uses a Btrfs filesystem image as the program "memory":

import drgn
import os
import sys
def btrfs_debugger(dev):

file = open(dev, 'rb')
size = file.seek(0, 2)
def read_file(address, count, physical, offset):
file.seek(offset)
return file.read(count)
platform = drgn.Platform(drgn.Architecture.UNKNOWN,
drgn.PlatformFlags.IS_LITTLE_ENDIAN)
prog = drgn.Program(platform)
prog.add_memory_segment(0, size, read_file)
prog.load_debug_info([f'/lib/modules/{os.uname().release}/kernel/fs/btrfs/btrfs.ko'])
return prog prog = btrfs_debugger(sys.argv[1] if len(sys.argv) >= 2 else '/dev/sda') print(drgn.Object(prog, 'struct btrfs_super_block', address=65536))


drgn.Program.add_type_finder() and drgn.Program.add_symbol_finder() are the equivalent methods for plugging in types and symbols.

Environment Variables

Some of drgn's behavior can be modified through environment variables:

The maximum number of individual errors to report in a drgn.MissingDebugInfoError. Any additional errors are truncated. The default is 5; -1 is unlimited.
Whether to prefer using ORC over DWARF for stack unwinding (0 or 1). The default is 0. Note that drgn will always fall back to ORC for functions lacking DWARF call frame information and vice versa. This environment variable is mainly intended for testing and may be ignored in the future.
Whether drgn should use libdwfl to find debugging information for core dumps instead of its own implementation (0 or 1). The default is 0. This environment variable is mainly intended as an escape hatch in case of bugs in drgn's implementation and will be ignored in the future.
Whether drgn should use libkdumpfile for ELF vmcores (0 or 1). The default is 0. This functionality will be removed in the future.
Whether drgn should use /proc/modules and /sys/module to find loaded kernel modules for the running kernel instead of getting them from the core dump (0 or 1). The default is 1. This environment variable is mainly intended for testing and may be ignored in the future.

API Reference

Programs

A Program represents a crashed or running program. It can be used to lookup type definitions, access variables, and read arbitrary memory.

The main functionality of a Program is looking up objects (i.e., variables, constants, or functions). This is usually done with the [] operator.

Create a Program with no target program. It is usually more convenient to use one of the Program Constructors.
platform (Optional[Platform]) -- The platform of the program, or None if it should be determined automatically when a core dump or symbol file is added.


Flags which apply to this program.
ProgramFlags


Platform that this program runs on, or None if it has not been determined yet.
Optional[Platform]


Default programming language of the program.

This is used for interpreting the type name given to type() and when creating an Object without an explicit type.

For the Linux kernel, this defaults to Language.C. For userspace programs, this defaults to the language of main in the program, falling back to Language.C. This heuristic may change in the future.

This can be explicitly set to a different language (e.g., if the heuristic was incorrect).

Language


__getitem__(name)
Implement self[name]. Get the object (variable, constant, or function) with the given name.

This is equivalent to prog.object(name) except that this raises KeyError instead of LookupError if no objects with the given name are found.

If there are multiple objects with the same name, one is returned arbitrarily. In this case, the variable(), constant(), function(), or object() methods can be used instead.

>>> prog['jiffies']
Object(prog, 'volatile unsigned long', address=0xffffffff94c05000)
    
name (str) -- Object name.
Object


__contains__(name)
Implement name in self. Return whether an object (variable, constant, or function) with the given name exists in the program.
name (str) -- Object name.
bool


Get the variable with the given name.

>>> prog.variable('jiffies')
Object(prog, 'volatile unsigned long', address=0xffffffff94c05000)
    

This is equivalent to prog.object(name, FindObjectFlags.VARIABLE, filename).

  • name (str) -- The variable name.
  • filename (Optional[str]) -- The source code file that contains the definition. See Filenames.

LookupError -- if no variables with the given name are found in the given file
Object


Get the constant (e.g., enumeration constant) with the given name.

Note that support for macro constants is not yet implemented for DWARF files, and most compilers don't generate macro debugging information by default anyways.

>>> prog.constant('PIDTYPE_MAX')
Object(prog, 'enum pid_type', value=4)
    

This is equivalent to prog.object(name, FindObjectFlags.CONSTANT, filename).

  • name (str) -- The constant name.
  • filename (Optional[str]) -- The source code file that contains the definition. See Filenames.

LookupError -- if no constants with the given name are found in the given file
Object


Get the function with the given name.

>>> prog.function('schedule')
Object(prog, 'void (void)', address=0xffffffff94392370)
    

This is equivalent to prog.object(name, FindObjectFlags.FUNCTION, filename).

  • name (str) -- The function name.
  • filename (Optional[str]) -- The source code file that contains the definition. See Filenames.

LookupError -- if no functions with the given name are found in the given file
Object


Get the object (variable, constant, or function) with the given name.
  • name (str) -- The object name.
  • flags (FindObjectFlags) -- Flags indicating what kind of object to look for.
  • filename (Optional[str]) -- The source code file that contains the definition. See Filenames.

LookupError -- if no objects with the given name are found in the given file
Object


Get a symbol containing the given address, or a symbol with the given name.

Global symbols are preferred over weak symbols, and weak symbols are preferred over other symbols. In other words: if a matching SymbolBinding.GLOBAL or SymbolBinding.UNIQUE symbol is found, it is returned. Otherwise, if a matching SymbolBinding.WEAK symbol is found, it is returned. Otherwise, any matching symbol (e.g., SymbolBinding.LOCAL) is returned. If there are multiple matching symbols with the same binding, one is returned arbitrarily. To retrieve all matching symbols, use symbols().

address_or_name (Union[IntegerLike, str]) -- Address or name to search for. This parameter is positional-only.
LookupError -- if no symbol contains the given address or matches the given name
Symbol


Get a list of global and local symbols, optionally matching a name or address.

If a string argument is given, this returns all symbols matching that name. If an integer-like argument given, this returns a list of all symbols containing that address. If no argument is given, all symbols in the program are returned. In all cases, the symbols are returned in an unspecified order.

address_or_name (Union[None, IntegerLike, str]) -- Address or name to search for. This parameter is positional-only.
List[Symbol]


Get the stack trace for the given thread in the program.

thread may be a thread ID (as defined by gettid(2)), in which case this will unwind the stack for the thread with that ID. The ID may be a Python int or an integer Object

thread may also be a struct pt_regs or struct pt_regs * object, in which case the initial register values will be fetched from that object.

Finally, if debugging the Linux kernel, thread may be a struct task_struct * object, in which case this will unwind the stack for that task. See drgn.helpers.linux.pid.find_task().

This is implemented for the Linux kernel (both live and core dumps) as well as userspace core dumps; it is not yet implemented for live userspace processes.

thread (Union[Object, IntegerLike]) -- Thread ID, struct pt_regs object, or struct task_struct * object.
StackTrace


Get the type with the given name.

>>> prog.type('long')
prog.int_type(name='long', size=8, is_signed=True)
    
  • name (str) -- The type name.
  • filename (Optional[str]) -- The source code file that contains the definition. See Filenames.

LookupError -- if no types with the given name are found in the given file
Type


Return the given type.

This is mainly useful so that helpers can use prog.type() to get a Type regardless of whether they were given a str or a Type. For example:

def my_helper(obj: Object, type: Union[str, Type]) -> bool:

# type may be str or Type.
type = obj.prog_.type(type)
# type is now always Type.
return sizeof(obj) > sizeof(type)


type (Type) -- Type.
The exact same type.
Type


Get an iterator over all of the threads in the program.
Iterator[Thread]


Get the thread with the given thread ID.
tid (IntegerLike) --

Thread ID (as defined by gettid(2)).

LookupError -- if no thread has the given thread ID
Thread


Get the main thread of the program.

This is only defined for userspace programs.

ValueError -- if the program is the Linux kernel
Thread


Get the thread that caused the program to crash.

For userspace programs, this is the thread that received the fatal signal (e.g., SIGSEGV or SIGQUIT).

For the kernel, this is the thread that panicked (either directly or as a result of an oops, BUG_ON(), etc.).

ValueError -- if the program is live (i.e., not a core dump)
Thread


Read size bytes of memory starting at address in the program. The address may be virtual (the default) or physical if the program supports it.

>>> prog.read(0xffffffffbe012b40, 16)
b'swapper/0'
    
  • address (IntegerLike) -- The starting address.
  • size (IntegerLike) -- The number of bytes to read.
  • physical (bool) -- Whether address is a physical memory address. If False, then it is a virtual memory address. Physical memory can usually only be read when the program is an operating system kernel.

  • FaultError -- if the address range is invalid or the type of address (physical or virtual) is not supported by the program
  • ValueError -- if size is negative

bytes






Read an unsigned integer from the program's memory in the program's byte order.

read_u8(), read_u16(), read_u32(), and read_u64() read an 8-, 16-, 32-, or 64-bit unsigned integer, respectively. read_word() reads a program word-sized unsigned integer.

For signed integers, alternate byte order, or other formats, you can use read() and int.from_bytes() or the struct module.

  • address (IntegerLike) -- Address of the integer.
  • physical (bool) -- Whether address is a physical memory address; see read().

FaultError -- if the address is invalid; see read()
int


Define a region of memory in the program.

If it overlaps a previously registered segment, the new segment takes precedence.

  • address (IntegerLike) -- Address of the segment.
  • size (IntegerLike) -- Size of the segment in bytes.
  • physical (bool) -- Whether to add a physical memory segment. If False, then this adds a virtual memory segment.
  • read_fn (Callable[[int, int, int, bool], bytes]) -- Callable to call to read memory from the segment. It is passed the address being read from, the number of bytes to read, the offset in bytes from the beginning of the segment, and whether the address is physical: (address, count, offset, physical). It should return the requested number of bytes as bytes or another buffer type.

None


Register a callback for finding types in the program.

Callbacks are called in reverse order of the order they were added until the type is found. So, more recently added callbacks take precedence.

fn (Callable[[TypeKind, str, Optional[str]], Type]) -- Callable taking a TypeKind, name, and filename: (kind, name, filename). The filename should be matched with filename_matches(). This should return a Type.
None


Register a callback for finding objects in the program.

Callbacks are called in reverse order of the order they were added until the object is found. So, more recently added callbacks take precedence.

fn (Callable[[Program, str, FindObjectFlags, Optional[str]], Object]) -- Callable taking a program, name, FindObjectFlags, and filename: (prog, name, flags, filename). The filename should be matched with filename_matches(). This should return an Object.
None


Set the program to a core dump.

This loads the memory segments from the core dump and determines the mapped executable and libraries. It does not load any debugging symbols; see load_default_debug_info().

path (Path) -- Core dump file path.
None


Set the program to the running operating system kernel.

This loads the memory of the running kernel and thus requires root privileges. It does not load any debugging symbols; see load_default_debug_info().



Set the program to a running process.

This loads the memory of the process and determines the mapped executable and libraries. It does not load any debugging symbols; see load_default_debug_info().

pid (int) -- Process ID.
None


Load debugging information for a list of executable or library files.

Note that this is parallelized, so it is usually faster to load multiple files at once rather than one by one.

  • paths (Optional[Iterable[Path]]) -- Paths of binary files.
  • default (bool) --

    Also load debugging information which can automatically be determined from the program.

    For the Linux kernel, this tries to load vmlinux and any loaded kernel modules from a few standard locations.

    For userspace programs, this tries to load the executable and any loaded libraries.

    This implies main=True.

  • main (bool) --

    Also load debugging information for the main executable.

    For the Linux kernel, this tries to load vmlinux.

    This is currently ignored for userspace programs.


MissingDebugInfoError -- if debugging information was not available for some files; other files with debugging information are still loaded
None


Load debugging information which can automatically be determined from the program.

This is equivalent to load_debug_info(None, True).



Dictionary for caching program metadata.

This isn't used by drgn itself. It is intended to be used by helpers to cache metadata about the program. For example, if a helper for a program depends on the program version or an optional feature, the helper can detect it and cache it for subsequent invocations:

def my_helper(prog):

try:
have_foo = prog.cache['have_foo']
except KeyError:
have_foo = detect_foo_feature(prog)
prog.cache['have_foo'] = have_foo
if have_foo:
return prog['foo']
else:
return prog['bar']


Dict[Any, Any]



Bases: enum.Flag

ProgramFlags are flags that can apply to a Program (e.g., about what kind of program it is).

The program is the Linux kernel.

The program is currently running (e.g., it is the running operating system kernel or a running process).


Bases: enum.Flag

FindObjectFlags are flags for Program.object(). These can be combined to search for multiple kinds of objects at once.






A thread in a program.
Thread ID (as defined by gettid(2)).
int


If the program is the Linux kernel, the struct task_struct * object for this thread. Otherwise, not defined.
Object


Get the stack trace for this thread.

This is equivalent to prog.stack_trace(thread.tid). See Program.stack_trace().

StackTrace



Filenames

The Program.type(), Program.object(), Program.variable(), Program.constant(), and Program.function() methods all take a filename parameter to distinguish between multiple definitions with the same name. The filename refers to the source code file that contains the definition. It is matched with filename_matches(). If multiple definitions match, one is returned arbitrarily.

Return whether a filename containing a definition (haystack) matches a filename being searched for (needle).

The filename is matched from right to left, so 'stdio.h', 'include/stdio.h', 'usr/include/stdio.h', and '/usr/include/stdio.h' would all match a definition in /usr/include/stdio.h. If needle is None or empty, it matches any definition. If haystack is None or empty, it only matches if needle is also None or empty.

  • haystack (Optional[str]) -- Path of file containing definition.
  • needle (Optional[str]) -- Filename to match.

bool


Program Constructors

The drgn command line interface automatically creates a Program named prog. However, drgn may also be used as a library without the CLI, in which case a Program must be created manually.

Create a Program from a core dump file. The type of program (e.g., userspace or kernel) is determined automatically.
path (Path) -- Core dump file path.
Program


Create a Program from the running operating system kernel. This requires root privileges.
Program


Create a Program from a running program with the given PID. This requires appropriate permissions (on Linux, ptrace(2) attach permissions).
pid (int) -- Process ID of the program to debug.
Program


Platforms

A Platform represents the environment (i.e., architecture and ABI) that a program runs on.
Create a Platform.
  • arch (Architecture) -- Platform.arch
  • flags (Optional[PlatformFlags]) -- Platform.flags; if None, default flags for the architecture are used.



Instruction set architecture of this platform.
Architecture


Flags which apply to this platform.
PlatformFlags


Processor registers on this platform.
Sequence[Register]



Bases: enum.Enum

An Architecture represents an instruction set architecture.

The x86-64 architecture, a.k.a. AMD64 or Intel 64.

The 32-bit x86 architecture, a.k.a. i386 or IA-32.

The AArch64 architecture, a.k.a. ARM64.

The 32-bit Arm architecture.

The 64-bit PowerPC architecture.

The 64-bit RISC-V architecture.

The 32-bit RISC-V architecture.

An architecture which is not known to drgn. Certain features are not available when the architecture is unknown, but most of drgn will still work.


Bases: enum.Flag

PlatformFlags are flags describing a Platform.

Platform is 64-bit.

Platform is little-endian.


A Register represents information about a processor register.
Names of this register.
Sequence[str]



The platform of the host which is running drgn.
Platform


Languages

A Language represents a programming language supported by drgn.

This class cannot be constructed; there are singletons for the supported languages.

Name of the programming language.
str


The C programming language.
Language


The C++ programming language.
Language



Objects

An Object represents a symbol or value in a program. An object may exist in the memory of the program (a reference), it may be a constant or temporary computed value (a value), or it may be absent entirely (an absent object).

All instances of this class have two attributes: prog_, the program that the object is from; and type_, the type of the object. Reference objects also have an address_ and a bit_offset_. Objects may also have a bit_field_size_.

repr() of an object returns a Python representation of the object:

>>> print(repr(prog['jiffies']))
Object(prog, 'volatile unsigned long', address=0xffffffffbf005000)
    

str() returns a "pretty" representation of the object in programming language syntax:

>>> print(prog['jiffies'])
(volatile unsigned long)4326237045
    

The output format of str() can be modified by using the format_() method instead:

>>> sysname = prog['init_uts_ns'].name.sysname
>>> print(sysname)
(char [65])"Linux"
>>> print(sysname.format_(type_name=False))
"Linux"
>>> print(sysname.format_(string=False))
(char [65]){ 76, 105, 110, 117, 120 }
    

NOTE:

The drgn CLI is set up so that objects are displayed in the "pretty" format instead of with repr() (the latter is the default behavior of Python's interactive mode). Therefore, it's usually not necessary to call print() in the drgn CLI.


Objects support the following operators:

  • Arithmetic operators: +, -, *, /, %
  • Bitwise operators: <<, >>, &, |, ^, ~
  • Relational operators: ==, !=, <, >, <=, >=
  • Subscripting: [] (Python does not have a unary * operator, so pointers are dereferenced with ptr[0])
  • Member access: . (Python does not have a -> operator, so . is also used to access members of pointers to structures)
  • The address-of operator: drgn.Object.address_of_() (this is a method because Python does not have a & operator)
  • Array length: len()

These operators all have the semantics of the program's programming language. For example, adding two objects from a program written in C results in an object with a type and value according to the rules of C:

>>> Object(prog, 'unsigned long', 2**64 - 1) + Object(prog, 'int', 1)
Object(prog, 'unsigned long', value=0)

If only one operand to a binary operator is an object, the other operand will be converted to an object according to the language's rules for literals:

>>> Object(prog, 'char', 0) - 1
Object(prog, 'int', value=-1)

The standard int(), float(), and bool() functions convert an object to that Python type. Conversion to bool uses the programming language's notion of "truthiness". Additionally, certain Python functions will automatically coerce an object to the appropriate Python type (e.g., hex(), round(), and list subscripting).

Object attributes and methods are named with a trailing underscore to avoid conflicting with structure, union, or class members. The attributes and methods always take precedence; use member_() if there is a conflict.

Objects are usually obtained directly from a Program, but they can be constructed manually, as well (for example, if you got a variable address from a log file).

Create a value object given its type and value.
  • prog (Program) -- Program to create the object in.
  • type (Union[str, Type]) -- Type of the object.
  • value (Union[IntegerLike, float, bool, Mapping[str, Any], Sequence[Any]]) -- Value of the object. See value_().
  • bit_field_size (Optional[IntegerLike]) -- Size in bits of the object if it is a bit field. The default is None, which means the object is not a bit field.



Create a value object from a "literal".

This is used to emulate a literal number in the source code of the program. The type is deduced from value according to the language's rules for literals.

value (Union[int, float, bool]) -- Value of the literal.


Create a reference object.
  • address (IntegerLike) -- Address of the object in the program.
  • bit_offset (IntegerLike) -- Offset in bits from address to the beginning of the object.




Program that this object is from.
Program


Type of this object.
Type


Whether this object is absent.

This is False for all values and references (even if the reference has an invalid address).

bool


Address of this object if it is a reference, None if it is a value or absent.
Optional[int]


Offset in bits from this object's address to the beginning of the object if it is a reference, None otherwise. This can only be non-zero for scalars.
Optional[int]


Size in bits of this object if it is a bit field, None if it is not.
Optional[int]


__getattribute__(name)
Implement self.name.

If name is an attribute of the Object class, then this returns that attribute. Otherwise, it is equivalent to member_().

>>> print(prog['init_task'].pid)
(pid_t)0
    
name (str) -- Attribute name.
Object


__getitem__(idx)
Implement self[idx]. Get the array element at the given index.

>>> print(prog['init_task'].comm[0])
(char)115
    

This is only valid for pointers and arrays.

NOTE:

Negative indices behave as they would in the object's language (as opposed to the Python semantics of indexing from the end of the array).


idx (IntegerLike) -- The array index.
TypeError -- if this object is not a pointer or array
Object


__len__()
Implement len(self). Get the number of elements in this object.

>>> len(prog['init_task'].comm)
16
    

This is only valid for arrays.

TypeError -- if this object is not an array with complete type
int


Get the value of this object as a Python object.

For basic types (integer, floating-point, boolean), this returns an object of the directly corresponding Python type (int, float, bool). For pointers, this returns the address value of the pointer. For enums, this returns an int. For structures and unions, this returns a dict of members. For arrays, this returns a list of values.

  • FaultError -- if reading the object causes a bad memory access
  • TypeError -- if this object has an unreadable type (e.g., void)

Any


Read a null-terminated string pointed to by this object.

This is only valid for pointers and arrays. The element type is ignored; this operates byte-by-byte.

For pointers and flexible arrays, this stops at the first null byte.

For complete arrays, this stops at the first null byte or at the end of the array.

  • FaultError -- if reading the string causes a bad memory access
  • TypeError -- if this object is not a pointer or array

bytes


Get a member of this object.

This is valid for structures, unions, and pointers to either.

Normally the dot operator (.) can be used to accomplish the same thing, but this method can be used if there is a name conflict with an Object member or method.

name (str) -- Name of the member.
  • TypeError -- if this object is not a structure, union, class, or a pointer to one of those
  • LookupError -- if this object does not have a member with the given name

Object


Get a pointer to this object.

This corresponds to the address-of (&) operator in C. It is only possible for reference objects, as value objects don't have an address in the program.

As opposed to address_, this returns an Object, not an int.

ValueError -- if this object is a value
Object


Read this object (which may be a reference or a value) and return it as a value object.

This is useful if the object can change in the running program (but of course nothing stops the program from modifying the object while it is being read).

As opposed to value_(), this returns an Object, not a standard Python type.

  • FaultError -- if reading this object causes a bad memory access
  • TypeError -- if this object has an unreadable type (e.g., void)

Object


Return the binary representation of this object's value.
bytes


Return a value object from its binary representation.
  • prog (Program) -- Program to create the object in.
  • type (Union[str, Type]) -- Type of the object.
  • bytes (bytes) -- Buffer containing value of the object.
  • bit_offset (IntegerLike) -- Offset in bits from the beginning of bytes to the beginning of the object.
  • bit_field_size (Optional[IntegerLike]) -- Size in bits of the object if it is a bit field. The default is None, which means the object is not a bit field.

Object


Format this object in programming language syntax.

Various format options can be passed (as keyword arguments) to control the output. Options that aren't passed or are passed as None fall back to a default. Specifically, obj.format_() (i.e., with no passed options) is equivalent to str(obj).

>>> workqueues = prog['workqueues']
>>> print(workqueues)
(struct list_head){

.next = (struct list_head *)0xffff932ecfc0ae10,
.prev = (struct list_head *)0xffff932e3818fc10, } >>> print(workqueues.format_(type_name=False, ... member_type_names=False, ... member_names=False, ... members_same_line=True)) { 0xffff932ecfc0ae10, 0xffff932e3818fc10 }
  • columns (Optional[IntegerLike]) -- Number of columns to limit output to when the expression can be reasonably wrapped. Defaults to no limit.
  • dereference (Optional[bool]) -- If this object is a pointer, include the dereferenced value. This does not apply to structure, union, or class members, or array elements, as dereferencing those could lead to an infinite loop. Defaults to True.
  • symbolize (Optional[bool]) -- Include a symbol name and offset for pointer objects. Defaults to True.
  • string (Optional[bool]) -- Format the values of objects with string type as strings. For C, this applies to pointers to and arrays of char, signed char, and unsigned char. Defaults to True.
  • char (Optional[bool]) -- Format objects with character type as character literals. For C, this applies to char, signed char, and unsigned char. Defaults to False.
  • type_name (Optional[bool]) -- Include the type name of this object. Defaults to True.
  • member_type_names (Optional[bool]) -- Include the type names of structure, union, and class members. Defaults to True.
  • element_type_names (Optional[bool]) -- Include the type names of array elements. Defaults to False.
  • members_same_line (Optional[bool]) -- Place multiple structure, union, and class members on the same line if they fit within the specified number of columns. Defaults to False.
  • elements_same_line (Optional[bool]) -- Place multiple array elements on the same line if they fit within the specified number of columns. Defaults to True.
  • member_names (Optional[bool]) -- Include the names of structure, union, and class members. Defaults to True.
  • element_indices (Optional[bool]) -- Include the indices of array elements. Defaults to False.
  • implicit_members (Optional[bool]) -- Include structure, union, and class members which have an implicit value (i.e., for C, zero-initialized). Defaults to True.
  • implicit_elements (Optional[bool]) -- Include array elements which have an implicit value (i.e., for C, zero-initialized). Defaults to False.

str



Get an object representing NULL casted to the given type.

This is equivalent to Object(prog, type, 0).

  • prog (Program) -- The program.
  • type (Union[str, Type]) -- The type.

Object


Get the value of the given object casted to another type.

Objects with a scalar type (integer, boolean, enumerated, floating-point, or pointer) can be casted to a different scalar type. Other objects can only be casted to the same type. This always results in a value object. See also drgn.reinterpret().

  • type (Union[str, Type]) -- The type to cast to.
  • obj (Object) -- The object to cast.

Object


Get a copy of the given object reinterpreted as another type and/or byte order.

This reinterprets the raw memory of the object, so an object can be reinterpreted as any other type. However, value objects with a scalar type cannot be reinterpreted, as their memory layout in the program is not known. Reinterpreting a reference results in a reference, and reinterpreting a value results in a value. See also drgn.cast().

  • type (Union[str, Type]) -- The type to reinterpret as.
  • obj (Object) -- The object to reinterpret.

Object


Get the containing object of a pointer object.

This corresponds to the container_of() macro in C.

  • ptr (Object) -- Pointer to member in containing object.
  • type (Union[str, Type]) -- Type of containing object.
  • member (str) -- Name of member in containing object. May include one or more member references and zero or more array subscripts.

Pointer to containing object.
  • TypeError -- if ptr is not a pointer or type is not a structure, union, or class type
  • ValueError -- if the member is not byte-aligned (e.g., because it is a bit field)
  • LookupError -- if type does not have a member with the given name

Object


Symbols

A Symbol represents an entry in the symbol table of a program, i.e., an identifier along with its corresponding address range in the program.
Name of this symbol.
str


Start address of this symbol.
int


Size of this symbol in bytes.
int


Linkage behavior and visibility of this symbol.
SymbolBinding


Kind of entity represented by this symbol.
SymbolKind



Bases: enum.Enum

A SymbolBinding describes the linkage behavior and visibility of a symbol.

Unknown.

Not visible outside of the object file containing its definition.

Globally visible.

Globally visible but may be overridden by a non-weak global symbol.

Globally visible even if dynamic shared object is loaded locally. See GCC's -fno-gnu-unique option.


Bases: enum.Enum

A SymbolKind describes the kind of entity that a symbol represents.

Unknown or not defined.

Data object (e.g., variable or array).

Function or other executable code.

Object file section.

Source file.

Data object in common block.

Thread-local storage entity.

Indirect function.


Stack Traces

Stack traces are retrieved with Program.stack_trace().

A StackTrace is a sequence of StackFrame.

len(trace) is the number of stack frames in the trace. trace[0] is the innermost stack frame, trace[1] is its caller, and trace[len(trace) - 1] is the outermost frame. Negative indexing also works: trace[-1] is the outermost frame and trace[-len(trace)] is the innermost frame. It is also iterable:

for frame in trace:

if frame.name == 'io_schedule':
print('Thread is doing I/O')


str() returns a pretty-printed stack trace:

>>> prog.stack_trace(1)
#0  context_switch (kernel/sched/core.c:4339:2)
#1  __schedule (kernel/sched/core.c:5147:8)
#2  schedule (kernel/sched/core.c:5226:3)
#3  do_wait (kernel/exit.c:1534:4)
#4  kernel_wait4 (kernel/exit.c:1678:8)
#5  __do_sys_wait4 (kernel/exit.c:1706:13)
#6  do_syscall_64 (arch/x86/entry/common.c:47:14)
#7  entry_SYSCALL_64+0x7c/0x15b (arch/x86/entry/entry_64.S:112)
#8  0x4d49dd

The format is subject to change. The drgn CLI is set up so that stack traces are displayed with str() by default.


A StackFrame represents a single frame in a thread's call stack.

str() returns a pretty-printed stack frame:

>>> prog.stack_trace(1)[0]
#0 at 0xffffffffb64ac287 (__schedule+0x227/0x606) in context_switch at kernel/sched/core.c:4339:2 (inlined)
    

This includes more information than when printing the full stack trace. The format is subject to change. The drgn CLI is set up so that stack frames are displayed with str() by default.

The [] operator can look up function parameters, local variables, and global variables in the scope of the stack frame:

>>> prog.stack_trace(1)[0]['prev'].pid
(pid_t)1
>>> prog.stack_trace(1)[0]['scheduler_running']
(int)1
    
Name of the function at this frame, or None if it could not be determined.

The name cannot be determined if debugging information is not available for the function, e.g., because it is implemented in assembly. It may be desirable to use the symbol name or program counter as a fallback:

name = frame.name
if name is None:

try:
name = frame.symbol().name
except LookupError:
name = hex(frame.pc)


Optional[str]


Whether this frame is for an inlined call.

An inline frame shares the same stack frame in memory as its caller. Therefore, it has the same registers (including program counter and thus symbol).

bool


Whether this stack frame was interrupted (for example, by a hardware interrupt, signal, trap, etc.).

If this is True, then the register values in this frame are the values at the time that the frame was interrupted.

This is False if the frame is for a function call, in which case the register values are the values when control returns to this frame. In particular, the program counter is the return address, which is typically the instruction after the call instruction.

bool


Program counter at this stack frame.
int


__getitem__(name)
Implement self[name]. Get the object (variable, function parameter, constant, or function) with the given name in the scope of this frame.

If the object exists but has been optimized out, this returns an absent object.

name (str) -- Object name.
Object


__contains__(name)
Implement name in self. Return whether an object with the given name exists in the scope of this frame.
name (str) -- Object name.
bool


Get the source code location of this frame.
Location as a (filename, line, column) triple.
LookupError -- if the source code location is not available
Tuple[str, int, int]


Get the function symbol at this stack frame.

This is equivalent to:

prog.symbol(frame.pc - (0 if frame.interrupted else 1))


Symbol


Get the value of the given register at this stack frame.
reg (str) -- Register name.
  • ValueError -- if the register name is not recognized
  • LookupError -- if the register value is not known

int


Get the values of all available registers at this stack frame as a dictionary with the register names as keys.
Dict[str, int]



Types

A Type object describes a type in a program. Each kind of type (e.g., integer, structure) has different attributes (e.g., name, size). Types can also have qualifiers (e.g., constant, atomic). Accessing an attribute which does not apply to a type raises an AttributeError.

repr() of a Type returns a Python representation of the type:

>>> print(repr(prog.type('sector_t')))
prog.typedef_type(name='sector_t', type=prog.int_type(name='unsigned long', size=8, is_signed=False))
    

str() returns a representation of the type in programming language syntax:

>>> print(prog.type('sector_t'))
typedef unsigned long sector_t
    

The drgn CLI is set up so that types are displayed with str() instead of repr() by default.

This class cannot be constructed directly. Instead, use one of the Type Constructors.

Program that this type is from.
Program


Kind of this type.
TypeKind


If this is a primitive type (e.g., int or double), the kind of primitive type. Otherwise, None.
Optional[PrimitiveType]


Bitmask of this type's qualifier.
Qualifiers


Programming language of this type.
Language


Name of this type. This is present for integer, boolean, floating-point, and typedef types.
str


Tag of this type, or None if this is an anonymous type. This is present for structure, union, class, and enumerated types.
Optional[str]


Size of this type in bytes, or None if this is an incomplete type. This is present for integer, boolean, floating-point, structure, union, class, and pointer types.
Optional[int]


Number of elements in this type, or None if this is an incomplete type. This is only present for array types.
Optional[int]


Whether this type is signed. This is only present for integer types.
bool


Byte order of this type: 'little' if it is little-endian, or 'big' if it is big-endian. This is present for integer, boolean, floating-point, and pointer types.
str


Type underlying this type, defined as follows:
  • For typedef types, the aliased type.
  • For enumerated types, the compatible integer type, which is None if this is an incomplete type.
  • For pointer types, the referenced type.
  • For array types, the element type.
  • For function types, the return type.

For other types, this attribute is not present.

Type


List of members of this type, or None if this is an incomplete type. This is present for structure, union, and class types.
Optional[Sequence[TypeMember]]


List of enumeration constants of this type, or None if this is an incomplete type. This is only present for enumerated types.
Optional[Sequence[TypeEnumerator]]


List of parameters of this type. This is only present for function types.
Sequence[TypeParameter]


Whether this type takes a variable number of arguments. This is only present for function types.
bool


List of template parameters of this type. This is present for structure, union, class, and function types.
Sequence[TypeTemplateParameter]


Get a descriptive full name of this type.


Get whether this type is complete (i.e., the type definition is known). This is always False for void types. It may be False for structure, union, class, enumerated, and array types, as well as typedef types where the underlying type is one of those. Otherwise, it is always True.


Get a copy of this type with different qualifiers.

Note that the original qualifiers are replaced, not added to.

qualifiers (Qualifiers) -- New type qualifiers.
Type


Get a copy of this type with no qualifiers.


Look up a member in this type by name.

If this type has any unnamed members, this also matches members of those unnamed members, recursively. If the member is found in an unnamed member, TypeMember.bit_offset and TypeMember.offset are adjusted accordingly.

name (str) -- Name of the member.
  • TypeError -- if this type is not a structure, union, or class type
  • LookupError -- if this type does not have a member with the given name

TypeMember


Return whether this type has a member with the given name.

If this type has any unnamed members, this also matches members of those unnamed members, recursively.

name (str) -- Name of the member.
TypeError -- if this type is not a structure, union, or class type
bool



A TypeMember represents a member of a structure, union, or class type.
Create a TypeMember.
object_or_type (Union[Object, Type, Callable[[], Union[Object, Type]]]) --

One of:

1.
TypeMember.object as an Object.
2.
TypeMember.type as a Type. In this case, object is set to an absent object with that type.
3.
A callable that takes no arguments and returns one of the above. It is called when object or type is first accessed, and the result is cached.

  • name (Optional[str]) -- TypeMember.name
  • bit_offset (int) -- TypeMember.bit_offset



Member as an Object.

This is the default initializer for the member, or an absent object if the member has no default initializer. (However, the DWARF specification as of version 5 does not actually support default member initializers, so this is usually absent.)

Object


Member type.

This is a shortcut for TypeMember.object.type.

Type


Member name, or None if the member is unnamed.
Optional[str]


Offset of the member from the beginning of the type in bits.
int


Offset of the member from the beginning of the type in bytes. If the offset is not byte-aligned, accessing this attribute raises ValueError.
int


Size in bits of this member if it is a bit field, None if it is not.

This is a shortcut for TypeMember.object.bit_field_size_.

Optional[int]



A TypeEnumerator represents a constant in an enumerated type.

Its name and value may be accessed as attributes or unpacked:

>>> prog.type('enum pid_type').enumerators[0].name
'PIDTYPE_PID'
>>> name, value = prog.type('enum pid_type').enumerators[0]
>>> value
0
    
Create a TypeEnumerator.
  • name (str) -- TypeEnumerator.name
  • value (int) -- TypeEnumerator.value



Enumerator name.
str


Enumerator value.
int



A TypeParameter represents a parameter of a function type.
Create a TypeParameter.
default_argument_or_type (Union[Object, Type, Callable[[], Union[Object, Type]]]) --

One of:

1.
TypeParameter.default_argument as an Object.
2.
TypeParameter.type as a Type. In this case, default_argument is set to an absent object with that type.
3.
A callable that takes no arguments and returns one of the above. It is called when default_argument or type is first accessed, and the result is cached.

name (Optional[str]) -- TypeParameter.name



Default argument for parameter.

If the parameter does not have a default argument, then this is an absent object.

NOTE:

Neither GCC nor Clang emits debugging information for default arguments (as of GCC 10 and Clang 11), and drgn does not yet parse it, so this is usually absent.


Object


Parameter type.

This is the same as TypeParameter.default_argument.type_.

Type


Parameter name, or None if the parameter is unnamed.
Optional[str]



A TypeTemplateParameter represents a template parameter of a structure, union, class, or function type.
Create a TypeTemplateParameter.
argument (Union[Type, Object, Callable[[], Union[Type, Object]]]) --

One of:

1.
TypeTemplateParameter.argument as a Type if the parameter is a type template parameter.
2.
TypeTemplateParameter.argument as a non-absent Object if the parameter is a non-type template parameter.
3.
A callable that takes no arguments and returns one of the above. It is called when argument is first accessed, and the result is cached.

  • name (Optional[str]) -- TypeTemplateParameter.name
  • is_default (bool) -- TypeTemplateParameter.is_default



Template argument.

If this is a type template parameter, then this is a Type. If this is a non-type template parameter, then this is an Object.

Union[Type, Object]


Template parameter name, or None if the parameter is unnamed.
Optional[str]


Whether argument is the default for the template parameter.

NOTE:

There are two ways to interpret this:
1.
The argument was omitted entirely and thus defaulted to the default argument.
2.
The (specified or defaulted) argument is the same as the default argument.



Compilers are inconsistent about which interpretation they use.

GCC added this information in version 4.9. Clang added it in version 11 (and only when emitting DWARF version 5). If the program was compiled by an older version, this is always false.



bool



Bases: enum.Enum

A TypeKind represents a kind of type.

Void type.

Integer type.

Boolean type.

Floating-point type.

Complex type.

Structure type.

Union type.

Class type.

Enumerated type.

Type definition (a.k.a. alias) type.

Pointer type.

Array type.

Function type.



Bases: enum.Flag

Qualifiers are modifiers on types.

No qualifiers.

Constant type.

Volatile type.

Restrict type.

Atomic type.


Get the offset (in bytes) of a member in a Type.

This corresponds to offsetof() in C.

  • type (Type) -- Structure, union, or class type.
  • member (str) -- Name of member. May include one or more member references and zero or more array subscripts.

  • TypeError -- if type is not a structure, union, or class type
  • ValueError -- if the member is not byte-aligned (e.g., because it is a bit field)
  • LookupError -- if type does not have a member with the given name

int


Type Constructors

Custom drgn types can be created with the following factory functions. These can be used just like types obtained from Program.type().

Create a new void type. It has kind TypeKind.VOID.
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Create a new integer type. It has kind TypeKind.INT.
  • name (str) -- Type.name
  • size (IntegerLike) -- Type.size
  • is_signed (bool) -- Type.is_signed
  • byteorder (Optional[str]) -- Type.byteorder, or None to use the program's default byte order.
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Create a new boolean type. It has kind TypeKind.BOOL.
  • name (str) -- Type.name
  • size (IntegerLike) -- Type.size
  • byteorder (Optional[str]) -- Type.byteorder, or None to use the program's default byte order.
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Create a new floating-point type. It has kind TypeKind.FLOAT.
  • name (str) -- Type.name
  • size (IntegerLike) -- Type.size
  • byteorder (Optional[str]) -- Type.byteorder, or None to use the program's default byte order.
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Create a new structure type. It has kind TypeKind.STRUCT.
  • tag (Optional[str]) -- Type.tag
  • size (IntegerLike) -- Type.size
  • members (Sequence[TypeMember]) -- Type.members
  • template_parameters (Sequence[TypeTemplateParameter]) -- Type.template_parameters
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type



Create a new union type. It has kind TypeKind.UNION. Otherwise, this is the same as as struct_type().



Create a new class type. It has kind TypeKind.CLASS. Otherwise, this is the same as as struct_type().



Create a new enumerated type. It has kind TypeKind.ENUM.
  • tag (Optional[str]) -- Type.tag
  • type (Type) -- The compatible integer type (Type.type)
  • enumerators (Sequence[TypeEnumerator]) -- Type.enumerators
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type



Create a new typedef type. It has kind TypeKind.TYPEDEF.
  • name (str) -- Type.name
  • type (Type) -- The aliased type (Type.type)
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Create a new pointer type. It has kind TypeKind.POINTER,
  • type (Type) -- The referenced type (Type.type)
  • size (Optional[int]) -- Type.size, or None to use the program's default pointer size.
  • byteorder (Optional[str]) -- Type.byteorder, or None to use the program's default byte order.
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Create a new array type. It has kind TypeKind.ARRAY.
  • type (Type) -- The element type (Type.type)
  • length (Optional[int]) -- Type.length
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Create a new function type. It has kind TypeKind.FUNCTION.
  • type (Type) -- The return type (Type.type)
  • parameters (Sequence[TypeParameter]) -- Type.parameters
  • is_variadic (bool) -- Type.is_variadic
  • template_parameters (Sequence[TypeTemplateParameter]) -- Type.template_parameters
  • qualifiers (Qualifiers) -- Type.qualifiers
  • lang -- Type.language

Type


Miscellaneous

Get the size of a Type or Object in bytes.
type_or_obj (Union[Type, Object]) -- Entity to get the size of.
TypeError -- if the type does not have a size (e.g., because it is incomplete or void)
int


Execute a script.

The script is executed in the same context as the caller: currently defined globals are available to the script, and globals defined by the script are added back to the calling context.

This is most useful for executing scripts from interactive mode. For example, you could have a script named exe.py:

"""Get all tasks executing a given file."""
import sys
from drgn.helpers.linux.fs import d_path
from drgn.helpers.linux.pid import find_task
def task_exe_path(task):

if task.mm:
return d_path(task.mm.exe_file.f_path).decode()
else:
return None tasks = [
task for task in for_each_task(prog)
if task_exe_path(task) == sys.argv[1] ]


Then, you could execute it and use the defined variables and functions:

>>> execscript('exe.py', '/usr/bin/bash')
>>> tasks[0].pid
(pid_t)358442
>>> task_exe_path(find_task(prog, 357954))
'/usr/bin/vim'
  • path (str) -- File path of the script.
  • args (str) -- Zero or more additional arguments to pass to the script. This is a variable argument list.

None


Bases: Protocol

An int or integer-like object.

Parameters annotated with this type expect an integer which may be given as a Python int or an Object with integer type.


Filesystem path.

Parameters annotated with this type accept a filesystem path as str, bytes, or os.PathLike.


Exceptions

Bases: Exception

This error is raised when a bad memory access is attempted (i.e., when accessing a memory address which is not valid in a program).

  • message (str) -- FaultError.message
  • address (int) -- FaultError.address



Error message.
str


Address that couldn't be accessed.
int



Bases: Exception

This error is raised when one or more files in a program do not have debug information.


Bases: Exception

This error is raised when attempting to use an absent object.


Bases: Exception

This error is raised when attempting to access beyond the bounds of a value object.


Helpers

The drgn.helpers package contains subpackages which provide helpers for working with particular types of programs. Currently, there are only helpers for the Linux kernel. In the future, there may be helpers for, e.g., glibc and libstdc++.

Generic Helpers

The top-level drgn.helpers module provides generic helpers that may be useful for scripts or for implementing other helpers.

Bases: Exception

Error raised by a validator when an inconsistent or invalid state is detected.


Format an ASCII byte value as a character, possibly escaping it. Non-printable characters are always escaped. Non-printable characters other than \0, \a, \b, \t, \n, \v, \f, and \r are escaped in hexadecimal format (e.g., \x7f). By default, printable characters are never escaped.
  • c (int) -- The character to escape.
  • escape_single_quote (bool) -- Whether to escape single quotes to \'.
  • escape_double_quote (bool) -- Whether to escape double quotes to \".
  • escape_backslash (bool) -- Whether to escape backslashes to \\.

str


Escape an iterable of ASCII byte values (e.g., bytes or bytearray). See escape_ascii_character().
buffer (Iterable[int]) -- The byte array.
str


Get an enum.IntEnum class from an enumerated drgn.Type.
  • type (drgn.Type) -- The enumerated type to convert.
  • name (str) -- The name of the IntEnum type to create.
  • exclude (Container[str]) -- Container (e.g., list or set) of enumerator names to exclude from the created IntEnum.
  • prefix (str) -- Prefix to strip from the beginning of enumerator names.

Type[enum.IntEnum]


Get a human-readable representation of a bitmask of flags.

By default, flags are specified by their bit number:

>>> decode_flags(2, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)])
'ITALIC'
    

They can also be specified by their value:

>>> decode_flags(2, [("BOLD", 1), ("ITALIC", 2), ("UNDERLINE", 4)],
...              bit_numbers=False)
'ITALIC'
    

Multiple flags are combined with "|":

>>> decode_flags(5, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)])
'BOLD|UNDERLINE'
    

If there are multiple names for the same bit, they are all included:

>>> decode_flags(2, [("SMALL", 0), ("BIG", 1), ("LARGE", 1)])
'BIG|LARGE'
    

If there are any unknown bits, their raw value is included:

>>> decode_flags(27, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)])
'BOLD|ITALIC|0x18'
    

Zero is returned verbatim:

>>> decode_flags(0, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)])
'0'
    
  • value (drgn.IntegerLike) -- Bitmask to decode.
  • flags (Iterable[Tuple[str, int]]) -- List of flag names and their bit numbers or values.
  • bit_numbers (bool) -- Whether flags specifies the bit numbers (where 0 is the least significant bit) or values of the flags.

str


Get a human-readable representation of a bitmask of flags where the flags are specified by an enumerated drgn.Type.

This supports enums where the values are bit numbers:

>>> print(bits_enum)
enum style_bits {

BOLD = 0,
ITALIC = 1,
UNDERLINE = 2, } >>> decode_enum_type_flags(5, bits_enum) 'BOLD|UNDERLINE'

Or the values of the flags:

>>> print(flags_enum)
enum style_flags {

BOLD = 1,
ITALIC = 2,
UNDERLINE = 4, } >>> decode_enum_type_flags(5, flags_enum, bit_numbers=False) 'BOLD|UNDERLINE'

See decode_flags().

  • value (drgn.IntegerLike) -- Bitmask to decode.
  • type (drgn.Type) -- Enumerated type with bit numbers for enumerators.
  • bit_numbers (bool) -- Whether the enumerator values specify the bit numbers or values of the flags.

str


Linux Kernel

The drgn.helpers.linux package contains several modules for working with data structures and subsystems in the Linux kernel. The helpers are available from the individual modules in which they are defined and from this top-level package. E.g., the following are both valid:

>>> from drgn.helpers.linux.list import list_for_each_entry
>>> from drgn.helpers.linux import list_for_each_entry

Iterator macros (for_each_foo) are a common idiom in the Linux kernel. The equivalent drgn helpers are implemented as Python generators. For example, the following code in C:

list_for_each(pos, head)

do_something_with(pos);


Translates to the following code in Python:

for pos in list_for_each(head):

do_something_with(pos)


Bit Operations

The drgn.helpers.linux.bitops module provides helpers for common bit operations in the Linux kernel.

Iterate over all set (one) bits in a bitmap.
  • bitmap (drgn.Object) -- unsigned long *
  • size (drgn.IntegerLike) -- Size of bitmap in bits.

Iterator[int]


Iterate over all clear (zero) bits in a bitmap.
  • bitmap (drgn.Object) -- unsigned long *
  • size (drgn.IntegerLike) -- Size of bitmap in bits.

Iterator[int]


Return whether a bit in a bitmap is set.
  • nr (drgn.IntegerLike) -- Bit number.
  • bitmap (drgn.Object) -- unsigned long *

bool


Block Layer

The drgn.helpers.linux.block module provides helpers for working with the Linux block layer, including disks (struct gendisk) and partitions.

Since Linux v5.11, partitions are represented by struct block_device. Before that, they were represented by struct hd_struct.

Get a disk's device number.
disk (drgn.Object) -- struct gendisk *
dev_t
drgn.Object


Get the name of a disk (e.g., sda).
disk (drgn.Object) -- struct gendisk *
bytes


Iterate over all disks in the system.
Iterator of struct gendisk * objects.
Iterator[drgn.Object]


Print all of the disks in the system.


Get a partition's device number.
part (drgn.Object) -- struct block_device * or struct hd_struct * depending on the kernel version.
dev_t
drgn.Object


Get the name of a partition (e.g., sda1).
part (drgn.Object) -- struct block_device * or struct hd_struct * depending on the kernel version.
bytes


Iterate over all partitions in the system.
Iterator of struct block_device * or struct hd_struct * objects depending on the kernel version.
Iterator[drgn.Object]


Print all of the partitions in the system.


Boot

The drgn.helpers.linux.boot module provides helpers for inspecting the Linux kernel boot configuration.

Get the kernel address space layout randomization offset (zero if it is disabled).


Return whether 5-level paging is enabled.


BPF

The drgn.helpers.linux.bpf module provides helpers for working with BPF interface in include/linux/bpf.h, include/linux/bpf-cgroup.h, etc.

Iterate over all BTF objects.

This is only supported since Linux v4.18.

Iterator of struct btf * objects.
Iterator[drgn.Object]


Iterate over all BPF links.

This is only supported since Linux v5.8.

Iterator of struct bpf_link * objects.
Iterator[drgn.Object]


Iterate over all BPF maps.

This is only supported since Linux v4.13.

Iterator of struct bpf_map * objects.
Iterator[drgn.Object]


Iterate over all BPF programs.

This is only supported since Linux v4.13.

Iterator of struct bpf_prog * objects.
Iterator[drgn.Object]


Iterate over all cgroup BPF programs of the given attach type attached to the given cgroup.
  • cgrp (drgn.Object) -- struct cgroup *
  • bpf_attach_type (drgn.IntegerLike) -- enum bpf_attach_type

Iterator of struct bpf_prog * objects.
Iterator[drgn.Object]


Iterate over all effective cgroup BPF programs of the given attach type for the given cgroup.
  • cgrp (drgn.Object) -- struct cgroup *
  • bpf_attach_type (drgn.IntegerLike) -- enum bpf_attach_type

Iterator of struct bpf_prog * objects.
Iterator[drgn.Object]


Cgroup

The drgn.helpers.linux.cgroup module provides helpers for working with the cgroup interface in include/linux/cgroup.h. Only cgroup v2 is supported.

Get the cgroup for a socket from the given struct sock_cgroup_data * (usually from struct sock::sk_cgrp_data).
skcd (drgn.Object) -- struct sock_cgroup_data *
struct cgroup *
drgn.Object


Return the parent cgroup of the given cgroup if it exists, NULL otherwise.
cgrp (drgn.Object) -- struct cgroup *
struct cgroup *
drgn.Object


Get the name of the given cgroup.
cgrp (drgn.Object) -- struct cgroup *
bytes


Get the full path of the given cgroup.
cgrp (drgn.Object) -- struct cgroup *
bytes


Look up a cgroup from its default hierarchy path .
path (drgn.Path) -- Path name.
drgn.Object


Get the next child (or NULL if there is none) of the given parent starting from the given position (NULL to initiate traversal).
  • pos (drgn.Object) -- struct cgroup_subsys_state *
  • parent (drgn.Object) -- struct cgroup_subsys_state *

struct cgroup_subsys_state *
drgn.Object


Get the next pre-order descendant (or NULL if there is none) of the given css root starting from the given position (NULL to initiate traversal).
  • pos (drgn.Object) -- struct cgroup_subsys_state *
  • root (drgn.Object) -- struct cgroup_subsys_state *

struct cgroup_subsys_state *
drgn.Object


Iterate through children of the given css.
css (drgn.Object) -- struct cgroup_subsys_state *
Iterator of struct cgroup_subsys_state * objects.
Iterator[drgn.Object]


Iterate through the given css's descendants in pre-order.
css (drgn.Object) -- struct cgroup_subsys_state *
Iterator of struct cgroup_subsys_state * objects.
Iterator[drgn.Object]


CPU Masks

The drgn.helpers.linux.cpumask module provides helpers for working with CPU masks from include/linux/cpumask.h.

Iterate over all of the CPUs in the given mask.
mask (drgn.Object) -- struct cpumask
Iterator[int]


Iterate over all online CPUs.
Iterator[int]


Iterate over all possible CPUs.
Iterator[int]


Iterate over all present CPUs.
Iterator[int]


Devices

The drgn.helpers.linux.device module provides helpers for working with Linux devices, including the kernel encoding of dev_t.

Return the major ID of a kernel dev_t.
dev (drgn.IntegerLike) -- dev_t object or :class:int.
int


Return the minor ID of a kernel dev_t.
dev (drgn.IntegerLike) -- dev_t object or :class:int.
int


Return a kernel dev_t from the major and minor IDs.
  • major (drgn.IntegerLike) -- Device major ID.
  • minor (drgn.IntegerLike) -- Device minor ID.

int


Virtual Filesystem Layer

The drgn.helpers.linux.fs module provides helpers for working with the Linux virtual filesystem (VFS) layer, including mounts, dentries, and inodes.

Look up the given path name.
  • prog_or_root -- struct path * object to use as root directory, or Program to use the initial root filesystem.
  • path -- Path to lookup.
  • allow_negative -- Whether to allow returning a negative dentry (i.e., a dentry for a non-existent path).

struct path
Exception -- if the dentry is negative and allow_negative is False, or if the path is not present in the dcache. The latter does not necessarily mean that the path does not exist; it may be uncached. On a live system, you can make the kernel cache the path by accessing it (e.g., with open() or os.stat()):

>>> path_lookup(prog, '/usr/include/stdlib.h')
...
Exception: could not find '/usr/include/stdlib.h' in dcache
>>> open('/usr/include/stdlib.h').close()
>>> path_lookup(prog, '/usr/include/stdlib.h')
(struct path){

.mnt = (struct vfsmount *)0xffff8b70413cdca0,
.dentry = (struct dentry *)0xffff8b702ac2c480, }
drgn.Object


Return the full path of a dentry given a struct path.
path (drgn.Object) -- struct path or struct path *
bytes


Return the full path of a dentry given a mount and dentry.
  • vfsmnt (drgn.Object) -- struct vfsmount *
  • dentry (drgn.Object) -- struct dentry *

bytes


Return the path of a dentry from the root of its filesystem.
dentry (drgn.Object) -- struct dentry *
bytes


Return any path of an inode from the root of its filesystem.
inode (drgn.Object) -- struct inode *
Path, or None if the inode has no aliases.
Optional[bytes]


Return an iterator over all of the paths of an inode from the root of its filesystem.
inode (drgn.Object) -- struct inode *
Iterator[bytes]


Get the source device name for a mount.
mnt (drgn.Object) -- struct mount *
bytes


Get the path of a mount point.
mnt (drgn.Object) -- struct mount *
bytes


Get the filesystem type of a mount.
mnt (drgn.Object) -- struct mount *
bytes


Iterate over all of the mounts in a given namespace.
  • prog_or_ns (Union[drgn.Program, drgn.Object]) -- struct mnt_namespace * to iterate over, or Program to iterate over initial mount namespace.
  • src (Optional[drgn.Path]) -- Only include mounts with this source device name.
  • dst (Optional[drgn.Path]) -- Only include mounts with this destination path.
  • fstype (Optional[Union[str, bytes]]) -- Only include mounts with this filesystem type.

Iterator of struct mount * objects.
Iterator[drgn.Object]


Print the mount table of a given namespace. The arguments are the same as for_each_mount(). The output format is similar to /proc/mounts but prints the value of each struct mount *.


Return the kernel file descriptor of the fd of a given task.
  • task (drgn.Object) -- struct task_struct *
  • fd (drgn.IntegerLike) -- File descriptor.

struct file *
drgn.Object


Iterate over all of the files open in a given task.
task (drgn.Object) -- struct task_struct *
Iterator of (fd, struct file *) tuples.
Iterator[Tuple[int, drgn.Object]]


Print the open files of a given task.
task (drgn.Object) -- struct task_struct *
None


IDR

The drgn.helpers.linux.idr module provides helpers for working with the IDR data structure in include/linux/idr.h. An IDR provides a mapping from an ID to a pointer. This currently only supports Linux v4.11+; before this, IDRs were not based on radix trees.

Look up the entry with the given ID in an IDR.
  • idr (drgn.Object) -- struct idr *
  • id (drgn.IntegerLike) -- Entry ID.

void * found entry, or NULL if not found.
drgn.Object


Iterate over all of the entries in an IDR.
idr (drgn.Object) -- struct idr *
Iterator of (index, void *) tuples.
Iterator[Tuple[int, drgn.Object]]


Kconfig

The drgn.helpers.linux.kconfig module provides helpers for reading the Linux kernel build configuration.

Get the kernel build configuration as a mapping from the option name to the value.

>>> get_kconfig(prog)['CONFIG_SMP']
'y'
>>> get_kconfig(prog)['CONFIG_HZ']
'300'
    

This is only supported if the kernel was compiled with CONFIG_IKCONFIG. Note that most Linux distributions do not enable this option.

Mapping[str, str]


Kernfs

The drgn.helpers.linux.kernfs module provides helpers for working with the kernfs pseudo filesystem interface in include/linux/kernfs.h.

Get the name of the given kernfs node.
kn (drgn.Object) -- struct kernfs_node *
bytes


Get full path of the given kernfs node.
kn (drgn.Object) -- struct kernfs_node *
bytes


Find the kernfs node with the given path from the given parent kernfs node.
  • parent (drgn.Object) -- struct kernfs_node *
  • path (drgn.Path) -- Path name.

struct kernfs_node * (NULL if not found)
drgn.Object


Linked Lists

The drgn.helpers.linux.list module provides helpers for working with the doubly-linked list implementations (struct list_head and struct hlist_head) in include/linux/list.h.

Return whether a list is empty.
head (drgn.Object) -- struct list_head *
bool


Return whether a list has only one element.
head (drgn.Object) -- struct list_head *
bool


Return the first entry in a list.

The list is assumed to be non-empty.

See also list_first_entry_or_null().

  • head (drgn.Object) -- struct list_head *
  • type (Union[str, drgn.Type]) -- Entry type.
  • member (str) -- Name of list node member in entry type.

type *
drgn.Object


Return the first entry in a list or NULL if the list is empty.

See also list_first_entry().

  • head (drgn.Object) -- struct list_head *
  • type (Union[str, drgn.Type]) -- Entry type.
  • member (str) -- Name of list node member in entry type.

type *
drgn.Object


Return the last entry in a list.

The list is assumed to be non-empty.

  • head (drgn.Object) -- struct list_head *
  • type (Union[str, drgn.Type]) -- Entry type.
  • member (str) -- Name of list node member in entry type.

type *
drgn.Object


Return the next entry in a list.
  • pos (drgn.Object) -- type*
  • member (str) -- Name of list node member in entry type.

type *
drgn.Object


Return the previous entry in a list.
  • pos (drgn.Object) -- type*
  • member (str) -- Name of list node member in entry type.

type *
drgn.Object


Iterate over all of the nodes in a list.
head (drgn.Object) -- struct list_head *
Iterator of struct list_head * objects.
Iterator[drgn.Object]


Iterate over all of the nodes in a list in reverse order.
head (drgn.Object) -- struct list_head *
Iterator of struct list_head * objects.
Iterator[drgn.Object]


Iterate over all of the entries in a list.
  • type (Union[str, drgn.Type]) -- Entry type.
  • head (drgn.Object) -- struct list_head *
  • member (str) -- Name of list node member in entry type.

Iterator of type * objects.
Iterator[drgn.Object]


Iterate over all of the entries in a list in reverse order.
  • type (Union[str, drgn.Type]) -- Entry type.
  • head (drgn.Object) -- struct list_head *
  • member (str) -- Name of list node member in entry type.

Iterator of type * objects.
Iterator[drgn.Object]


Validate that the next and prev pointers in a list are consistent.
head (drgn.Object) -- struct list_head *
ValidationError -- if the list is invalid
None


Like list_for_each(), but validates the list like validate_list() while iterating.
head (drgn.Object) -- struct list_head *
ValidationError -- if the list is invalid
Iterator[drgn.Object]


Like list_for_each_entry(), but validates the list like validate_list() while iterating.
  • type (Union[str, drgn.Type]) -- Entry type.
  • head (drgn.Object) -- struct list_head *
  • member (str) -- Name of list node member in entry type.

ValidationError -- if the list is invalid
Iterator[drgn.Object]


Return whether a hash list is empty.
head (drgn.Object) -- struct hlist_head *
bool


Iterate over all of the nodes in a hash list.
head (drgn.Object) -- struct hlist_head *
Iterator of struct hlist_node * objects.
Iterator[drgn.Object]


Iterate over all of the entries in a hash list.
  • type (Union[str, drgn.Type]) -- Entry type.
  • head (drgn.Object) -- struct hlist_head *
  • member (str) -- Name of list node member in entry type.

Iterator of type * objects.
Iterator[drgn.Object]


Nulls Lists

The drgn.helpers.linux.list_nulls module provides helpers for working with the special version of lists (struct hlist_nulls_head and struct hlist_nulls_node) in include/linux/list_nulls.h where the end of list is not a NULL pointer, but a "nulls" marker.

Return whether a a pointer is a nulls marker.
pos (drgn.Object) -- struct hlist_nulls_node *
bool


Return whether a nulls hash list is empty.
head (drgn.Object) -- struct hlist_nulls_head *
bool


Iterate over all the entries in a nulls hash list.
  • type (Union[str, drgn.Type]) -- Entry type.
  • head (drgn.Object) -- struct hlist_nulls_head *
  • member (str) -- Name of list node member in entry type.

Iterator of type * objects.
Iterator[drgn.Object]


Memory Management

The drgn.helpers.linux.mm module provides helpers for working with the Linux memory management (MM) subsystem. Only AArch64 and x86-64 are currently supported.

Iterate over all pages in the system.
Iterator of struct page * objects.
Iterator[drgn.Object]


Get a human-readable representation of the flags set on a page.

>>> decode_page_flags(page)
'PG_uptodate|PG_dirty|PG_lru|PG_reclaim|PG_swapbacked|PG_readahead|PG_savepinned|PG_isolated|PG_reported'
    
page (drgn.Object) -- struct page *
str


Get the physical address of a page frame number (PFN) given as an Object.
pfn (drgn.Object) -- unsigned long
phys_addr_t
drgn.Object


Get the physical address of a page frame number (PFN) given as a Program and an integer.
pfn (drgn.IntegerLike) -- Page frame number.
phys_addr_t
drgn.Object


Get the page frame number (PFN) of a physical address given as an Object.
addr (drgn.Object) -- phys_addr_t
unsigned long
drgn.Object


Get the page frame number (PFN) of a physical address given as a Program and an integer.
addr (int) -- Physical address.
unsigned long
drgn.Object


Get the page frame number (PFN) of a page.
page (drgn.Object) -- struct page *
unsigned long
drgn.Object


Get the physical address of a page.
page (drgn.Object) -- struct page *
phys_addr_t
drgn.Object


Get the directly mapped virtual address of a page.
page (drgn.Object) -- struct page *
void *
drgn.Object


Get the page with a page frame number (PFN) given as an Object.
pfn (drgn.Object) -- unsigned long
struct page *
drgn.Object


Get the page with a page frame number (PFN) given as a Program and an integer.
pfn (drgn.IntegerLike) -- Page frame number.
struct page *
drgn.Object


Get the directly mapped virtual address of a page frame number (PFN) given as an Object.
pfn (drgn.Object) -- unsigned long
void *
drgn.Object


Get the directly mapped virtual address of a page frame number (PFN) given as a Program and an integer.
pfn (drgn.IntegerLike) -- Page frame number.
void *
drgn.Object


Get the page containing a directly mapped physical address given as an Object.
addr (drgn.Object) -- phys_addr_t
struct page *
drgn.Object


Get the page containing a directly mapped physical address given as a Program and an integer.
addr (drgn.IntegerLike) -- Physical address.
struct page *
drgn.Object


Get the directly mapped virtual address of a physical address given as an Object.
addr (drgn.Object) -- phys_addr_t
void *
drgn.Object


Get the directly mapped virtual address of a physical address given as a Program and an integer.
addr (drgn.IntegerLike) -- Physical address.
void *
drgn.Object


Get the page containing a directly mapped virtual address given as an Object.
addr (drgn.Object) -- void *
struct page *
drgn.Object


Get the page containing a directly mapped virtual address given as a Program and an integer.
addr (drgn.IntegerLike) -- Virtual address.
struct page *
drgn.Object


Get the page frame number (PFN) of a directly mapped virtual address given as an Object.
addr (drgn.Object) -- void *
unsigned long
drgn.Object


Get the page frame number (PFN) of a directly mapped virtual address given as a Program and an integer.
addr (drgn.IntegerLike) -- Virtual address.
unsigned long
drgn.Object


Get the physical address of a directly mapped virtual address given as an Object.
addr (drgn.Object) -- void *
phys_addr_t
drgn.Object


Get the physical address of a directly mapped virtual address given as a Program and an integer.
addr (drgn.IntegerLike) -- Virtual address.
phys_addr_t
drgn.Object


Read memory from a task's virtual address space.

>>> task = find_task(prog, 1490152)
>>> access_process_vm(task, 0x7f8a62b56da0, 12)
b'hello, world'
    
  • task (drgn.Object) -- struct task_struct *
  • address (drgn.IntegerLike) -- Starting address.
  • size (drgn.IntegerLike) -- Number of bytes to read.

bytes


Read memory from a virtual address space. This is similar to access_process_vm(), but it takes a struct mm_struct * instead of a struct task_struct *.

>>> task = find_task(prog, 1490152)
>>> access_remote_vm(task.mm, 0x7f8a62b56da0, 12)
b'hello, world'
    
  • mm (drgn.Object) -- struct mm_struct *
  • address (drgn.IntegerLike) -- Starting address.
  • size (drgn.IntegerLike) -- Number of bytes to read.

bytes


Get the list of command line arguments of a task.

>>> cmdline(find_task(prog, 1495216))
[b'vim', b'drgn/helpers/linux/mm.py']
    

$ tr '\0' ' ' < /proc/1495216/cmdline
vim drgn/helpers/linux/mm.py


task (drgn.Object) -- struct task_struct *
List[bytes]


Get the list of environment variables of a task.

>>> environ(find_task(prog, 1497797))
[b'HOME=/root', b'PATH=/usr/local/sbin:/usr/local/bin:/usr/bin', b'LOGNAME=root']
    

$ tr '\0' '\n' < /proc/1497797/environ
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin
LOGNAME=root


task (drgn.Object) -- struct task_struct *
List[bytes]


Networking

The drgn.helpers.linux.net module provides helpers for working with the Linux kernel networking subsystem.

Get a socket from an inode referring to the socket.
inode (drgn.Object) -- struct inode *
struct socket *
ValueError -- If inode does not refer to a socket
drgn.Object


Get the inode of a socket.
sock (drgn.Object) -- struct socket *
struct inode *
drgn.Object


Iterate over all network namespaces in the system.
Iterator of struct net * objects.
Iterator[drgn.Object]


Get a network namespace from a network namespace NSFS inode, e.g. /proc/$PID/ns/net or /var/run/netns/$NAME.
inode (drgn.Object) -- struct inode *
struct net *
ValueError -- if inode is not a network namespace inode
drgn.Object


Get a network namespace from a task and a file descriptor referring to a network namespace NSFS inode, e.g. /proc/$PID/ns/net or /var/run/netns/$NAME.
  • task (drgn.Object) -- struct task_struct *
  • fd (drgn.IntegerLike) -- File descriptor.

struct net *
ValueError -- If fd does not refer to a network namespace inode
drgn.Object


Iterate over all TX queues for a network device.
dev (drgn.Object) -- struct net_device *
Iterator of struct netdev_queue * objects.
Iterator[drgn.Object]


Get the network device with the given interface index number.
  • prog_or_net (Union[drgn.Program, drgn.Object]) -- struct net * containing the device, or Program to use the initial network namespace.
  • ifindex (drgn.IntegerLike) -- Network interface index number.

struct net_device * (NULL if not found)
drgn.Object


Get the network device with the given interface name.
  • prog_or_net (Union[drgn.Program, drgn.Object]) -- struct net * containing the device, or Program to use the initial network namespace.
  • name (Union[str, bytes]) -- Network interface name.

struct net_device * (NULL if not found)
drgn.Object


Check whether a socket is a full socket, i.e., not a time-wait or request socket.
sk (drgn.Object) -- struct sock *
bool


Iterate over all the entries in a nulls hash list of sockets specified by struct hlist_nulls_head head.
head (drgn.Object) -- struct hlist_nulls_head *
Iterator of struct sock * objects.
Iterator[drgn.Object]


NUMA Node Masks

The drgn.helpers.linux.nodemask module provides helpers for working with NUMA node masks from include/linux/nodemask.h.

Iterate over all of the NUMA nodes in the given mask.
mask (drgn.Object) -- nodemask_t
Iterator[int]


Iterate over all NUMA nodes in the given state.
state (drgn.IntegerLike) -- enum node_states (e.g., N_NORMAL_MEMORY)
Iterator[int]


Iterate over all possible NUMA nodes.
Iterator[int]


Iterate over all online NUMA nodes.
Iterator[int]


Return whether the given NUMA node has the given state.
  • node (drgn.IntegerLike) -- NUMA node number.
  • state (drgn.Object) -- enum node_states (e.g., N_NORMAL_MEMORY)

bool


Per-CPU

The drgn.helpers.linux.percpu module provides helpers for working with per-CPU allocations from include/linux/percpu.h and per-CPU counters from include/linux/percpu_counter.h.

Return the per-CPU pointer for a given CPU.

>>> prog["init_net"].loopback_dev.pcpu_refcnt
(int *)0x2c980
>>> per_cpu_ptr(prog["init_net"].loopback_dev.pcpu_refcnt, 7)
*(int *)0xffff925e3ddec980 = 4
    
  • ptr (drgn.Object) -- Per-CPU pointer, i.e., type __percpu *. For global variables, it's usually easier to use per_cpu().
  • cpu (drgn.IntegerLike) -- CPU number.

type * object.
drgn.Object


Return the per-CPU variable for a given CPU.

>>> print(repr(prog["runqueues"]))
Object(prog, 'struct rq', address=0x278c0)
>>> per_cpu(prog["runqueues"], 6).curr.comm
(char [16])"python3"
    
  • var (drgn.Object) -- Per-CPU variable, i.e., type __percpu (not a pointer; use per_cpu_ptr() for that).
  • cpu (drgn.IntegerLike) -- CPU number.

type object.
drgn.Object


Return the sum of a per-CPU counter.
fbc (drgn.Object) -- struct percpu_counter *
int


Process IDS

The drgn.helpers.linux.pid module provides helpers for looking up process IDs and processes.

Return the struct pid * for the given PID number.
prog_or_ns (Union[drgn.Program, drgn.Object]) -- struct pid_namespace * object, or Program to use initial PID namespace.
struct pid *
drgn.Object


Return the task with the given PID.
prog_or_ns (Union[drgn.Program, drgn.Object]) -- struct pid_namespace * object, or Program to use initial PID namespace.
struct task_struct *
drgn.Object


Return the struct task_struct * containing the given struct pid * of the given type.
  • pid (drgn.Object) -- struct pid *
  • pid_type (drgn.IntegerLike) -- enum pid_type

struct task_struct *
drgn.Object


Iterate over all PIDs in a namespace.
prog_or_ns (Union[drgn.Program, drgn.Object]) -- struct pid_namespace * to iterate over, or Program to iterate over initial PID namespace.
Iterator of struct pid * objects.
Iterator[drgn.Object]


Iterate over all of the tasks visible in a namespace.
prog_or_ns (Union[drgn.Program, drgn.Object]) -- struct pid_namespace * to iterate over, or Program to iterate over initial PID namespace.
Iterator of struct task_struct * objects.
Iterator[drgn.Object]


Log Buffer

The drgn.helpers.linux.printk module provides helpers for reading the Linux kernel log buffer.

Bases: NamedTuple

Kernel log record.

Message text.
bytes


syslog(3) facility.
int


Log level.
int


Sequence number.
int


Timestamp in nanoseconds.
int


Thread ID of thread that logged this record, if available.

This is available if the message was logged from task context and if the kernel saves the printk() caller ID.

As of Linux 5.10, the kernel always saves the caller ID. From Linux 5.1 through 5.9, it is saved only if the kernel was compiled with CONFIG_PRINTK_CALLER. Before that, it is never saved.

Optional[int]


Processor ID of CPU that logged this record, if available.

This is available only if the message was logged when not in task context (e.g., in an interrupt handler) and if the kernel saves the printk() caller ID.

See caller_tid for when the kernel saves the caller ID.

Optional[int]


Whether this record is a continuation of a previous record.
bool


Additional metadata for the message.

See the /dev/kmsg documentation for an explanation of the keys and values.

Dict[bytes, bytes]



Get a list of records in the kernel log buffer.
List[PrintkRecord]


Get the contents of the kernel log buffer formatted like dmesg(1).

The format of each line is:

[   timestamp] message


If you need to format the log buffer differently, use get_printk_records() and format it yourself.

bytes


Radix Trees

The drgn.helpers.linux.radixtree module provides helpers for working with radix trees from include/linux/radix-tree.h.

Look up the entry at a given index in a radix tree.
  • root (drgn.Object) -- struct radix_tree_root *
  • index (drgn.IntegerLike) -- Entry index.

void * found entry, or NULL if not found.
drgn.Object


Iterate over all of the entries in a radix tree.
root (drgn.Object) -- struct radix_tree_root *
Iterator of (index, void *) tuples.
Iterator[Tuple[int, drgn.Object]]


Red-Black Trees

The drgn.helpers.linux.rbtree module provides helpers for working with red-black trees from include/linux/rbtree.h.

Return whether a red-black tree is empty.
node -- struct rb_root *
bool


Return whether a red-black tree node is empty, i.e., not inserted in a tree.
node (drgn.Object) -- struct rb_node *
bool


Return the parent node of a red-black tree node.
node (drgn.Object) -- struct rb_node *
struct rb_node *
drgn.Object


Return the first node (in sort order) in a red-black tree, or NULL if the tree is empty.
root (drgn.Object) -- struct rb_root *
struct rb_node *
drgn.Object


Return the last node (in sort order) in a red-black tree, or NULL if the tree is empty.
root (drgn.Object) -- struct rb_root *
struct rb_node *
drgn.Object


Return the next node (in sort order) after a red-black node, or NULL if the node is the last node in the tree or is empty.
node (drgn.Object) -- struct rb_node *
struct rb_node *
drgn.Object


Return the previous node (in sort order) before a red-black node, or NULL if the node is the first node in the tree or is empty.
node (drgn.Object) -- struct rb_node *
struct rb_node *
drgn.Object


Iterate over all of the nodes in a red-black tree, in sort order.
root (drgn.Object) -- struct rb_root *
Iterator of struct rb_node * objects.
Iterator[drgn.Object]


Iterate over all of the entries in a red-black tree in sorted order.
  • type (Union[str, drgn.Type]) -- Entry type.
  • root (drgn.Object) -- struct rb_root *
  • member (str) -- Name of struct rb_node member in entry type.

Iterator of type * objects.
Iterator[drgn.Object]


Find an entry in a red-black tree given a key and a comparator function.

Note that this function does not have an analogue in the Linux kernel source code, as tree searches are all open-coded.

  • type (Union[str, drgn.Type]) -- Entry type.
  • root (drgn.Object) -- struct rb_root *
  • member (str) -- Name of struct rb_node member in entry type.
  • key (KeyType) -- Key to find.
  • cmp (Callable[[KeyType, drgn.Object], int]) -- Callback taking key and entry that returns < 0 if the key is less than the entry, > 0 if the key is greater than the entry, and 0 if the key matches the entry.

type * found entry, or NULL if not found.
drgn.Object


Validate a red-black tree.

This checks that:

1.
The tree is a valid binary search tree ordered according to cmp.
2.
If allow_equal is False, there are no nodes that compare equal according to cmp.
3.
The rb_parent pointers are consistent.
4.
The red-black tree requirements are satisfied: the root node is black, no red node has a red child, and every path from any node to any of its descendant leaf nodes goes through the same number of black nodes.

  • type (Union[str, drgn.Type]) -- Entry type.
  • root (drgn.Object) -- struct rb_root *
  • member (str) -- Name of struct rb_node member in entry type.
  • cmp (Callable[[drgn.Object, drgn.Object], int]) -- Callback taking two type * entry objects that returns < 0 if the first entry is less than the second entry, > 0 if the first entry is greater than the second entry, and 0 if they are equal.
  • allow_equal (bool) -- Whether the tree may contain entries that compare equal to each other.

ValidationError -- if the tree is invalid
None


Like rbtree_inorder_for_each_entry(), but validates the red-black tree like validate_rbtree() while iterating.
  • type (Union[str, drgn.Type]) -- Entry type.
  • root (drgn.Object) -- struct rb_root *
  • member (str) -- Name of struct rb_node member in entry type.
  • cmp (Callable[[drgn.Object, drgn.Object], int]) -- Callback taking two type * entry objects that returns < 0 if the first entry is less than the second entry, > 0 if the first entry is greater than the second entry, and 0 if they are equal.
  • allow_equal (bool) -- Whether the tree may contain entries that compare equal to each other.

ValidationError -- if the tree is invalid
Iterator[drgn.Object]


CPU Scheduler

The drgn.helpers.linux.sched module provides helpers for working with the Linux CPU scheduler.

Return the idle thread (PID 0, a.k.a swapper) for the given CPU.

>>> idle_task(prog, 1).comm
(char [16])"swapper/1"
    
cpu (drgn.IntegerLike) -- CPU number.
struct task_struct *
drgn.Object


Get the state of the task as a character (e.g., 'R' for running). See ps(1) for a description of the process state codes.
task (drgn.Object) -- struct task_struct *
str


Slab Allocator

The drgn.helpers.linux.slab module provides helpers for working with the Linux slab allocator.

WARNING:

Beware of slab merging when using these helpers. See slab_cache_is_merged().


Return whether a slab cache has been merged with any other slab caches.

Unless configured otherwise, the kernel may merge slab caches of similar sizes together. See the SLUB users guide and slab_merge/slab_nomerge in the kernel parameters documentation.

This can cause confusion, as only the name of the first cache will be found, and objects of different types will be mixed in the same slab cache.

For example, suppose that we have two types, struct foo and struct bar, which have the same size but are otherwise unrelated. If the kernel creates a slab cache named foo for struct foo, then another slab cache named bar for struct bar, then slab cache foo will be reused instead of creating another cache for bar. So the following will fail:

find_slab_cache(prog, "bar")


And the following will also return struct bar * objects errantly casted to struct foo *:

slab_cache_for_each_allocated_object(

find_slab_cache(prog, "foo"), "struct foo" )


Unfortunately, these issues are difficult to work around generally, so one must be prepared to handle them on a case-by-case basis (e.g., by looking up the slab cache by its variable name and by checking that members of the structure make sense for the expected type).

slab_cache (drgn.Object) -- struct kmem_cache *
bool


Iterate over all slab caches.
Iterator of struct kmem_cache * objects.
Iterator[drgn.Object]


Return the slab cache with the given name.
name (Union[str, bytes]) -- Slab cache name.
struct kmem_cache *
Optional[drgn.Object]


Print the name and struct kmem_cache * value of all slab caches.


Iterate over all allocated objects in a given slab cache.

Only the SLUB and SLAB allocators are supported; SLOB does not store enough information to identify objects in a slab cache.

>>> dentry_cache = find_slab_cache(prog, "dentry")
>>> next(slab_cache_for_each_allocated_object(dentry_cache, "struct dentry"))
*(struct dentry *)0xffff905e41404000 = {

... }
  • slab_cache (drgn.Object) -- struct kmem_cache *
  • type (Union[str, drgn.Type]) -- Type of object in the slab cache.

Iterator of type * objects.
Iterator[drgn.Object]


Traffic Control (TC)

The drgn.helpers.linux.tc module provides helpers for working with the Linux kernel Traffic Control (TC) subsystem.

Get a Qdisc from a device and a major handle number. It is worth noting that conventionally handles are hexadecimal, e.g. 10: in a tc command means major handle 0x10.
  • dev (drgn.Object) -- struct net_device *
  • major (drgn.IntegerLike) -- Qdisc major handle number.

struct Qdisc * (NULL if not found)
drgn.Object


TCP

The drgn.helpers.linux.tcp module provides helpers for working with the TCP protocol in the Linux kernel.

Return the TCP protocol state of a socket.
sk (drgn.Object) -- struct sock *
TCP state enum value.
drgn.Object


Users

The drgn.helpers.linux.user module provides helpers for working with users in the Linux kernel.

Return the user structure with the given UID.
uid (Union[drgn.Object, drgn.IntegerLike]) -- kuid_t object or integer.
struct user_struct * (NULL if not found)
drgn.Object


Iterate over all users in the system.
Iterator of struct user_struct * objects.
Iterator[drgn.Object]


Case Studies

These are writeups of real-world problems solved with drgn.

Using Stack Trace Variables to Find a Kyber Bug

Author: Omar Sandoval
Date: June 9th, 2021

Jakub Kicinski reported a crash in the Kyber I/O scheduler when he was testing Linux 5.12. He captured a core dump and asked me to debug it. This is a quick writeup of that investigation.

First, we can get the task that crashed:

>>> task = per_cpu(prog["runqueues"], prog["crashing_cpu"]).curr


Then, we can get its stack trace:

>>> trace = prog.stack_trace(task)
>>> trace
#0  queued_spin_lock_slowpath (../kernel/locking/qspinlock.c:471:3)
#1  queued_spin_lock (../include/asm-generic/qspinlock.h:85:2)
#2  do_raw_spin_lock (../kernel/locking/spinlock_debug.c:113:2)
#3  spin_lock (../include/linux/spinlock.h:354:2)
#4  kyber_bio_merge (../block/kyber-iosched.c:573:2)
#5  blk_mq_sched_bio_merge (../block/blk-mq-sched.h:37:9)
#6  blk_mq_submit_bio (../block/blk-mq.c:2182:6)
#7  __submit_bio_noacct_mq (../block/blk-core.c:1015:9)
#8  submit_bio_noacct (../block/blk-core.c:1048:10)
#9  submit_bio (../block/blk-core.c:1125:9)
#10 submit_stripe_bio (../fs/btrfs/volumes.c:6553:2)
#11 btrfs_map_bio (../fs/btrfs/volumes.c:6642:3)
#12 btrfs_submit_data_bio (../fs/btrfs/inode.c:2440:8)
#13 submit_one_bio (../fs/btrfs/extent_io.c:175:9)
#14 submit_extent_page (../fs/btrfs/extent_io.c:3229:10)
#15 __extent_writepage_io (../fs/btrfs/extent_io.c:3793:9)
#16 __extent_writepage (../fs/btrfs/extent_io.c:3872:8)
#17 extent_write_cache_pages (../fs/btrfs/extent_io.c:4514:10)
#18 extent_writepages (../fs/btrfs/extent_io.c:4635:8)
#19 do_writepages (../mm/page-writeback.c:2352:10)
#20 __writeback_single_inode (../fs/fs-writeback.c:1467:8)
#21 writeback_sb_inodes (../fs/fs-writeback.c:1732:3)
#22 __writeback_inodes_wb (../fs/fs-writeback.c:1801:12)
#23 wb_writeback (../fs/fs-writeback.c:1907:15)
#24 wb_check_background_flush (../fs/fs-writeback.c:1975:10)
#25 wb_do_writeback (../fs/fs-writeback.c:2063:11)
#26 wb_workfn (../fs/fs-writeback.c:2091:20)
#27 process_one_work (../kernel/workqueue.c:2275:2)
#28 worker_thread (../kernel/workqueue.c:2421:4)
#29 kthread (../kernel/kthread.c:292:9)
#30 ret_from_fork+0x1f/0x2a (../arch/x86/entry/entry_64.S:294)


It looks like kyber_bio_merge() tried to lock an invalid spinlock. For reference, this is the source code of kyber_bio_merge():

static bool kyber_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio,

unsigned int nr_segs) {
struct kyber_hctx_data *khd = hctx->sched_data;
struct blk_mq_ctx *ctx = blk_mq_get_ctx(hctx->queue);
struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw[hctx->type]];
unsigned int sched_domain = kyber_sched_domain(bio->bi_opf);
struct list_head *rq_list = &kcq->rq_list[sched_domain];
bool merged;
spin_lock(&kcq->lock);
merged = blk_bio_list_merge(hctx->queue, rq_list, bio, nr_segs);
spin_unlock(&kcq->lock);
return merged; }


When printed, the kcq structure containing the spinlock indeed looks like garbage (omitted for brevity).

A crash course on the Linux kernel block layer: for each block device, there is a "software queue" (struct blk_mq_ctx *ctx) for each CPU and a "hardware queue" (struct blk_mq_hw_ctx *hctx) for each I/O queue provided by the device. Each hardware queue has one or more software queues assigned to it. Kyber keeps additional data per hardware queue (struct kyber_hctx_data *khd) and per software queue (struct kyber_ctx_queue *kcq).

Let's try to figure out where the bad kcq came from. It should be an element of the khd->kcqs array (khd is optimized out, but we can recover it from hctx->sched_data):

>>> trace[4]["khd"]
(struct kyber_hctx_data *)<absent>
>>> hctx = trace[4]["hctx"]
>>> khd = cast("struct kyber_hctx_data *", hctx.sched_data)
>>> trace[4]["kcq"] - khd.kcqs
(ptrdiff_t)1
>>> hctx.nr_ctx
(unsigned short)1


So the kcq is for the second software queue, but the hardware queue is only supposed to have one software queue. Let's see which CPU was assigned to the hardware queue:

>>> hctx.ctxs[0].cpu
(unsigned int)6


Here's the problem: we're not running on CPU 6, we're running on CPU 19:

>>> prog["crashing_cpu"]
(int)19


And CPU 19 is assigned to a different hardware queue that actually does have two software queues:

>>> ctx = per_cpu_ptr(hctx.queue.queue_ctx, 19)
>>> other_hctx = ctx.hctxs[hctx.type]
>>> other_hctx == hctx
False
>>> other_hctx.nr_ctx
(unsigned short)2


The bug is that the caller gets the hctx for the current CPU, then kyber_bio_merge() gets the ctx for the current CPU, and if the thread is migrated to another CPU in between, they won't match. The fix is to get a consistent view of the hctx and ctx. The commit that fixes this is here.

Getting Debugging Symbols

Most Linux distributions don't install debugging symbols for installed packages by default. This page documents how to install debugging symbols on common distributions. If drgn prints an error like:

$ sudo drgn
could not get debugging information for:
kernel (could not find vmlinux for 5.14.14-200.fc34.x86_64)
...


Then you need to install debugging symbols.

Fedora

Fedora makes it very easy to install debugging symbols with the DNF debuginfo-install plugin, which is installed by default. Simply run sudo dnf debuginfo-install $package:

$ sudo dnf debuginfo-install python3


To find out what package owns a binary, use rpm -qf:

$ rpm -qf $(which python3)
python3-3.9.7-1.fc34.x86_64


To install symbols for the running kernel:

$ sudo dnf debuginfo-install kernel-$(uname -r)


Also see the Fedora documentation.

Debian

Debian requires you to manually add the debugging symbol repositories:

$ sudo tee /etc/apt/sources.list.d/debug.list << EOF
deb http://deb.debian.org/debian-debug/ $(lsb_release -cs)-debug main
deb http://deb.debian.org/debian-debug/ $(lsb_release -cs)-proposed-updates-debug main
EOF
$ sudo apt update


Then, debugging symbol packages can be installed with sudo apt install. Some debugging symbol packages are named with a -dbg suffix:

$ sudo apt install python3-dbg


And some are named with a -dbgsym suffix:

$ sudo apt install coreutils-dbgsym


You can use the find-dbgsym-packages command from the debian-goodies package to find the correct name:

$ sudo apt install debian-goodies
$ find-dbgsym-packages $(which python3)
libc6-dbg libexpat1-dbgsym python3.9-dbg zlib1g-dbgsym
$ find-dbgsym-packages $(which cat)
coreutils-dbgsym libc6-dbg


To install symbols for the running kernel:

$ sudo apt install linux-image-$(uname -r)-dbg


Also see the Debian documentation.

Ubuntu

On Ubuntu, you must install the debugging symbol archive signing key and manually add the debugging symbol repositories:

$ sudo apt update
$ sudo apt install ubuntu-dbgsym-keyring
$ sudo tee /etc/apt/sources.list.d/debug.list << EOF
deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse
EOF
$ sudo apt update


Like Debian, some debugging symbol packages are named with a -dbg suffix and some are named with a -dbgsym suffix:

$ sudo apt install python3-dbg
$ sudo apt install coreutils-dbgsym


You can use the find-dbgsym-packages command from the debian-goodies package to find the correct name:

$ sudo apt install debian-goodies
$ find-dbgsym-packages $(which python3)
libc6-dbg libexpat1-dbgsym python3.9-dbg zlib1g-dbgsym
$ find-dbgsym-packages $(which cat)
coreutils-dbgsym libc6-dbg


To install symbols for the running kernel:

$ sudo apt install linux-image-$(uname -r)-dbgsym


Also see the Ubuntu documentation.

Arch Linux

Arch Linux unfortunately does not make debugging symbols available. Packages must be manually rebuilt with debugging symbols enabled. See the ArchWiki and the feature request.

AUTHOR

Omar Sandoval

COPYRIGHT

Omar Sandoval

October 5, 2022 0.0.20