Scroll to navigation

deadlock(8) System Manager's Manual deadlock(8)


deadlock - Find potential deadlocks (lock order inversions) in a running program.


deadlock [-h] [--binary BINARY] [--dump-graph DUMP_GRAPH] [--verbose] [--lock-symbols LOCK_SYMBOLS] [--unlock-symbols UNLOCK_SYMBOLS] pid


deadlock finds potential deadlocks in a running process. The program attaches uprobes on `pthread_mutex_lock` and `pthread_mutex_unlock` by default to build a mutex wait directed graph, and then looks for a cycle in this graph. This graph has the following properties:

- Nodes in the graph represent mutexes.

- Edge (A, B) exists if there exists some thread T where lock(A) was called and lock(B) was called before unlock(A) was called.

If there is a cycle in this graph, this indicates that there is a lock order inversion (potential deadlock). If the program finds a lock order inversion, the program will dump the cycle of mutexes, dump the stack traces where each mutex was acquired, and then exit.

This program can only find potential deadlocks that occur while the program is tracing the process. It cannot find deadlocks that may have occurred before the program was attached to the process.

This tool does not work for shared mutexes or recursive mutexes.

Since this uses BPF, only the root user can use this tool.


CONFIG_BPF and bcc


show this help message and exit
If set, trace the mutexes from the binary at this path. For statically-linked binaries, this argument is not required. For dynamically-linked binaries, this argument is required and should be the path of the pthread library the binary is using. Example: /lib/x86_64-linux-gnu/
If set, this will dump the mutex graph to the specified file.
Print statistics about the mutex wait graph.
Comma-separated list of lock symbols to trace. Default is pthread_mutex_lock. These symbols cannot be inlined in the binary.
Comma-separated list of unlock symbols to trace. Default is pthread_mutex_unlock. These symbols cannot be inlined in the binary.
Specifies the maximum number of threads to trace. default 65536. Note. 40 bytes per thread.
Specifies the maximum number of edge cases that can be recorded. default 65536. Note. 88 bytes per edge case.
Pid to trace


# deadlock 181
# deadlock 181 --binary /lib/x86_64-linux-gnu/, binaries that contain `:` in the path cannot be attached with uprobes. As a workaround, we can create a symlink to the binary, and provide the symlink name instead with the `--binary` option:
# deadlock 181 --binary /usr/local/bin/lockinversion
# deadlock 181 --dump-graph graph.json
# deadlock 181 --verbose
# deadlock 181 --lock-symbols custom_mutex1_lock,custom_mutex2_lock --unlock_symbols custom_mutex1_unlock,custom_mutex2_unlock


This program does not output any fields. Rather, it will keep running until it finds a potential deadlock, or the user hits Ctrl-C. If the program finds a potential deadlock, it will output the stack traces and lock order inversion in the following format and exit:

[stack trace]
[stack trace]
[stack trace]
[stack trace]
[stack trace]
[stack trace]


This traces all mutex lock and unlock events and all thread creation events on the traced process. The overhead of this can be high if the process has many threads and mutexes. You should only run this on a process where the slowdown is acceptable.


This is from bcc.

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.




Unstable - in development.


Kenny Yu

2017-02-01 USER COMMANDS