Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Async & Concurrency

Dumping Stacks on Deadlock with faulthandler

A deadlocked test does not fail; it stops. The job runs to its platform limit, gets killed, and the log ends mid-sentence with no indication of which test was running or what it was waiting for. faulthandler closes that gap for the cost of one configuration line: after a chosen number of seconds with no progress, every thread's stack is written to stderr, and the resulting dump usually contains the complete diagnosis.

Prerequisites

  • Python 3.3+; faulthandler is in the standard library and pytest enables it by default.
  • pytest >= 8.0, plus pytest-timeout for the failure half of the arrangement.
  • A suite with real threads or subprocesses — a single-threaded hang is better served by pdb.

Solution

Arm faulthandler just below the suite's own ceiling so the dump lands before anything kills the process.

TOML
# pyproject.toml
[tool.pytest.ini_options]
# Dump every thread's stack after 45 s of a single test making no progress.
faulthandler_timeout = 45
# And fail the test at 60 s, giving the dump 15 s of headroom to be written.
timeout = 60
timeout_method = "thread"
Python
# conftest.py — an on-demand dump for local reproduction
import faulthandler
import signal


def pytest_configure(config):
    # kill -USR1 <pid> now prints every thread's stack without stopping anything.
    if hasattr(signal, "SIGUSR1"):
        faulthandler.register(signal.SIGUSR1, all_threads=True, chain=True)
Plain text
Timeout (0:00:45)!
Thread 0x00007f2ab3fff700 (most recent call first):
  File "app/cache.py", line 41 in refresh
  File "app/cache.py", line 88 in get
  File "threading.py", line 975 in _bootstrap_inner

Current thread 0x00007f2ac4a1e740 (most recent call first):
  File "app/cache.py", line 62 in evict
  File "app/cache.py", line 91 in put
  File "tests/test_cache.py", line 24 in test_concurrent_eviction
Timing of the dump relative to the suite timeout A timeline of one hung test. At forty-five seconds faulthandler writes every thread's stack while the process is alive. At sixty seconds pytest-timeout fails the test. If the order were reversed, or if faulthandler were unset, the process would be killed with no output. The dump has to happen while the process is still alive 0 s test runs, then blocks 45 s · faulthandler dumps every thread's stack to stderr 60 s · timeout fails the test suite continues Without the 45 s dump, all you get at 60 s is "the test timed out" and no stacks.
Fifteen seconds of headroom is generous; the dump itself takes milliseconds, but a loaded runner can delay the writing thread.

Why this works

faulthandler.dump_traceback_later(timeout, exit=...) starts a watchdog thread that sleeps for the timeout and, if nothing cancelled it, writes the Python stack of every thread using only pre-allocated buffers and async-signal-safe calls. That constraint is why it works when nothing else does: it does not allocate, does not take the GIL, and therefore produces output even when every other thread is blocked.

pytest wires this to the faulthandler_timeout option, resetting the watchdog before each test so the countdown measures a single test rather than the whole run. pytest-timeout is doing something different — failing the test — and the two are complementary rather than redundant.

Edge cases and failure modes

  • faulthandler_timeout set above the suite timeout. The process is killed before the dump is written, which is the same as having no dump at all. It must be lower.
  • A dump with only one thread. The hang is not a deadlock between threads; look at what that thread is blocked on — a socket read, a queue.get with no timeout, a subprocess that never exits.
  • Output swallowed by capture. faulthandler writes to the real file descriptor 2, so it survives pytest's capture, but a CI system that buffers job output may still delay it. -s guarantees it is interleaved in order.
  • C-level frames missing. The dump shows Python frames only. A thread blocked inside a C extension shows the Python call that entered it, which is usually enough to identify the library.
  • exit=True when called manually. dump_traceback_later(30, exit=True) aborts the process after dumping, which is right for a standalone script and wrong inside pytest, where it kills the whole session.

Which timeout mechanism to use where

Three mechanisms can end a hung test and they produce very different evidence, so the combination matters more than any one of them.

pytest-timeout with timeout_method = "signal" raises inside the running test at the exact line, producing an ordinary pytest failure with a real traceback and letting the suite continue. It works only on the main thread of a Unix process, so a suite with worker threads or a Windows runner cannot rely on it.

timeout_method = "thread" runs a watchdog that dumps every thread's stack and then kills the process. It works everywhere and gives good evidence, but the session ends — remaining tests do not run.

faulthandler_timeout dumps without ending anything, which is why it belongs below whichever of the two above is configured: the stacks are written first, and the failure mechanism then does its job.

For most suites the right combination is faulthandler_timeout plus the thread method, because a suite that hangs is usually going to need a human anyway and the complete stacks are worth more than the remaining tests. Where the suite is long and a single hang should not cost the whole run, the signal method keeps the session alive at the cost of covering only the main thread — acceptable when the threads are library-owned and the main thread is where the test's own code blocks. Whichever pair is chosen, write the two numbers down next to each other in the configuration with a comment, because the relationship between them — dump first, fail second — is the part that breaks silently when somebody later raises one of them in isolation.

Three timeout mechanisms and what each leaves behind Three rows comparing the signal method, the thread method and faulthandler on three properties: whether the suite continues afterwards, whether all thread stacks are captured, and where each one works. The signal method continues the suite but only covers the main thread on Unix; the thread method captures everything but ends the session; faulthandler captures everything and ends nothing. Layer them; do not choose between them mechanism suite continues all thread stacks where it works timeout_method=signal yes no Unix main thread timeout_method=thread no yes everywhere faulthandler_timeout yes yes everywhere faulthandler first for the evidence, then signal or thread to end the test.
The bottom row is the only one with no drawback, which is why it should always be set — it produces evidence without deciding anything.

Reading the dump

The dump is a list of threads, each with its frames innermost-first. Three questions answer almost every case.

What is each thread blocked on? The innermost frame names it: acquire on a lock, wait on a condition, recv on a socket, join on another thread. A thread whose innermost frame is ordinary application code is not blocked — it is spinning, which is a different bug.

What does each blocked thread already hold? Walk outward until you find the with self._lock: that thread is inside. The dump does not say so directly, but the frame at that line tells you which lock the thread acquired on its way in.

Is there a cycle? Thread A holds lock 1 and waits for lock 2; thread B holds lock 2 and waits for lock 1. That is the deadlock, and the fix is a global acquisition order rather than more locks.

Plain text
Thread 0x…700 (most recent call first):
  File "threading.py", line 327 in wait          ← blocked here
  File "app/cache.py", line 41 in refresh        ← inside `with self._write_lock`
  File "app/cache.py", line 88 in get

Current thread 0x…740 (most recent call first):
  File "threading.py", line 327 in wait          ← blocked here
  File "app/cache.py", line 62 in evict          ← inside `with self._index_lock`
  File "app/cache.py", line 91 in put

Two threads, two locks, opposite order: a complete diagnosis from a dump that cost one configuration line. The repair is to establish that _index_lock is always taken before _write_lock, everywhere, which is a rule a reviewer can check by reading rather than a race anyone has to reproduce.

Dumping a process you did not start

The signal registration above pays off when the hang happens somewhere you cannot easily add configuration: a long-running local reproduction, a worker inside a container, a process a colleague is looking at.

Bash
# Find the stuck process and ask it for a dump. Nothing is killed.
pgrep -af "pytest tests/test_cache.py"
kill -USR1 12345                       # stacks appear on the process's stderr

# Inside a container, the same signal via the runtime:
docker kill --signal=SIGUSR1 my-test-container

Because faulthandler.register is armed at pytest_configure, every test session in the repository responds to the signal, including ones started by an IDE. That makes "send USR1 and read the stacks" the first move against any hang, ahead of attaching a debugger — it is faster, it is non-invasive, and it works on a process whose environment cannot be changed.

Two arguments to register matter. all_threads=True is what makes it useful; the default dumps only the current thread, which for a deadlock is the least interesting one. And chain=True preserves any handler already installed for that signal, so registering does not silently break something else that used SIGUSR1.

Three ways to obtain a stack dump, by situation Three rows. A timeout-driven dump is automatic and suits CI. A signal-driven dump is on demand and suits local or containerised reproduction. A debugger attach gives interactive inspection but requires changing how the process runs and is the slowest to set up. Reach for these in this order 1 · faulthandler_timeout automatic, no interaction, right for CI · gives stacks at a fixed delay 2 · kill -USR1 on demand, non-invasive, works in containers · nothing is stopped 3 · attach a debugger interactive but slowest to set up · needs the process started differently
The first two answer "what is everything waiting on", which is the whole question for a deadlock. The third is for when the answer to that raises a further question about values.

Frequently Asked Questions

Why does my deadlocked test produce no output at all? Because nothing asked for one. pytest's own timeout kills the process, and a CI platform's job timeout kills it harder. faulthandler_timeout set below the suite timeout writes every thread's stack while the process is still alive, which is the only window in which the dump can be produced.

Does faulthandler work with pytest-timeout? They complement each other. Set faulthandler_timeout lower than pytest-timeout's value so the stacks are written first and the timeout then fails the test cleanly. pytest-timeout's thread method also dumps stacks, so on Unix the signal method plus faulthandler often gives the most readable combination.

Can I dump stacks without waiting for a timeout? Yes. faulthandler.register(signal.SIGUSR1) lets you send a signal to a stuck process and get a dump immediately, which is invaluable when reproducing a hang locally. It also works against a process running under a debugger or inside a container.

← Back to Testing Threads & Race Conditions