Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Debugging & Performance

Tracking Down a Hung await with Task Stacks

An async program that hangs gives you nothing to go on. There is no exception and no traceback, often no log line — the process just stops making progress. In tests, the symptom is a CI job that runs until its global timeout and is killed, leaving a log that ends in the middle of the test list. The usual culprits are an await on something that will never complete (a future nobody resolves, a queue nobody fills, a lock never released) or a synchronous call that blocks the event loop thread so nothing else can run.

Those two cases need different tools, and telling them apart is the first step. If the event loop is still running, you can ask it where every task is waiting, and the answer points at the await that never returns. If the loop itself is blocked, no coroutine can run — including a diagnostic one — and you need a thread-level stack dump, which shows the synchronous call holding the thread.

Prerequisites

Solution

Python
# app/diagnostics.py — install once at startup (Unix).
import asyncio
import faulthandler
import signal
import sys

def install_task_dump(loop: asyncio.AbstractEventLoop) -> None:
    def dump() -> None:
        tasks = asyncio.all_tasks(loop)
        print(f"--- {len(tasks)} tasks ---", file=sys.stderr)
        for task in tasks:
            print(f"\n{task.get_name()}  {task.get_coro().__qualname__}", file=sys.stderr)
            task.print_stack(file=sys.stderr)
    loop.add_signal_handler(signal.SIGUSR1, dump)        # runs on the loop: needs it alive
    faulthandler.register(signal.SIGUSR2, all_threads=True)  # runs anywhere: works when blocked
Bash
kill -USR1 <pid>          # where is each task awaiting?
kill -USR2 <pid>          # what is each thread executing right now?
py-spy dump --pid <pid>   # the same, from outside, no code changes

# Python 3.14+: the await graph of a running process, from outside.
python -m asyncio pstree <pid>
Python
# Tests: bound every external wait, and dump stacks if something still hangs.
async def test_consumer_drains_queue(broker):
    async with asyncio.timeout(5):
        await consumer.run_until_empty()
TOML
# pyproject.toml
[tool.pytest.ini_options]
timeout = 60
timeout_method = "thread"      # dumps every thread's stack on timeout
Is the loop blocked, or is a task waiting? A decision flow starts with sending SIGUSR1 to trigger the task dump coroutine. If the dump prints, the loop is alive and the task stacks show which await is waiting on something that never completes. If nothing prints, the loop thread is blocked by synchronous code, and a thread stack dump from faulthandler or py-spy shows the blocking call. First question: does the loop still run? kill -USR1 task dump on the loop dump prints loop alive, a task waits nothing prints loop thread blocked read task stacks find the await that never returns faulthandler / py-spy dump find the synchronous call
A coroutine-based dump that never prints is itself the diagnosis: nothing on the loop can run.

Why this works

loop.add_signal_handler schedules the callback on the event loop, so it runs only when the loop gets a chance to process callbacks. That is a feature: if it runs, the loop is healthy and the hang is a task waiting on something. asyncio.all_tasks returns every unfinished task, and task.print_stack() prints the frames of the task's coroutine chain as of its current suspension point — the innermost frame is the await it is stuck on.

faulthandler.register installs a C-level signal handler that writes the Python stack of every thread directly to a file descriptor, without needing the interpreter to reach a safe point in Python code. It works even when the loop thread is stuck in a blocking socket.recv, a time.sleep, or a deadlock on a threading.Lock. py-spy dump reads the same information from outside the process by inspecting its memory, which also works on processes that never installed a handler.

Python 3.14 adds python -m asyncio ps and pstree, which use the new remote debugging interface to read the task graph of another process — which tasks exist and which task is awaiting which — without any in-process setup. It shows exactly the "who waits on whom" structure that makes deadlocks between tasks visible.

Reading a task dump

A typical task dump from a hung worker:

Plain text
--- 3 tasks ---

Task-1  main
  File "app/worker.py", line 88, in main
    await asyncio.gather(consumer(q), producer(q))

consumer  consumer
  File "app/worker.py", line 52, in consumer
    item = await q.get()

producer  producer
  File "app/worker.py", line 70, in producer
    await client.fetch_page(cursor)
  File "app/client.py", line 31, in fetch_page
    async with self._lock:

main waits on gather, which is normal. consumer waits on an empty queue, which is also normal — it is waiting for the producer. The producer is waiting to acquire self._lock. The question becomes: who holds the lock? Search the dump for any other task inside a async with self._lock block; if there is none, the lock was acquired and never released — often by a code path that returned early or raised inside a manual acquire() without finally. The dump turned "it hangs" into "the client's lock is leaked on some error path", which is directly fixable.

The await chain in the hung worker Main awaits gather of consumer and producer. The consumer awaits q.get on an empty queue, waiting for the producer. The producer awaits the client lock, which no task holds because it was leaked on an error path. The whole chain is stuck behind the leaked lock. Follow the waits to the thing nobody will release main · gather consumer await q.get() producer async with _lock _lock held by no live task A lock with no holder in the dump was leaked on an error path.
The end of the chain is the bug: a resource something is waiting for that nothing alive will ever release.

Preventing hangs with timeouts at the boundaries

Dumps explain hangs after the fact. Timeouts turn future hangs into errors with tracebacks, which is far cheaper to debug. The principle is to bound every wait on something outside the program's control, at the point where the wait happens.

Network calls are the obvious case: HTTP clients, database drivers and message brokers should all have connect and read timeouts configured, not left at "wait forever". Internal coordination is the less obvious case. A queue.get() that waits for a producer, an event.wait() for a signal from another task, a lock acquisition — each of these hangs forever if the other side has a bug. Wrapping them in async with asyncio.timeout(...) with a generous limit costs nothing in the normal case and converts a silent hang into a TimeoutError whose traceback names the exact await.

Choose limits from what the operation should take, multiplied by a safety margin, and put them in configuration rather than scattered literals. In tests, set them much tighter than production — a queue that should fill within milliseconds in a test can have a one-second limit — so that a hang fails the test quickly and precisely instead of waiting for the global pytest-timeout. The global timeout remains as the last line of defence, with timeout_method = "thread" so that even an unexpected hang produces every thread's stack in the CI log.

Layers of timeouts Three nested layers bound a test. Innermost, per-await timeouts on queues, events, locks and network calls raise TimeoutError at the exact await. Next, the pytest-timeout global limit dumps all thread stacks if something unbounded hangs. Outermost, the CI job timeout kills the job with no diagnostics and should never be the one that fires. The innermost timeout should always fire first CI job timeout — kills, no diagnostics pytest-timeout (thread) — dumps every stack asyncio.timeout at each await TimeoutError with the exact await in the traceback
Each outer layer is a fallback with worse diagnostics; tight inner limits keep hangs in the innermost box.

Hangs that only happen in CI

A hang that never reproduces locally is usually a timing difference, and the dump is still the fastest route to it. Make sure CI produces one: pytest-timeout with the thread method prints every thread's stack when a test exceeds its limit, and adding faulthandler.dump_traceback_later(timeout - 10, exit=False) in a session fixture gives a second dump shortly before, in case the first is lost. Both land in the job log.

Common CI-only causes show up clearly in those stacks. A test waiting on a service container that has not finished starting shows an await inside the client's connect call. A test that relies on a background task being scheduled before an assertion shows the task still pending, because the slower runner has not reached it yet. And a deadlock between two tasks that only interleave badly under load shows each task waiting on a lock or event owned by the other. In each case, the fix is to replace an assumption about timing with explicit synchronisation — a readiness probe, an awaited event, a consistent lock order — rather than adding a sleep.

Edge cases and failure modes

  • Windows. add_signal_handler is not available with the default Proactor loop. Trigger the dump from a debug HTTP endpoint or a watchdog thread using loop.call_soon_threadsafe.
  • Hidden inner tasks. gather and wait_for create inner tasks; they appear in all_tasks with generated names. Name your own tasks with create_task(..., name=...) so dumps are readable.
  • Awaiting a future from another loop. A future created on one loop and awaited on another never completes. The dump shows the await; check where the future was created.
  • Timeouts that swallow context. asyncio.timeout raises TimeoutError at the hung await, with that await in the traceback — keep it rather than catching and replacing it.
  • Deadlock across threads. A coroutine awaiting run_in_executor whose worker is blocked on a lock the loop thread holds needs a thread dump, not a task dump.

Frequently Asked Questions

How do I see where every asyncio task is waiting? Iterate asyncio.all_tasks() and call task.print_stack() on each, from inside the loop — for example from a signal handler registered with loop.add_signal_handler. On Python 3.14, python -m asyncio pstree <pid> shows the await tree of a running process from outside.

What if the whole event loop is stuck, not just one task? Then something is blocking the loop thread synchronously and no coroutine can run, including a debugging one. Use faulthandler.dump_traceback or py-spy dump to see the thread's stack; it will show the blocking call.

How do I stop a hung async test from blocking CI forever? Use pytest-timeout with timeout_method = thread, which dumps all thread stacks when the limit is hit, and wrap awaited operations in asyncio.timeout() so they fail with a traceback at the await that hung.

← Back to Debugging Async Code and Event Loops