Diagnosing "Task Was Destroyed but It Is Pending"

Task was destroyed but it is pending! is one of asyncio's least helpful messages. It appears at the end of a test run, or when a service shuts down, or at some random moment when the garbage collector happens to run — usually nowhere near the code that created the task. It prints the task's repr, maybe a coroutine name, and nothing else. Its sibling, Task exception was never retrieved, is worse: it means an error already happened, and nobody noticed.

Both have the same root: a task that nobody is responsible for. Either the code dropped its reference after create_task, so the task is only weakly held and can be collected mid-flight, or the program ended without cancelling and awaiting background work. The fixes are structural — hold references, shut down deliberately, or use TaskGroup so ownership is automatic — and asyncio's debug mode turns the unhelpful message into one that says where the task came from.

Prerequisites

Python 3.11 or later (for TaskGroup), pytest-asyncio >= 0.23.
Background from Debugging async code and event loops.

Solution

# Before — fire-and-forget: the only strong reference is dropped immediately.
async def handle(request):
    asyncio.create_task(send_audit_event(request))    # may be collected mid-flight
    return response

# After (1) — hold references for background work that outlives a request.
_background: set[asyncio.Task] = set()

def spawn(coro) -> asyncio.Task:
    task = asyncio.create_task(coro)
    _background.add(task)
    task.add_done_callback(_background.discard)
    task.add_done_callback(_log_failure)
    return task

def _log_failure(task: asyncio.Task) -> None:
    if not task.cancelled() and task.exception() is not None:
        log.error("background task failed", exc_info=task.exception())

async def shutdown() -> None:
    for t in list(_background):
        t.cancel()
    await asyncio.gather(*_background, return_exceptions=True)

# After (2) — structured concurrency when the work belongs to a scope.
async def process_batch(items):
    async with asyncio.TaskGroup() as tg:
        for item in items:
            tg.create_task(process(item))
    # All tasks finished, or all were cancelled and errors raised, before this line.

# Where was the leaked task created?
PYTHONASYNCIODEBUG=1 pytest tests/test_api.py -W error::pytest.PytestUnraisableExceptionWarning

The asyncio documentation says it plainly: save a reference to the result of create_task, or the task may disappear mid-execution.

Why this works

The event loop keeps its set of all tasks in a WeakSet, so the loop alone does not keep a task alive. While a task is actively scheduled — its next step queued with call_soon — the loop's ready queue holds a strong reference. While it is suspended waiting on a future, the future's callbacks hold one, as long as something holds the future. In the gaps, a task whose creator dropped the reference can have no strong references at all, and the garbage collector destroys it. Its __del__ notices it was still pending and logs the warning.

The done-callback pattern closes the gap: the module-level set holds each task strongly until it completes, then discard removes it so the set does not grow forever. TaskGroup does the same internally and adds structured semantics — the async with block does not exit until every task in it has finished, so ownership is guaranteed by the code's shape.

At shutdown, the loop closes. Any task still pending at that point is destroyed with the same warning. asyncio.run cancels remaining tasks for you, but code that manages its own loop, or background tasks that ignore cancellation, can still leave pending tasks behind. An explicit shutdown() that cancels and awaits them makes the end of the program as deliberate as the start.

Finding the leaking task in a test suite

In a large suite, the warning often appears attributed to the wrong test — whichever one happened to trigger garbage collection. Three settings make it precise.

PYTHONASYNCIODEBUG=1 (or asyncio.run(..., debug=True)) records a traceback at every create_task call. The destroyed-task warning then includes source_traceback, pointing at the line that created the task. That alone usually identifies the culprit.

-W error::pytest.PytestUnraisableExceptionWarning makes pytest fail the test during which the warning surfaced. Combined with gc.collect() in an autouse fixture's teardown, the warning surfaces at the end of the test that leaked the task, not later.

Finally, pytest-asyncio's per-test event loops close at test end, so a background task spawned by one test and never awaited is destroyed at that test's teardown — which is the right place to report it.

Together they turn an end-of-run message into a red test with a traceback pointing at the create_task line.

Choosing between a task set and a TaskGroup

Both fixes hold references; they differ in who owns the task's lifetime, and choosing the wrong one produces its own problems.

A TaskGroup ties tasks to a block of code. It is the right choice whenever the work has a natural end that the caller waits for: processing a batch, fanning out requests for one response, running a producer and consumer together until the queue drains. The block exits only when every task has finished, errors propagate as an ExceptionGroup, and a failure in one task cancels its siblings. There is nothing to shut down later, because nothing outlives the block.

A task set with done callbacks is for work that deliberately outlives its caller: an audit event sent after the response has returned, a cache refresh triggered by a request, a long-running consumer started at application startup. The request handler cannot wait for these without defeating their purpose, so ownership moves to the application, and the application must cancel and await them at shutdown. Frameworks often provide a hook for this — a lifespan handler in Starlette and FastAPI, on_cleanup in aiohttp — and the shutdown() function belongs there.

The mistake to avoid is using a set where a group fits. A request handler that spawns five tasks into a global set and returns without waiting leaves those tasks running after the response, with nobody reporting their errors to the caller. If the result matters to the response, use a group; if it does not, the set is fine, but log failures from the done callback so they are not lost.

If the response depends on the result, the task belongs in a group; if not, in a set with a shutdown hook.

Testing that shutdown actually cleans up

A shutdown routine that is never exercised tends to rot: a new background task is added without registering it, or a coroutine starts swallowing cancellation, and the warnings return. A focused test keeps it honest. Start the application in the test's event loop, trigger the code paths that spawn background work — a request that sends an audit event, a timer that schedules a refresh — then call the shutdown hook and assert that nothing is left:

async def test_shutdown_leaves_no_tasks(app):
    await app.startup()
    await app.client.post("/orders", json=ORDER)
    await app.shutdown()
    current = asyncio.current_task()
    leftover = [t for t in asyncio.all_tasks() if t is not current]
    assert leftover == [], [t.get_coro() for t in leftover]

asyncio.all_tasks() returns every task not yet finished on the running loop. After a correct shutdown, only the test's own task remains. When the assertion fails, the message lists the leftover coroutines by name, which points straight at the spawner that forgot to register its task or the coroutine that refused to cancel. Run this test with debug mode enabled and the creation tracebacks are available too. It takes seconds to run and catches the regression at the moment it is introduced, rather than weeks later as a warning in someone's deployment logs.

Edge cases and failure modes

Tasks that swallow CancelledError. A coroutine that catches CancelledError and continues cannot be shut down; gather waits forever. Re-raise it after cleanup.
asyncio.shield. A shielded inner task keeps running when the outer is cancelled and can outlive shutdown. Track and await shielded work explicitly.
Libraries spawning tasks. Some clients start background tasks (keepalives, reconnect loops) and expect close() or aclose(). Use them as async context managers.
Sync code calling asyncio.run repeatedly. Each call creates and closes a loop; tasks spawned into one and not awaited are destroyed at its close.
Never-retrieved exceptions. The done callback in the solution logs failures immediately instead of at garbage collection.

Frequently Asked Questions

What does "Task was destroyed but it is pending!" mean? An asyncio Task object was garbage collected, or the loop was closed, while the task had not finished. Either nothing held a reference to it, so it could be collected mid-flight, or the program shut down without cancelling and awaiting it.

Why does asyncio need me to keep a reference to tasks? The event loop only keeps weak references to tasks. If your code drops the object returned by create_task, the task can be garbage collected before it completes, silently stopping its work.

How is "Task exception was never retrieved" different? That task did finish, but with an exception, and nothing awaited it or called result() or exception(). The exception is reported when the task is garbage collected, often far from where it happened.

Debugging Async Code and Event Loops — asyncio debugging fundamentals.
Tracing Unawaited Coroutine Warnings — the coroutine-level sibling.
Tracking Down a Hung await with Task Stacks — tasks that never finish.
Getting Useful Tracebacks from Threads and Tasks — surfacing background exceptions.

← Back to Debugging Async Code and Event Loops