Debugging a Test That Only Fails Under xdist

A test that passes on its own and in a serial run, and fails only with pytest -n auto, is telling you it depends on something the parallel run changes. The list of such things is short. Two workers can reach for the same external resource at the same moment — a file at a fixed path, a TCP port, a database or schema name, a cache directory. A session-scoped fixture that "runs once" actually runs once per worker, so setup that assumes exclusivity collides with itself. And the tests that share a worker process are a different, partly random set from the serial run, so leaked global state from a neighbour now reaches a test it never reached before.

Each cause has a recognisable signature and a direct fix. The hard part is reproduction: the failure depends on timing and on which tests landed on which worker. Logging the worker assignment and replaying one worker's sequence serially turns most xdist-only failures into ordinary, deterministic ones.

Prerequisites

pytest >= 8.0, pytest-xdist >= 3.5, optionally filelock.
Background from Debugging tests in CI and containers.

Solution

# 1. See which worker ran what, in a stable order.
pytest -n 4 -v -p no:randomly 2>&1 | tee xdist.log
#   [gw2] PASSED tests/test_export.py::test_writes_report
#   [gw2] FAILED tests/test_export.py::test_report_is_valid_csv

# 2. Replay gw2's sequence serially.
grep '^\[gw2\]' xdist.log | awk '{print $3}' > gw2.txt
pytest -p no:randomly $(cat gw2.txt)

# 3. Or test for a resource collision: run the one test on many workers at once.
pytest -n 8 --count=16 tests/test_export.py::test_report_is_valid_csv   # pytest-repeat

# Parallel-safe replacements for the usual collisions.
import socket
import pytest
from filelock import FileLock

@pytest.fixture
def report_path(tmp_path):
    return tmp_path / "report.csv"            # not Path("/tmp/report.csv")

@pytest.fixture
def free_port():
    with socket.socket() as s:
        s.bind(("127.0.0.1", 0))
        return s.getsockname()[1]              # not a hard-coded 8080

@pytest.fixture(scope="session")
def db_name(worker_id):
    return f"test_app_{worker_id}"             # gw0, gw1… or "master" without xdist

@pytest.fixture(scope="session")
def shared_fixture_data(tmp_path_factory, worker_id):
    root = tmp_path_factory.getbasetemp().parent   # shared across workers
    marker = root / "fixture-data.ready"
    with FileLock(str(root / "fixture-data.lock")):
        if not marker.exists():
            build_expensive_fixture_data(root / "fixture-data")
            marker.touch()
    return root / "fixture-data"

The signature usually identifies the column: nondeterministic victims suggest a collision, setup errors a fixture, and reproducible replays a neighbour.

Why this works

-v output under xdist prefixes each result with the worker id, and -p no:randomly removes one source of variation so the same command produces the same assignment more often. The default --dist load scheduler hands out tests in small chunks, so assignment still depends on timing, but within one run the log records exactly which tests each worker ran and in which order. Replaying that sequence serially recreates the neighbour relationships without the parallelism. If the failure reproduces, the cause is state left by an earlier test on the same worker; if it does not, the cause needs concurrency — a resource collision.

Running a single test many times across many workers isolates collisions. A test that uses /tmp/report.csv will pass alone and fail as soon as two copies run at the same moment, because one worker reads the file while another truncates it. A test that is safe passes all sixteen repetitions.

The worker_id fixture, provided by xdist, returns gw0, gw1 and so on — or master when xdist is not active — which makes it the natural key for per-worker resources. tmp_path_factory.getbasetemp().parent is a directory shared by all workers in one run, which gives the file lock and the ready-marker somewhere common to live.

Recognising the neighbour case

When replaying a worker's sequence reproduces the failure, bisect it. Remove the first half of the preceding tests and re-run; if it still fails, the culprit is in the second half. Within a handful of runs you have a pair: test A, then test B fails. What A leaves behind is typically one of a small set: a module-level cache or singleton it populated, an environment variable it set with os.environ instead of monkeypatch, a patched attribute it never restored, the current working directory it changed, or a logging handler it attached.

The fix belongs in A, not B. Replace direct mutation with monkeypatch so pytest restores it, clear caches in a fixture teardown, and treat any global state the application exposes as something tests must reset. Once fixed, the pair passes in both orders, and the suite no longer depends on which tests happen to share a process.

xdist did not create the bug; it removed the accidental reset that used to hide it.

Making a suite parallel-safe by default

Fixing xdist failures one at a time works, but a few conventions stop most of them from appearing in the first place. They are cheap to adopt and easy to enforce in review.

No fixed paths. Every file a test writes goes under tmp_path or tmp_path_factory. A quick search for string literals starting with /tmp or relative paths like "output/" in the test directory finds most violations. Application code that writes to a configurable directory should have that directory pointed at tmp_path by a fixture.

No fixed ports. Servers started in tests bind to port 0 and report the port the operating system chose. Clients get the port from the fixture, never from a constant.

Per-worker names for shared services. Databases, schemas, message-broker queues, cache key prefixes and S3 buckets used by tests include the worker_id. With one PostgreSQL container per CI job, a database per worker keeps workers from seeing each other's data while sharing the expensive container.

No unguarded global mutation. Environment variables, module attributes, the working directory and singletons are changed through monkeypatch, which restores them at teardown. Application-level caches get an explicit reset that an autouse fixture calls.

Random order locally. pytest-randomly in the development environment shuffles test order on every run, which surfaces neighbour dependencies on developers' machines long before xdist rearranges them in CI. When it finds one, the printed seed reproduces the order exactly.

The first four remove shared resources and state; the fifth finds whatever slips through, on a laptop instead of in CI.

When the failure is a crash, not an assertion

Sometimes the symptom under xdist is not a failing assertion but a worker dying: [gw3] node down: Not properly terminated followed by replacing crashed worker gw3. xdist restarts the worker and carries on, and the test that was running is reported as failed with little detail. The usual causes are a segfault in a C extension, the out-of-memory killer reaping a worker that grew too large, or a test that calls os._exit or sys.exit in a subprocess path that turned out to be the worker itself.

Memory is the most common in CI. Each worker holds its own copy of everything the tests import and cache, so -n auto on a runner with many cores and little memory can exceed the limit even when a serial run fits comfortably. Check the runner's kernel log or the CI job's memory graph around the crash, and try -n 2 or -n 4 instead of auto. For segfaults, enable faulthandler output to a file per worker — -p faulthandler is on by default, but its output goes to the worker's stderr, which xdist may not show — and run the crashed worker's sequence serially to reproduce it with a full stack.

Edge cases and failure modes

Output ordering. Worker output interleaves; never infer order from the combined log without the [gwN] prefixes.
--dist loadscope and loadfile. These keep a module or class on one worker, which can hide or reveal neighbour issues. Reproduce with the same mode CI uses.
Databases. One database per worker (test_app_{worker_id}) is simplest; per-test transactions inside it keep tests independent.
Environment variables. Workers inherit the parent environment at startup. Changes made by one worker are invisible to others, which can mask or create differences from serial runs.
Coverage and plugins. Plugins that write to fixed paths — coverage data, reports — need parallel-aware configuration; see the coverage guide below.

Frequently Asked Questions

Why does a test pass alone but fail with pytest -n auto? Under xdist the test runs in a different process, next to different neighbours, possibly at the same moment as tests on other workers. Failures come from shared external resources such as files, ports and databases, from session fixtures running once per worker, or from state left by whichever tests shared its worker.

How do I reproduce the exact order a worker ran tests in? Run with -v to log which worker ran each test, or use --dist loadfile to keep each module's tests together on one worker. Then run that worker's tests serially in the same order with -p no:randomly and the node ids listed explicitly.

Do session-scoped fixtures run once under xdist? No. Each worker is a separate process with its own session, so a session fixture runs once per worker. Fixtures that create shared external resources must coordinate across workers, for example with a file lock, or be made per-worker.

Debugging Tests in CI and Containers — CI-only failure strategy.
Bisecting Test Order Dependencies — finding the interfering test.
pytest-xdist vs pytest-parallel Performance — how xdist schedules work.
Why pytest-cov Reports Zero Under xdist — plugins and parallelism.

← Back to Debugging Tests in CI and Containers