pytest-xdist vs pytest-parallel Performance

A suite that runs in eight minutes sequentially still takes six under pytest-parallel thread mode but two under pytest-xdist -n auto — or the reverse, depending entirely on whether the work is CPU-bound or I/O-bound. The choice between the two runners is not a benchmark contest but a match between workload profile and concurrency primitive: pytest-xdist spawns isolated interpreter processes over execnet, while pytest-parallel uses in-process threads or multiprocessing. This guide compares their execution models, fixture and pickling constraints, and the failure modes that only appear under concurrency.

Prerequisites

• pytest >= 8.0, Python 3.9+.
• pytest-xdist >= 3.0 (actively maintained). pytest-parallel has been effectively unmaintained since 2021 and does not officially support recent pytest releases — confirm it imports against your pinned pytest before relying on it.
• For benchmarking: pytest-benchmark, memory_profiler, and cProfile from the standard library.

The collection caching that reduces per-worker startup cost is detailed in Optimizing Test Discovery; per-worker fixture instantiation builds on Managing Conftest Hierarchies.

Solution

Start from the workload profile, then pick the runner whose worker model fits.

pytest-xdist uses execnet to spawn isolated interpreters that communicate over pickled RPC, so each worker pays full interpreter startup and conftest.py evaluation but gains absolute memory isolation. The module- and session-scoped fixtures you rely on for expensive setup are instantiated once per worker, not once per run. pytest-parallel uses multiprocessing.Pool (process mode) or ThreadPoolExecutor (thread mode); thread mode shares the interpreter heap for near-zero startup cost but exposes you to GIL contention on CPU-bound work and race conditions on any shared global state.

Run each with the appropriate flags and capture metrics:

# pytest-xdist: auto-detect CPUs, group tests by module to reuse fixtures.
pytest -n auto --dist loadscope --benchmark-only --benchmark-save=xdist

# pytest-parallel: thread pool by default; force processes for CPU-bound work.
pytest --workers auto --benchmark-only --benchmark-save=parallel

To find serialization bottlenecks, profile the run with cProfile or py-spy and watch for multiprocessing.reduction (pytest-parallel) or execnet.remote (pytest-xdist) dominating the call graph — both signal non-picklable fixtures or excessive parametrization.

Why this works

The two runners win in opposite regimes because the cost they avoid differs. pytest-xdist's process isolation removes the GIL ceiling and prevents cross-test contamination, so CPU-bound suites scale past eight cores and loadscope/loadfile distribution amortizes expensive fixture setup. pytest-parallel's thread mode skips interpreter duplication entirely, cutting peak RSS by 60-80% and eliminating spawn latency, which dominates total time for short I/O-bound suites where the GIL is released during blocking calls. Picking the wrong one means paying a penalty (interpreter startup, or GIL contention) that exceeds the parallelism gain.

Edge cases and failure modes

• Fixture scope leaks. Module/session fixtures instantiate per worker in both runners' process modes; a fixture wrapping a mutable singleton produces failures that vanish at -n 1. Diagnose with pytest --setup-show -n auto, then narrow scope or key resources by os.getpid().
• Pickling errors in pytest-parallel. TypeError: cannot pickle 'function' object arises from closures or dynamically generated fixtures crossing the multiprocessing queue. Move closures to module level or switch to thread mode for I/O-bound work; execnet handles more types but still fails on file descriptors and C extensions.
• Coverage fragmentation. Workers overwrite each other's .coverage. Use --cov-append, then coverage combine; for xdist add --cov-context to attribute branches to workers. See memory profiling with tracemalloc for tracking worker RSS growth.
• Hypothesis example DB desync. Independent workers build separate example databases, defeating shrinking. Point them at a shared DirectoryBasedExampleDatabase(".hypothesis/examples") — see Hypothesis Framework Fundamentals.
• OS resource exhaustion. OSError: [Errno 24] Too many open files or OOM kills under high concurrency. Raise ulimit -n to 65536, set --max-worker-restart=3, and use connection pooling with bounded max_overflow.
• Order-dependent flakiness surfaced by sharding. Distributing tests across workers reorders execution and exposes hidden state coupling that passed in a fixed sequential order. Reproduce with -p no:randomly to pin ordering, then quarantine the offenders as covered in debugging flaky tests with pytest-rerunfailures.

Making a parallel suite reproducible

Parallelism converts hidden coupling into intermittent failure, so the work of adopting it is mostly the work of partitioning shared resources. Four resources account for nearly every cross-worker collision, and each has a standard fix keyed on the worker id.

PYTEST_XDIST_WORKER is set in every worker process (gw0, gw1, …) and is the partition key for all of them:

# conftest.py
import os
import pytest

def worker_id() -> str:
    return os.environ.get("PYTEST_XDIST_WORKER", "master")

@pytest.fixture(scope="session")
def database_url():
    # One database per worker; created by CI before the run, dropped after.
    return f"postgresql:///test_{worker_id()}"

@pytest.fixture(scope="session")
def http_port():
    # Deterministic port per worker beats binding to 0 and hoping.
    return 8100 + int(worker_id().removeprefix("gw") or 0)

@pytest.fixture
def scratch_dir(tmp_path_factory):
    # tmp_path_factory is already worker-safe; never hardcode /tmp/fixtures.
    return tmp_path_factory.mktemp("scratch")

Databases, ports, temporary directories and cache files are the four. Anything else that a test writes — a fixture file, a log path, a coverage data file — needs the same treatment or a guarantee that only one worker touches it.

Distribution mode is the second decision. The default --dist load hands each test to whichever worker is free, which maximises throughput but means class- and module-scoped fixtures can be built on several workers at once. --dist loadscope keeps every test in a class or module on one worker, and --dist loadfile does the same per file. For a suite with expensive module-scoped fixtures, loadfile is frequently faster than load despite worse balancing, because the fixture is built once per file instead of once per worker per file.

$ pytest -n 8 --dist loadfile          # module fixtures built once each
$ pytest -n 8 --dist load              # best balancing, most fixture rebuilds
$ pytest -n 8 --dist loadgroup         # honour @pytest.mark.xdist_group

loadgroup is the escape hatch for the tests that genuinely cannot run beside each other: mark them with the same group and xdist pins them to one worker without serialising the rest of the suite.

The third decision is worker count, and more is not better. Each worker is a full interpreter with its own imports and its own fixture setup, so the fixed cost per worker is real: on a suite with a three-second import cost, eight workers spend twenty-four seconds importing before the first test runs. Past the point where import and fixture overhead exceed the time saved, adding workers makes the run slower — and on CI runners with two cores, -n 8 is pure contention. Measure with -n auto as the baseline and step down as well as up.

Finally, keep the run reproducible when it fails. -p no:randomly (if you use pytest-randomly) and a fixed PYTHONHASHSEED remove two sources of variation, and --dist loadfile makes the assignment of tests to workers deterministic for a fixed worker count. A parallel failure you can reproduce locally with the same distribution mode is a normal bug; one you cannot is a week of guessing.

Frequently Asked Questions

Why does pytest-parallel fail with 'cannot pickle local object' while pytest-xdist works?pytest-parallel uses standard multiprocessing, whose pickle protocol cannot serialize lambdas, closures, or non-picklable C extensions. pytest-xdist uses execnet's custom serialization, which handles more object types. Refactor closures into module-level functions, avoid dynamic fixtures, or use thread mode for I/O-bound work.

Can pytest-xdist and pytest-parallel be combined for nested parallelism? No. Both override pytest_runtestloop and pytest_collection_modifyitems to control distribution, so nesting them causes hook recursion, worker deadlocks, and dropped tests. Pick one runner per suite based on whether the workload is CPU-bound or I/O-bound.

Which runner is faster, and should I still consider pytest-parallel?pytest-xdist wins for CPU-bound suites with heavy fixtures via loadscope/loadfile distribution; pytest-parallel's thread mode wins for lightweight I/O-bound tests by avoiding interpreter duplication. But pytest-parallel has been effectively unmaintained since 2021 and lacks support for recent pytest releases, so verify compatibility before adopting it. Does coverage measurement work under xdist? Yes, with one requirement: pytest-cov must combine the per-worker data files, which it does automatically when it is the plugin collecting coverage. Reporting zero percent under -n almost always means coverage was started outside pytest — for example by invoking coverage run -m pytest — so each worker wrote its own file and nothing combined them. Let pytest-cov own the run, and set parallel = true plus concurrency = multiprocessing in the coverage configuration when subprocesses are involved.

Related

• Debugging flaky tests with pytest-rerunfailures — quarantine the order-dependent failures that sharding exposes.
• Managing conftest hierarchies — control the per-worker conftest.py evaluation that dominates xdist startup cost.
• Mastering pytest fixtures — scope and key fixtures so per-worker instantiation stays isolated.
• CPU profiling with cProfile and py-spy — confirm a suite is CPU-bound before reaching for process workers.
• Memory profiling with tracemalloc — track worker RSS growth when tuning concurrency limits.

← Back to Optimizing Test Discovery