Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Debugging & Performance

Reproducing CI-Only Test Failures Locally

A failure that only happens in CI is not mysterious; it is a failure whose cause is in an environment dimension you have not matched yet. There are about six such dimensions, they can be enumerated, and working through them in order turns the investigation from guesswork into a checklist that usually terminates in ten minutes.

Prerequisites

  • The artefact bundle from the failing run — log, junit XML and environment snapshot, as set up in debugging tests in CI and containers.
  • Docker or Podman, and the CI image reference.

Solution

Work down the list. Each step is one command and each eliminates a class of cause.

1. The same image, pinned by digest. A tag moves; a digest does not, and "it works in the image" is meaningless if the image is not the one CI ran.

Bash
$ docker run --rm -it -v "$PWD:/src" -w /src \
    ghcr.io/acme/ci-python@sha256:6b1c... bash

2. The same command. Copy it verbatim from the workflow file, including every flag. -n auto, -p no:randomly, --dist loadfile and -o overrides all change behaviour, and a locally-typed approximation is the most common reason a reproduction attempt fails.

3. The same order. If the suite uses pytest-randomly, the seed is in the run header — reuse it. Otherwise, extract the executed order from the junit XML and replay it:

Bash
$ pytest -p randomly -p no:cacheprovider --randomly-seed=1701 -q      # same shuffle
$ python -c "
import xml.etree.ElementTree as ET
root = ET.parse('artifacts/junit.xml').getroot()
for case in root.iter('testcase'):
    print(f\"{case.get('file')}::{case.get('name')}\")
" > order.txt
$ pytest -q $(tr '\n' ' ' < order.txt)                                # exact order

4. The same parallelism. -n auto resolves to the CPU count, which differs between a laptop and a runner. Pin it to the number CI used, and pin the distribution mode too.

5. The same resource limits. A two-CPU, four-gigabyte runner behaves differently from a sixteen-core workstation:

Bash
$ docker run --rm -it --cpus=2 --memory=4g -v "$PWD:/src" -w /src \
    ghcr.io/acme/ci-python@sha256:6b1c... pytest -q -n 2

6. The same locale, timezone and hash seed. These come from the environment snapshot:

Bash
$ docker run --rm -e TZ=UTC -e LC_ALL=C.UTF-8 -e PYTHONHASHSEED=0 ...
Six dimensions to match, in order A vertical checklist of the six dimensions that differ between a laptop and a CI runner: the image pinned by digest, the exact command, the test order and seed, the parallel worker count, the CPU and memory limits, and the locale, timezone and hash seed. Six dimensions to match, in order the image, by digest a tag is not a version Digests are in the run log the exact command every flag matters Copy it from the workflow file the order and seed the most common cause Seed is printed in the header workers and resources -n auto is not a number Pin both -n and --dist locale, TZ, hash seed formatting and iteration order All three come from env.txt
Work down the list and stop when the failure appears: the step that reproduced it is the diagnosis.

Why this works

A test failure is deterministic given the same code and the same inputs. When CI and a laptop disagree, the inputs differ — and the inputs to a test suite include far more than the source tree. Test order determines which state a test inherits from its predecessors; worker count determines which tests share a database, a port or a temporary path; CPU count and memory affect timing-sensitive assertions and garbage-collection behaviour; and locale, timezone and hash seed change formatting, sorting and dictionary iteration order. Matching all six makes the two runs the same experiment, at which point the failure either reproduces or the remaining difference is genuinely unusual.

Which dimension explains which symptom A table matching four CI-only symptoms - failure only in a full run, failure only under parallelism, timing assertion failures, and formatting or ordering differences - to the dimension responsible. Which dimension explains which symptom Criterion Likely dimension Confirm by only in a full run test order replay the recorded order only under -n worker partitioning run with -n 1 timing assertions CPU limits run with --cpus matched formatting or sorting locale, hash seed export the CI values
The symptom names the dimension, which is why the checklist rarely needs to be worked through in full.

Isolating an order dependency

If step three reproduces the failure, the cause is a hidden dependency between tests, and pytest can find it faster than reading code.

Bash
$ pip install pytest-randomly            # if not already present
$ pytest -q --randomly-seed=1701         # reproduce
$ pytest -q --randomly-seed=1701 -p no:randomly tests/test_a.py tests/test_b.py

For a large suite, bisect the ordering rather than the code. pytest --lf reruns the last failure alone — if it passes in isolation, the dependency is confirmed. Then halve the preceding test list repeatedly until a minimal pair remains:

Bash
$ pytest -q $(head -n 400 order.txt | tr '\n' ' ')      # still fails?
$ pytest -q $(head -n 200 order.txt | tr '\n' ' ')      # narrow

The pair that remains is almost always one test mutating global state and another reading it — a module-level cache, an environment variable set without monkeypatch, a registry that was appended to. The fix belongs in the first test, and the techniques are in replacing singletons and module globals in tests and patching environment variables with monkeypatch.setenv.

When it still will not reproduce

Three residual causes account for most of the remainder, and each has a distinguishing test.

Network and DNS. CI runners resolve differently, have different egress rules, and often no outbound access at all. A test that passes locally because it silently reached the real API will fail in CI with a timeout — run locally with networking disabled (docker run --network=none) to confirm.

Time of day and clock skew. A test that fails only on scheduled runs is usually reading the wall clock: a date boundary, a token expiry, a business-hours check. Run with the CI job's timestamp faked to confirm.

Filesystem semantics. Case sensitivity, path length limits and file ordering from os.listdir differ between macOS, Linux and network mounts. A test asserting on a sorted directory listing may pass on one and fail on the other; run inside the Linux image rather than natively on macOS.

If none of these apply and the failure remains CI-only, the next tool is a remote debugger attached to the containerised run, which is covered in attaching debugpy to a container.

Edge cases and failure modes

  • Bind-mounting the working tree changes filesystem behaviour. Mount performance and case sensitivity differ from a copied-in tree, so a reproduction that needs the exact filesystem should build the image rather than mounting.
  • --randomly-seed only applies when the plugin is installed. A seed pasted into a run without the plugin is silently ignored.
  • Junit XML does not record execution order under xdist reliably. Use the per-test artefact dump to record worker and start time instead.
  • Matching --cpus does not match CPU model. Timing-sensitive tests can still differ; treat wall-clock assertions as a defect rather than something to reproduce exactly.
  • Cached CI layers hide dependency changes. A pip install in a cached layer may resolve to different versions than a fresh run; compare pip freeze from the snapshot.
  • Some failures are genuinely load-dependent. A runner executing three jobs concurrently has different timing than an idle laptop, and the honest fix is removing the timing dependency.

Preventing the next CI-only failure

Reproducing one failure is worth doing; removing the class is worth more. Four changes narrow the gap between a laptop and a runner permanently.

Run the container locally by default. A make test target that runs the CI image with the CI command means developers and CI execute the same thing, and the whole category of "works on my machine" disappears without anybody being disciplined about it.

Pin the non-deterministic dimensions in configuration rather than in the workflow. PYTHONHASHSEED, TZ, LC_ALL and the worker count belong in the project's test configuration, where both environments read them, rather than in a CI file only the runner sees.

Randomise order locally, not just in CI. If CI shuffles and laptops do not, order-dependent failures are found by CI first every time. Installing pytest-randomly for everyone moves that discovery to the machine where debugging is easy.

Assert the environment in a fixture. A session-scoped check that fails fast when the timezone, locale or worker configuration is unexpected turns a subtle behavioural difference into an explicit message:

Python
import os, time, pytest

@pytest.fixture(scope="session", autouse=True)
def _assert_expected_environment():
    assert time.tzname[0] == "UTC", f"expected UTC, got {time.tzname[0]}"
    assert os.environ.get("PYTHONHASHSEED") == "0", "set PYTHONHASHSEED=0"

That fixture reads as bureaucratic until the first time it catches a runner image upgrade that changed the default locale — at which point it has paid for itself several times over.

Frequently Asked Questions

What is the single most common cause of a CI-only failure? Test order. CI often runs the full suite in a different order or with a different worker count than a developer running one file, and a hidden dependency between tests only shows in some orders.

How do I run CI's exact test order locally? Read the order from the junit XML or the log, then pass the node ids in that order, or reuse the same pytest-randomly seed which is printed in the run header.

Do resource limits matter for correctness? Yes. A container with two CPUs changes -n auto, timing-sensitive assertions and thread scheduling, and a low memory limit changes when the allocator returns memory. Match both when reproducing. Should I run the whole suite locally to reproduce, or just the failing test? Start with the failing test alone: if it fails in isolation, the environment is the cause and the checklist above applies. If it passes alone and fails in the full run, the cause is order or shared state, and the bisection procedure is the faster path. Running the whole suite as the first move wastes several minutes on the most likely outcome, which is that the test passes.

What each isolation level tells you A stack of four reproduction attempts in increasing scope: the test alone, its module, the full suite serially, and the full suite in parallel, with the conclusion each supports. What each isolation level tells you the test alone fails: environment. passes: shared state its module fails: a sibling test dirtied something full suite, serial fails: order dependence, bisect it full suite, parallel fails only here: worker partitioning
Each level costs more time and eliminates a different cause, so working upward finds the answer in the fewest minutes.

← Back to Debugging Tests in CI and Containers