The suite is green locally and red in CI, and the failure output is a stack trace with no context. Everything that would explain it — the state of the temporary directory, the environment, the order the tests ran in — was discarded when the runner terminated. Debugging in CI is therefore less about tools than about deciding, in advance, what evidence a failure must leave behind.
Prerequisites
- A CI system that can upload artefacts, and a container image you can run locally.
- pytest 7.0+, and debugpy 1.8+ for the remote-attach section.
Core concept: capture, then compare
Two activities solve most CI-only failures. The first is capturing enough state at failure time that no re-run is needed. The second is comparing the CI environment with the local one, because the failure is a difference and the difference is almost always in a short list.
- name: Run tests
run: |
pytest -q --junitxml=artifacts/junit.xml \
--basetemp=artifacts/tmp \
-p no:randomly -o cache_dir=artifacts/pytest_cache
env:
PYTHONHASHSEED: "0"
- name: Environment snapshot
if: always()
run: |
{ python -VV; pip freeze; env | sort; nproc; locale; date; } > artifacts/env.txt
- uses: actions/upload-artifact@v4
if: always()
with: { name: test-artifacts, path: artifacts/ }
if: always() is the line that matters most: without it, the upload step is skipped precisely when the tests failed. The environment snapshot is the second: python -VV, pip freeze, env, nproc and locale answer most "why is CI different" questions in one file.
Step-by-step implementation
1. Make the run reproducible. Pin the test order seed, the hash seed and the worker count. -n auto resolving to 2 locally and 16 in CI is by itself enough to change which tests share a database.
2. Keep the temporary directories. --basetemp=artifacts/tmp puts every tmp_path under a collectable path, so the files a failing test wrote are in the artefact bundle. That single change answers most assertion failures about file contents without any further investigation.
3. Record which test ran where. Under pytest-xdist, add the worker id to the junit output or log it from a hook, so a failure concentrated on one worker is visible as such rather than as random flakiness. The hook is the same pytest_runtest_makereport wrapper described in debugging flaky tests with pytest-rerunfailures.
4. Dump state at failure. A conftest.py hook that writes the failing test's locals and the environment to a file turns a stack trace into a diagnosis:
# conftest.py
import json, os, pathlib, pytest
ARTIFACTS = pathlib.Path(os.environ.get("ARTIFACTS", "artifacts"))
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
report = outcome.get_result()
if report.when == "call" and report.failed:
ARTIFACTS.mkdir(parents=True, exist_ok=True)
target = ARTIFACTS / f"{item.name}.json"
target.write_text(json.dumps({
"nodeid": item.nodeid,
"worker": os.environ.get("PYTEST_XDIST_WORKER", "master"),
"cwd": os.getcwd(),
"env": {k: v for k, v in os.environ.items() if k.startswith("APP_")},
"longrepr": str(report.longrepr)[:4000],
}, indent=2))
Reading a CI log like a diagnostic instrument
Most CI logs are read as a wall of text ending in a traceback. Configured deliberately, the same log answers four questions before the traceback is reached, and each answer costs one flag.
What ran, and in what order? -v prints every node id as it executes, and -p randomly prints the seed in the header. Together they make the run reproducible.
Why was a test skipped or expected to fail? -ra prints a short summary of every non-passing outcome with its reason. A suite reporting "142 passed, 38 skipped" tells you nothing; the same run with -ra tells you that 38 tests skipped because an environment variable was unset, which is usually the actual bug.
How long did each phase take? --durations=15 reports the slowest tests including setup and teardown, which is how a timeout in CI is attributed to a fixture rather than to the test that happened to be running.
What was the environment? The snapshot step from earlier: version, packages, environment, CPU count, locale.
$ pytest -q -ra -v --durations=15 --junitxml=artifacts/junit.xml
...
=========================== short test summary info ============================
SKIPPED [12] tests/integration/conftest.py:18: RUN_INTEGRATION not set
XFAIL tests/test_legacy.py::test_v1_format - known broken, see PROJ-4417
FAILED tests/test_orders.py::test_total - assert 250 == 300
=============================== slowest durations ==============================
41.02s setup tests/integration/test_checkout.py::test_flow
0.31s call tests/test_orders.py::test_total
Those two blocks are worth more than the traceback in most investigations: the skip reason explains why coverage is lower than expected, and the durations block explains a job timeout that the traceback cannot.
Verification
Confirm the capture works before you need it. Add a deliberately failing test on a branch, run the pipeline, and check that the artefact bundle contains the junit file, the temporary directories, the environment snapshot and the per-test dump. A capture pipeline that has never been exercised usually has one missing if: always().
$ unzip -l test-artifacts.zip | head
artifacts/env.txt
artifacts/junit.xml
artifacts/tmp/test_writes_report0/out/report.csv
artifacts/test_writes_report.json
Then verify the reproduction path: run the same image locally with the same entry point.
$ docker run --rm -it -v "$PWD:/src" -w /src ghcr.io/acme/ci-python:3.12 \
pytest -q -n 4 -p no:randomly
Container constraints that change test behaviour
A container is not a small virtual machine, and four of its differences from a developer laptop change how tests behave rather than merely how fast they run.
CPU quota is not CPU count. nproc inside a container usually reports the host's core count, while the cgroup quota limits actual usage. -n auto therefore starts sixteen workers on a two-CPU quota, and every one of them contends. Pin the worker count explicitly, or read the quota:
# conftest.py — worker count that respects the cgroup quota
import os, pathlib
def cpu_quota() -> int:
try:
quota, period = pathlib.Path("/sys/fs/cgroup/cpu.max").read_text().split()
if quota != "max":
return max(1, int(int(quota) / int(period)))
except (OSError, ValueError):
pass
return os.cpu_count() or 1
Memory limits kill rather than slow. An out-of-memory kill inside a container terminates the process with no traceback, and the CI log shows a truncated run and exit code 137. A test suite that "hangs and dies" in CI but runs locally is usually hitting the limit — check the exit code before looking anywhere else.
PID 1 does not reap children. A container whose entry point is pytest makes it PID 1, and zombie processes from subprocess tests accumulate because PID 1 has no default SIGCHLD handling. Run with --init or an init shim when tests spawn processes.
The filesystem may be read-only or overlay-backed. A read-only root filesystem breaks anything writing outside the mounted volumes, including .pytest_cache and coverage data files; point both at a writable path explicitly.
$ pytest -q -o cache_dir=/tmp/pytest_cache --basetemp=/tmp/pytest_tmp
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| Passes locally, fails in CI | different test order or worker count | pin the seed and -n, then re-run |
Fails only under -n | shared port, database or temp path | partition by PYTEST_XDIST_WORKER |
| Timeouts only in CI | slower runner, smaller CPU quota | raise timeouts or match container resources |
| Assertion about a file, no file to inspect | temporary directories not collected | --basetemp into the artefact path |
| Date or number formatting differs | locale or timezone unset in the image | pin LC_ALL and TZ in the job |
| Works in the image locally, not in CI | cached layer or different image digest | pin the image by digest, not by tag |
A capture policy worth adopting
The difference between a pipeline that can be debugged and one that cannot is a short, deliberate list of what every job keeps. Adopting it once removes most re-runs.
Always, on every run: the junit XML and the full log. Both are small, both are machine-readable, and having them for passing runs is what makes a comparison possible when a later run fails.
On failure: the pytest temporary directories, the environment snapshot, per-test failure dumps, and any application logs the tests produced. This is the bundle that answers "what state was the system in", and it costs nothing on green runs because it is gated on failure.
On a schedule: a full artefact bundle from a known-good run, kept as a baseline. Comparing a failing environment snapshot against a passing one from the same week is frequently the fastest route to a dependency change nobody noticed.
Never: secrets. An environment dump includes every variable, and CI environments are full of tokens. Filter the snapshot before writing it:
$ env | sort | grep -vE '(TOKEN|SECRET|PASSWORD|KEY|CREDENTIAL)' > artifacts/env.txt
The policy is worth writing into the pipeline template rather than into each job, so new services inherit it. And it is worth testing: a deliberately failing branch, run once per quarter, confirms the artefacts still upload and still contain what the policy says they do.
The underlying principle is that a CI failure is a measurement you get to make once. Everything the runner discards has to be recreated by re-running, and re-running a flaky failure is exactly the thing that does not work.
Making CI failures attributable
The last piece is organisational rather than technical: a failing pipeline needs an owner within minutes, or the artefacts are never read.
Three signals make attribution automatic. Name the failing test in the job summary, not just in the log — most CI systems render a markdown summary, and one line naming the test and its file routes the failure to the person who touched it. Group failures by directory, because a monorepo's pipeline failing on services/billing is a billing problem and everyone else can ignore it. And separate infrastructure failures from test failures with a distinct exit path: a job that fails because a container could not pull an image should not look like a test failure, or the on-call engineer starts debugging the wrong thing.
$ pytest -q --junitxml=artifacts/junit.xml || rc=$?
$ python - <<'EOF' >> "$GITHUB_STEP_SUMMARY"
import xml.etree.ElementTree as ET
root = ET.parse("artifacts/junit.xml").getroot()
failures = [c for c in root.iter("testcase") if c.find("failure") is not None]
print(f"### {len(failures)} failing test(s)")
for case in failures[:20]:
print(f"- `{case.get('classname')}::{case.get('name')}`")
EOF
$ exit ${rc:-0}
That snippet costs nothing and changes the experience of a red pipeline from "open the log and scroll" to "read three lines". Combined with the artefact bundle, it is usually enough to diagnose without a re-run — which is the whole objective, because a re-run of an intermittent failure is a coin toss rather than an investigation. One organisational habit closes the loop: when a CI-only failure is finally diagnosed, write the cause into the testing README as a one-line entry — "March 2026: locale unset in the runner image made date assertions fail". The list becomes the first thing to check next time, and after a handful of entries it is a far faster diagnostic than any tool, because the same handful of causes recur across every project.
Frequently Asked Questions
Why do tests pass locally and fail in CI? Because something differs and the difference is usually invisible: test ordering, parallelism, CPU count, timezone, locale, filesystem case sensitivity, available memory, or an environment variable your shell sets and the runner does not.
Is an interactive debugger usable in CI? Rarely and only deliberately. A prompt with no terminal hangs the job. Capture artefacts by default, and use a remote debugger only for a manually triggered debugging run.
What should every CI job upload on failure? The junit XML, the full log, any coverage data, the pytest temporary directories, and a dump of the environment. Those five cover most investigations without a re-run.
How do I reproduce a CI failure locally? Run the same container image with the same command, the same test order seed and the same worker count. Matching those four resolves the majority of CI-only failures.
Building a debuggable pipeline from the start
Everything above is easier to add before it is needed. Five properties distinguish a pipeline that can be debugged from one that produces only a red cross, and all five are configuration rather than code.
Deterministic by default. The seeds, the timezone, the locale and the worker count are pinned in the project configuration, so the same command produces the same run everywhere. This is the property that makes every other one useful.
Artefacts on failure, always. The upload step is gated on always() and the bundle contains the log, the junit XML, the temporary directories and the environment. Nothing needs deciding at incident time.
One command. The pipeline invokes the same make test a developer runs, in the same image. A workflow file containing a bespoke pytest invocation is a second, undocumented test configuration that will drift.
Fast feedback split from slow feedback. Unit tests, integration tests and the scheduled jobs are separate, so a failure names its category before anyone reads a log. A single job running everything means every failure looks the same at a distance.
Reproducible images. Pinned by digest, with the digest recorded in the run. "It worked yesterday" is answerable only when yesterday's image can be pulled today.
# Makefile — one entry point, used by developers and CI alike
IMAGE ?= ghcr.io/acme/ci-python@sha256:6b1c...
test:
docker run --rm -v "$(PWD):/src" -w /src -e TZ=UTC -e LC_ALL=C.UTF-8 -e PYTHONHASHSEED=0 --cpus=2 --memory=4g $(IMAGE) pytest -q -ra -n 2 --basetemp=artifacts/tmp --junitxml=artifacts/junit.xml
A team that adopts that Makefile stops having CI-only failures for the ordinary reasons, because there is no longer a meaningful difference between the two environments. What remains — genuine concurrency bugs, resource-limit interactions, upstream flakiness — is the set of problems worth spending debugging effort on.
Reading the exit code first
Before any log is opened, the exit code narrows the problem to a category. pytest's own codes are documented and distinct, and the shell adds a few that matter more.
Exit 1 is ordinary test failures. Exit 2 is an interrupted run — usually a timeout or a cancellation. Exit 3 is an internal error, which almost always means a broken plugin or hook rather than a broken test. Exit 4 is a usage error, typically a bad flag or a path that does not exist in the container. Exit 5 means no tests were collected at all, which is the silent failure worth alerting on separately: a green-looking pipeline that ran nothing.
Beyond pytest, 137 is a SIGKILL, which in a container means the memory limit was hit; 139 is a segmentation fault, pointing at a native extension; and 124 is the shell timeout command firing.
$ pytest -q; echo "exit: $?"
exit: 5
Adding that one line to the job, and alerting on 5 specifically, catches the class of failure where a path change or a marker typo silently empties the run — the one CI failure mode that looks exactly like success.
Related guides
- Reproducing CI-only test failures locally — the environment checklist in detail.
- Attaching debugpy to a container — a real debugger session against a containerised run.
- Debugging flaky tests with pytest-rerunfailures — recording every attempt so the pattern is visible.
- Post-mortem debugging with pdb.pm() — dumping frames when no terminal exists.