Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Debugging & Performance

Capturing Artifacts from a Failed CI Test Run

A CI failure you cannot reproduce is only debuggable through what the job left behind. Too often that is a truncated log: the assertion line, perhaps a captured log section, and nothing about the state that led there. The temporary directory with the generated file that failed validation is gone. The database the integration test wrote to was torn down with its container. The screenshot that would have shown the browser test's error dialog was never taken. Re-running the job may pass, and the only evidence of the bug disappears.

Keeping evidence is cheap if it is planned. pytest knows exactly when a test fails and has access to its fixtures, so a hook can copy the relevant state into a directory at that moment. The CI system then uploads that directory, and only when the job has failed, so passing runs cost nothing. With a consistent naming scheme, every artefact maps back to the test that produced it.

Prerequisites

Solution

Python
# conftest.py
import re
import shutil
from pathlib import Path

import pytest

ARTIFACTS = Path("test-artifacts")

def _slug(nodeid: str) -> str:
    return re.sub(r"[^A-Za-z0-9_.-]+", "_", nodeid)[:150]

@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    rep = yield
    if rep.failed and rep.when in ("setup", "call"):
        out = ARTIFACTS / _slug(item.nodeid)
        out.mkdir(parents=True, exist_ok=True)
        (out / "report.txt").write_text(rep.longreprtext)
        for name, content in rep.sections:
            (out / f"{_slug(name)}.txt").write_text(content)
        tmp = item.funcargs.get("tmp_path")
        if tmp and Path(tmp).exists():
            shutil.copytree(tmp, out / "tmp_path", dirs_exist_ok=True)
        for collect in item.stash.get(ARTIFACT_COLLECTORS, []):
            collect(out)                    # fixtures register extra collectors
    return rep

ARTIFACT_COLLECTORS = pytest.StashKey[list]()

@pytest.fixture
def artifact(request):
    """Let a fixture or test register a callable that saves evidence on failure."""
    collectors = request.node.stash.setdefault(ARTIFACT_COLLECTORS, [])
    return collectors.append
Python
# An integration fixture that dumps its database on failure.
@pytest.fixture
def db(postgres, artifact):
    artifact(lambda out: postgres.dump_to(out / "db.sql"))
    yield postgres.connect()
YAML
# .github/workflows/tests.yml (excerpt)
      - run: pytest --junitxml=test-artifacts/junit.xml -ra
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: test-artifacts-${{ matrix.python }}-${{ github.run_attempt }}
          path: test-artifacts/
          retention-days: 14
Collecting and uploading evidence only on failure When a test fails, the makereport hook writes the failure report, captured output and tmp_path into a folder named after the test, and runs any collectors that fixtures registered, such as a database dump. After pytest exits with failures, the CI upload step runs only because the job failed and uploads the whole artefact directory. Evidence captured at the moment of failure test fails fixtures still alive test-artifacts/<nodeid>/ report.txt · captured logs tmp_path/ db.sql · screenshot.png junit.xml (whole run) upload-artifact if: failure() · 14 days Passing runs write nothing and upload nothing.
Collecting inside pytest means fixtures still exist when evidence is taken; uploading from CI keeps it after the runner is gone.

Why this works

pytest_runtest_makereport runs three times per test — for setup, call and teardown — and receives the report for each phase. The hook sees the report after pytest has built it and before fixtures are torn down for the call phase, so item.funcargs still holds live fixture values: the tmp_path directory exists, the database connection is open, the browser page is still showing whatever went wrong. That is the only moment when most evidence can be collected.

The collector registry lets each fixture decide what evidence it can provide, instead of the hook knowing about every kind of resource. A database fixture registers a dump, a browser fixture registers a screenshot and the page HTML, an HTTP-mocking fixture registers the recorded requests. Tests that do not use those fixtures pay nothing.

The CI side is deliberately dumb. if: failure() runs the upload only when a previous step failed; the step uploads whatever is in the directory. Including the Python version and run attempt in the artefact name keeps matrix jobs and re-runs from overwriting each other's evidence.

The JUnit report as an index

--junitxml produces a structured record of every test in the run: name, outcome, duration, failure message and captured output. CI systems render it as a test summary, but it is also the index into the artefact directory. Each failed testcase element's name and class map to a node id, and that node id maps to a folder. When reviewing a failed run days later, start with the JUnit summary to see which tests failed, then open only their folders.

Two settings make the report more useful: junit_family = "xunit2" for the modern schema, and junit_logging = "all" to include captured logs and output in the XML itself, so even without the artefact directory the essentials survive in the test summary view.

From JUnit summary to per-test folder The JUnit XML report lists every test with its outcome. A failed test case, identified by class and name, maps to its pytest node id, which maps to the artefact folder of the same name containing that test's report, logs, tmp_path and database dump. Summary first, then only the folders you need junit.xml 1,204 passed 2 failed node id tests/test_export.py::test_csv artefact folder report · logs · tmp_path db.sql
Consistent naming turns a directory of evidence into something you can navigate from the test summary.

Evidence for different kinds of tests

What counts as evidence depends on what the test exercises, and it is worth deciding per fixture rather than collecting everything everywhere.

Unit tests rarely need more than the report and captured output. Their inputs are in the test code; if a failure is not reproducible from that, the problem is usually environmental — a timezone, locale or environment variable — and the most useful extra artefact is a small environment.json written once per session with platform, sys.version, the relevant environment variables and installed package versions from importlib.metadata.

Integration tests need the state of the systems they touched. A SQL dump of the test database (or just the tables the test wrote), the recorded HTTP interactions from a mocking fixture, and the logs of any service containers — collected with docker logs into the artefact folder — usually explain a failure that the assertion alone does not.

Browser tests need visual evidence. Playwright and Selenium fixtures should register a screenshot, the page HTML and, for Playwright, a trace file (context.tracing.stop(path=...)), which records every action, network request and DOM snapshot. Opening that trace in the Playwright trace viewer replays the failed test step by step, which is the closest thing to reproducing the failure without re-running it.

Crashes and hangs need process-level evidence: faulthandler output written to a file, core dumps, and the pytest-timeout stack dump. These must be configured before the run, because the hook will never be called.

Evidence by test kind Four columns list useful evidence per test kind. Unit tests: report, captured output and an environment snapshot. Integration tests: database dump, recorded HTTP traffic and service container logs. Browser tests: screenshot, page HTML and a Playwright trace. Crashes and hangs: faulthandler output, core dumps and timeout stack dumps, configured before the run. Collect what explains this kind of failure unit report · output environment.json integration db dump HTTP recordings container logs browser screenshot · HTML Playwright trace crash / hang faulthandler file core dump set up before the run
Each fixture registers the evidence it can provide; the hook just calls whatever was registered.

Using the artefacts: a short debugging routine

Artefacts are only valuable if someone actually opens them, and a fixed routine that everyone on the team follows makes that quick. Download the artefact bundle for the failed job and open junit.xml or the CI summary to list the failures. For each one, open its folder and read report.txt first — it is the same failure output as the log, but complete rather than truncated. Then read the captured log section from the same folder, looking for the last warning or error before the failure. Only then, with a hypothesis in mind, open the heavier evidence: the database dump, the recorded HTTP traffic, the trace.

Most CI-only failures fall into a handful of categories, and the artefacts usually settle which one within minutes. A difference in environment.json against a local run points at configuration. Rows in the database dump that the test did not create point at leaked state from an earlier test. A recorded HTTP call to a real host points at a missing mock. And an empty tmp_path where the test expected output points at a step that silently did nothing, often because a path or flag differs on the runner. Each category has a known next step, and the artefacts get you to it without re-running the job and hoping it fails again. Over time, the categories you keep hitting also tell you which fixtures deserve better evidence collectors, so the routine gets faster the more it is used.

Edge cases and failure modes

  • xdist workers. Each worker writes into the same directory; node-id-based folder names keep them apart. Avoid per-worker global files unless the worker id is in the name.
  • Huge tmp_path contents. Copying gigabytes of generated data fills artefact storage. Cap the copy size or copy selectively.
  • Secrets in artefacts. Logs and dumps may contain tokens or personal data. Scrub known secret patterns and keep artefacts private to the repository.
  • Crashes that kill pytest. A segfault never reaches the hook. Enable core dumps with ulimit -c unlimited and faulthandler output to a file in the artefact directory.
  • Teardown failures. Errors in fixture teardown produce a report with when == "teardown"; decide whether to collect for those too — the fixtures may already be partly gone.

Frequently Asked Questions

What should I save when a CI test fails? At minimum the JUnit XML report and full pytest output. Beyond that, whatever the failing test's own evidence is: captured logs, the tmp_path contents, HTTP recordings, screenshots for browser tests, database dumps for integration tests, and core dumps for crashes.

How do I upload artefacts only when tests fail? Write artefacts to a known directory from a pytest hook that runs only for failed tests, then use an upload step with if: failure() in GitHub Actions or when: on_failure in GitLab CI so successful runs do not pay the storage cost.

How long should CI artefacts be kept? Long enough to debug a failure after someone notices it — usually 7 to 14 days. Flaky-test investigations benefit from longer retention on the default branch.

← Back to Debugging Tests in CI and Containers