Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Pytest & CI

Debugging Flaky Tests with pytest-rerunfailures

A test that passes locally but fails roughly one run in twenty under CI — then passes on the automatic retry — is the canonical flaky test, and reaching for pytest-rerunfailures to silence it usually buries the real defect. Used deliberately, the plugin is a diagnostic amplifier: it forces a non-deterministic execution path into a repeatable state where you can capture sys.modules, GC object counts, and thread counts at the exact moment of failure. This guide shows the rerun mechanics, a state-capture hook, and the retry policy that surfaces root causes instead of hiding them.

Prerequisites

  • pytest-rerunfailures >= 14.0, pytest >= 8.0, Python 3.9+.
  • Optional companions: pytest-randomly (order shuffling) and pytest-xdist >= 3.0 (parallel runs — note reruns are worker-local).
  • Markers registered so @pytest.mark.flaky does not warn:
TOML
# pyproject.toml
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = ["flaky: test with known transient instability under controlled rerun"]

The collection ordering that makes order-dependent flakiness reproducible is covered in Optimizing Test Discovery; timing-based flakiness in async suites overlaps with Debugging Async Code and Event Loops.

Solution

pytest-rerunfailures registers a pytest_runtest_makereport implementation. On a call-phase failure it re-invokes pytest_runtest_protocol for the same item: setup runs again (so function-scoped fixtures are recreated) but teardown is deferred until the final attempt. Anchor your diagnostics to makereport, not teardown, and dump state the instant the failure is observed:

The rerun protocol for a single test item under --reruns=N For one item, pytest runs setup then the call phase. Function-scoped fixtures are recreated each attempt. The pytest_runtest_makereport hookwrapper snapshots sys.modules, GC object counts, and thread counts the instant the call fails, before any retry mutates process state. If the call failed and the rerun counter is still below N, the counter increments and setup and call repeat. Once the test passes or the counter reaches N, teardown runs exactly once, and any module, class, or session-scoped fixtures keep the mutated state they accumulated across attempts. One item under --reruns=N — where the snapshot fits one attempt — repeats while call fails and rerun < N setup — function-scoped fixtures recreated call — execute the test body (fails) pytest_runtest_makereport — hookwrapper snapshot sys.modules · gc_objects · threads (before retry mutates state) call failed and rerun < N ? yes — rerun += 1, up to N no — passed, or rerun == N teardown — runs ONCE, after the final attempt wider-scoped fixtures keep their mutated state
One item under --reruns: each attempt re-runs setup, call and teardown, and only the final attempt's report reaches the terminal summary.
Python
# conftest.py
import pytest, sys, gc, threading, json

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield                       # let the report be built first
    report = outcome.get_result()
    # Only snapshot on the failing call phase, before the plugin retries.
    if report.when == "call" and report.failed:
        snapshot = {
            "nodeid": item.nodeid,
            "sys_modules": len(sys.modules),       # detects import-time leakage
            "gc_objects": len(gc.get_objects()),   # rising count => retained objects
            "thread_count": len(threading.enumerate()),  # leaked workers / pools
            "rerun": getattr(report, "rerun", 0),  # which attempt this was
        }
        fname = f"flaky_state_{item.nodeid.replace('::', '_')}_{snapshot['rerun']}.json"
        with open(fname, "w") as fh:
            json.dump(snapshot, fh, indent=2)

Comparing gc_objects and thread_count across the per-attempt artifacts separates a memory leak (monotonic growth) from transient concurrency contention (flat counts, intermittent failure). Reproduce verbosely first:

Bash
# Stream real-time traces CI normally suppresses, and shuffle order.
pytest --reruns=3 --capture=no --log-cli-level=DEBUG -p randomly

Apply a delay only when timing is the suspect, and keep teardown explicit so state cannot leak across attempts:

Python
import pytest
from sqlalchemy import create_engine, text

@pytest.fixture
def db_session():
    conn = create_engine("sqlite:///:memory:").connect()
    yield conn
    conn.execute(text("DROP TABLE IF EXISTS test_data"))  # reset before final teardown
    conn.close()
    assert conn.closed, "Connection leak detected during rerun teardown"

@pytest.mark.flaky(reruns=3, reruns_delay=2)  # delay lets a transient partition heal
def test_writes_then_reads(db_session):
    ...

Why this works

The plugin replays only the setup and call phases per attempt and runs teardown exactly once after the last attempt, so a function-scoped fixture is genuinely fresh each retry while wider-scoped fixtures are not. Capturing state inside the makereport hookwrapper records the process exactly when the failure is observed — before any retry mutates it — turning a Heisenbug into a diff between artifacts. A passing-with-delay, failing-without-delay result is strong evidence of timing-dependent resource contention rather than an assertion error, which tells you where to look instead of just hiding the symptom.

Edge cases and failure modes

  • Wider-scoped fixtures compound corruption. Session/module/class fixtures never reset between reruns, so a parametrized test sharing one of them inherits mutated state from earlier failures. Convert to function scope or clear state in the yield teardown.
  • Hypothesis shrinking conflicts. Reruns break Hypothesis's deterministic shrinking, causing infinite reduction or false passes. Check the PYTEST_RERUN_COUNT environment variable and skip property tests during retries, or isolate Hypothesis suites from retry logic entirely — see Hypothesis Framework Fundamentals.
  • Worker-local reruns under pytest-xdist. A failure on worker A retries only on A; shared DBs, ports, or locks cause cross-worker cascades. Route flaky subsets to sequential execution.
  • Coverage / JUnit XML overwrite. Each attempt can overwrite reports. Use --cov-append plus coverage combine, and collect per-attempt JUnit artifacts to preserve failure history.
  • --reruns as a permanent fix. It masks genuine regressions and inflates pipeline time. Cap at 2-3, pair with failure-signature hashing, and quarantine tests that keep recurring.

Turning reruns into evidence instead of noise

A rerun that turns a red test green has told you something important, but only if you record it. Left at the default, --reruns deletes exactly the information you need: the first failure's traceback is discarded once a later attempt passes, and the suite reports a clean run.

Capture the discarded attempts with a pytest_runtest_makereport wrapper that writes one line per failed attempt to a durable file. Because the hook sees every attempt, the record is complete even when the final outcome is a pass.

Python
# conftest.py
import json, os, pathlib, pytest

FLAKE_LOG = pathlib.Path(os.environ.get("FLAKE_LOG", "flaky-attempts.jsonl"))

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when == "call" and report.failed:
        record = {
            "nodeid": item.nodeid,
            "attempt": getattr(item, "execution_count", 1),   # set by pytest-rerunfailures
            "worker": os.environ.get("PYTEST_XDIST_WORKER", "master"),
            "longrepr": str(report.longrepr)[:2000],
        }
        with FLAKE_LOG.open("a") as fh:
            fh.write(json.dumps(record) + "\n")

Ship that file as a CI artifact and the flakiness conversation changes character: instead of "test_checkout is flaky sometimes", you have thirty records showing that every failure is attempt one on worker gw3 with the same ConnectionResetError. That is a resource-contention bug with an address, not a mystery.

Three analysis habits follow from having the data. Group by node id and count attempts — a test that only ever fails on attempt one and passes on attempt two is order- or warm-up-dependent, while a test that fails on attempts one and two equally is genuinely non-deterministic. Group by worker id to expose contention: failures concentrated on one worker point at a shared port, directory or database that is not partitioned per worker. And group by exception type; a mix of unrelated exceptions from one test usually means the test is racing its own setup rather than fighting the environment.

Finally, put a ceiling on the mechanism so it cannot hide a real regression. --reruns 1 is enough to smooth over genuine infrastructure blips; --reruns 5 will make an outright broken test look intermittent. Pair it with --reruns-delay 1 so a rerun does not immediately re-hit a rate limiter, and restrict reruns to the tests that have earned them with --only-rerun and an exception pattern:

Bash
$ pytest --reruns 2 --reruns-delay 1 --only-rerun ConnectionResetError \
         --only-rerun 'Timeout.*upstream'

Anything outside that list fails on the first attempt, which keeps assertion errors — the failures that are always real — out of the rerun path entirely.

Reading the rerun record A table matching four rerun patterns to their likely cause and the fix: failures only on the first attempt, failures spread evenly across attempts, failures concentrated on one xdist worker, and mixed unrelated exceptions from one test. Reading the rerun record Criterion Likely cause Where to look Only ever attempt 1 warm-up or ordering session fixtures Even across attempts real non-determinism clock, randomness, threads One worker only shared resource ports, tmp dirs, DB names Mixed exceptions racing its own setup fixture teardown order
The pattern in the attempt log identifies the class of bug before anyone reads the traceback.

Frequently Asked Questions

Does pytest-rerunfailures reset fixture state between retry attempts? Only function-scoped fixtures are recreated per rerun attempt. Module, class, and session-scoped fixtures persist across retries and carry mutated state forward. To force a full reset, convert fixtures to function scope or clear mutable state in a yield-based teardown.

How do I stop pytest-rerunfailures from masking genuine failures? Keep --reruns low (2-3), add --reruns-delay to space retries, and log failure signatures in a pytest_runtest_makereport hook. Alert on identical failure hashes recurring across pipeline runs so persistent logical bugs are escalated rather than silently retried.

Can I combine pytest-rerunfailures with pytest-xdist? Yes, but reruns are worker-local: a test that fails on worker A retries only on worker A. Shared databases, file locks, or ports can cause cascading failures across workers, so route flaky subsets to sequential execution or isolate per-worker resources.

From rerun to root cause A four-stage pipeline: reruns absorb the failure so CI stays green, the makereport hook records every failed attempt, the attempt log is grouped by node, worker and exception, and the resulting pattern points at a specific class of fix. From rerun to root cause rerun absorbs it CI stays green hook records it one line per attempt group the log node, worker, error fix the class not the symptom A test still flaky after two weeks of records is a test to quarantine, not to rerun.
Reruns are only acceptable as a bridge: the log is what turns the bridge into a repair.

Should quarantined tests still run in CI? Yes, but not on the critical path. Run them in a non-blocking job that publishes the attempt log, so the record keeps accumulating while the pipeline stays trustworthy. A quarantined test that stops running is a test that will be deleted six months later without anyone knowing whether the underlying bug was fixed — the log is what distinguishes a repaired test from a forgotten one, and it is also the evidence you need when arguing for the infrastructure change that actually removes the flakiness.

← Back to Optimizing Test Discovery