Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Debugging & Performance

Bisecting Test Order Dependencies

Order-dependent tests are the most frustrating kind of flaky test, because nothing about them is random once you know the order. Test B passes alone, passes in the usual order, and fails whenever test A happens to run first. A leaves something behind — a module-level cache it populated, an environment variable it set directly, a patched function it never restored, a row in a shared database — and B trips over it. Randomised ordering, parallel runs and newly added tests all rearrange the sequence, so the failure appears and disappears for reasons that look like chance.

The way out is mechanical. Get a reproducible order that fails. Confirm the victim passes alone. Then bisect the tests that ran before it: run half of them, then the victim, and keep whichever half still produces the failure. With a few hundred predecessors, eight or nine runs find the polluter. The tool detect-test-pollution automates exactly this.

Prerequisites

Solution

Bash
# 1. Reproduce the failing order. pytest-randomly prints the seed:
#    Using --randomly-seed=2918476
pytest -p randomly --randomly-seed=2918476 -v 2>&1 | tee run.log

# 2. Save the node ids that ran before the victim, in order.
VICTIM="tests/test_billing.py::test_invoice_total"
grep -E ' (PASSED|FAILED|ERROR|SKIPPED)' run.log | awk '{print $1}' \
  | awk -v v="$VICTIM" '$0 == v {exit} {print}' > before.txt

# 3. Confirm the victim passes alone.
pytest "$VICTIM"

# 4. Let detect-test-pollution bisect the predecessors automatically.
detect-test-pollution --failing-test "$VICTIM" --testids-file before.txt
#   ...
#   the polluting test is: tests/test_settings.py::test_override_currency
Bash
# Manual bisection, when you want to see each step.
split -n l/2 before.txt half_
pytest -p no:randomly $(cat half_aa) "$VICTIM"   # fails? polluter is in half_aa
pytest -p no:randomly $(cat half_ab) "$VICTIM"   # else it is in half_ab — repeat
Bisecting the tests that ran before the victim Four rows show a bisection. The first row has 256 predecessor tests before the victim. Each subsequent row keeps the half that still makes the victim fail: 128, then 64, and so on, until after eight rounds a single polluting test remains. The number of runs grows with the logarithm of the number of predecessors. Halve until one polluter remains 256 tests victim 128 64 … 1 polluter log2(256) = 8 runs, each only as long as its half
Each round keeps the half that still breaks the victim, so the number of runs grows with the logarithm of the suite size.

Why this works

An order dependency means there exists some test P such that running P before the victim V makes V fail. If the dependency comes from a single polluter — by far the most common case — then among any split of the predecessors into two halves, exactly one half contains P, and running that half followed by V reproduces the failure. Keeping that half and repeating halves the search space each time, so 256 candidates need about eight rounds and 4,000 need about twelve. Each round runs only the candidate half, so later rounds are fast.

-p no:randomly in the manual steps matters: once you have a failing sequence, you want each bisection run to execute exactly the listed tests in exactly the listed order. pytest runs explicitly listed node ids in the order given when randomisation is off.

detect-test-pollution performs the same algorithm, with extra care: it first confirms the victim passes alone and fails after the full list, and it handles collection of node ids for you. When it finishes, it prints a single polluting test id.

Finding what the polluter leaves behind

Knowing the polluter is half the job; the other half is knowing what state it leaks. Read the polluter looking for side effects that outlive it:

  • Environment variables set with os.environ[...] = ... or os.putenv.
  • Module attributes assigned directly — settings.CURRENCY = "EUR" — or functions replaced without monkeypatch.
  • Caches filled as a side effect: functools.lru_cache on configuration loaders, class-level registries, memoised clients.
  • Process state: the working directory changed with os.chdir, the default timezone or locale changed, sys.path extended.
  • External state: database rows committed outside a rolled-back transaction, files written outside tmp_path, messages left on a queue.

A quick confirmation is to diff state around the polluter. Snapshot dict(os.environ), os.getcwd() and the relevant module attributes before and after running it alone; whatever differs is the leak. The fix is almost always to route the change through a fixture that restores it — monkeypatch.setenv, monkeypatch.setattr, monkeypatch.chdir, a cache-clearing teardown, or a transaction that rolls back — so the test can make whatever change it needs without leaving it for the next one.

Common leaks and their restoring replacements A table maps common leaks to fixes. Setting os.environ directly is replaced by monkeypatch.setenv. Assigning module attributes is replaced by monkeypatch.setattr. os.chdir is replaced by monkeypatch.chdir. Filled lru_cache is cleared in fixture teardown with cache_clear. Committed database rows are avoided with a transaction that rolls back. Every leak has a restoring twin leak restoring replacement os.environ["X"] = … monkeypatch.setenv("X", …) settings.CURRENCY = "EUR" monkeypatch.setattr(settings, "CURRENCY", "EUR") os.chdir(path) monkeypatch.chdir(path) @lru_cache filled as a side effect load_config.cache_clear() in teardown committed database rows per-test transaction, rolled back
Fix the polluter, not the victim: the victim was relying, correctly, on a clean starting state.

Preventing order dependencies from accumulating

Bisection finds one polluter at a time. Keeping new ones out is cheaper, and three habits do most of the work.

Randomise order everywhere. pytest-randomly shuffles modules, classes and tests on every run and reseeds random per test. Installed as a development dependency, it makes order dependencies surface on the laptop of whoever introduces them, usually within a few runs, instead of months later in CI. The seed line at the top of the output makes every failure reproducible, so a shuffled failure costs one extra command rather than an investigation.

Run each test in isolation occasionally. A scheduled job that runs every test file on its own — or every test, for small suites — catches the opposite problem: tests that pass only because an earlier test set something up. Those are order dependencies too, just with the roles reversed, and random ordering finds them less reliably because the helpful predecessor often still runs first by chance.

Make global state visible. An autouse fixture that snapshots os.environ, the working directory and a short list of known application globals before each test, and compares after, turns silent leaks into immediate failures naming the test that leaked and the key that changed. It is a few lines of code and removes the need for bisection in most future cases, because the polluter reports itself.

Python
@pytest.fixture(autouse=True)
def _no_leaked_state():
    env, cwd = dict(os.environ), os.getcwd()
    yield
    assert dict(os.environ) == env, "test leaked environment changes"
    assert os.getcwd() == cwd, "test changed the working directory"
Three guards against order dependencies Three cards describe guards. Random ordering with pytest-randomly surfaces polluters locally with a reproducible seed. Periodic isolated runs catch tests that only pass because of a helpful predecessor. A state-snapshot fixture fails the leaking test itself, naming what changed, so no bisection is needed. Catch the next polluter without bisecting random order pytest-randomly found locally, seed printed isolated runs each file on its own catches helpful predecessors state snapshot autouse before/after check the polluter fails itself
With a snapshot fixture in place, the next leak fails in the test that caused it, not in some unrelated victim.

A worked example: the currency that stuck

In one suite, test_invoice_total failed roughly one run in six under random ordering, always asserting that a total of 12.50 was 1250. Re-running with the printed seed reproduced it every time, and the test passed on its own. detect-test-pollution took the 312 tests that preceded it and, after nine rounds, named tests/test_settings.py::test_override_currency.

The polluter was three lines long. It assigned settings.CURRENCY = "JPY" directly to check that yen amounts render without decimals, asserted on the rendering, and ended. Japanese yen has no minor unit, so every later test in the same process that formatted money did so without decimals — including the invoice test, which then compared a string without a decimal point against its expectation.

The fix replaced the assignment with monkeypatch.setattr(settings, "CURRENCY", "JPY"), which restores the original value at teardown. The invoice test was left untouched, because it was correct: it assumed the default currency, which is what the application uses. After the change, the same seed passed, and a hundred further random orders passed too. The state-snapshot fixture above was added in the same change, extended to compare settings.CURRENCY, so that the next test to assign settings directly fails immediately and names itself. The whole investigation, from first reproduction to merged fix, took under an hour — most of it reading the polluter — because the bisection itself was automatic.

Edge cases and failure modes

  • Two polluters needed together. Occasionally the victim fails only after two tests both run. Bisection stalls because neither half alone reproduces; run the halves combined in different pairs, or use a delta-debugging approach.
  • The victim pollutes itself. Running the same test twice in one process fails the second time. pytest --count=2 from pytest-repeat on the victim alone detects this.
  • Order-dependent collection. Some leaks happen at import time, during collection, not in test bodies. Bisection over test ids still finds the module whose import pollutes.
  • Failures only under xdist. Replay the worker's sequence serially first; see the xdist guide below.
  • Fixing the victim instead. Making the victim reset global state hides the polluter and leaves every other test exposed. Fix the source.

Frequently Asked Questions

What is a test order dependency? A test whose outcome depends on which tests ran before it in the same process, because an earlier test changed shared state — a global, an environment variable, a cache, the working directory, a database row — and did not restore it.

How do I reproduce a random-order failure? pytest-randomly prints the seed at the top of each run. Re-run with -p randomly --randomly-seed=<seed> to get the same order, then save the failing sequence of node ids.

Is there a tool that finds the polluting test automatically? Yes. detect-test-pollution takes the failing test and a list of candidate tests, then bisects the candidates automatically until it finds the single test that makes the failing one fail when run before it.

← Back to Debugging Tests in CI and Containers