Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Hypothesis & Fuzzing

Shrinking Long Rule Sequences into Readable Repros

A state machine failure is only as useful as the sequence it prints. When Hypothesis's shrinker does its job, a failure found after two hundred random operations is reported as five: put('a'), put('b'), get('a'), put('c'), get('b'). That sequence is a bug report and a regression test in one. When shrinking struggles, the output is forty steps of mostly irrelevant operations with the real cause buried somewhere in the middle, and debugging starts with an afternoon of manual deletion.

The difference is rarely luck. Shrinking works by deleting and simplifying steps and checking whether the failure still happens. Anything that makes deletion invalid — a rule that depends on an object an earlier rule created, a precondition that blocks the shortened sequence, a check that only fires at the very end — limits how far it gets. Designing the machine with shrinking in mind produces short counterexamples reliably.

Prerequisites

Solution

Python
# Before — the check runs once, at teardown, so every step matters to the failure.
class AccountsMachine(RuleBasedStateMachine):
    ...
    def teardown(self):
        assert self.ledger.balance_total() == sum(self.model.values())
Python
# After — invariants detect divergence at the step that caused it,
# and readable reprs make the printed sequence self-explanatory.
from dataclasses import dataclass

@dataclass(frozen=True)
class AccountRef:
    name: str
    def __repr__(self):
        return f"acct({self.name!r})"

class AccountsMachine(RuleBasedStateMachine):
    accounts = Bundle("accounts")

    @rule(target=accounts, name=st.sampled_from("abcd"))
    def open(self, name):
        ref = AccountRef(name)
        self.ledger.open(ref.name)
        self.model.setdefault(ref.name, 0)
        return ref

    @rule(src=accounts, dst=accounts, amount=st.integers(0, 100))
    def transfer(self, src, dst, amount):
        ok = self.ledger.transfer(src.name, dst.name, amount)
        if ok:
            self.model[src.name] -= amount
            self.model[dst.name] += amount

    @invariant()
    def totals_match(self):
        assert self.ledger.balance_total() == sum(self.model.values())

    @invariant()
    def per_account_match(self):
        for name, bal in self.model.items():
            assert self.ledger.balance(name) == bal, name
Plain text
Falsifying example:
state = AccountsMachine()
a1 = state.open(name='a')
state.transfer(amount=1, dst=a1, src=a1)
state.teardown()
From a long failing run to a minimal sequence A long bar of forty rule steps represents the first failing run. Shrinking deletes steps that are not needed for the failure and simplifies arguments, producing a short bar of three steps: open account a, then transfer one unit from a to itself, which exposes a self-transfer bug. Forty steps found it; three explain it found 40 steps: opens, deposits, transfers, closes… delete steps · simplify arguments shrunk open('a') transfer(a → a, amount=1) invariant fails The minimal sequence names the bug: self-transfers double-count.
Shrinking removes everything the failure does not need and simplifies what remains: account names, amounts, the order of steps.

Why this works

Hypothesis shrinks the underlying choices a run made rather than the rules directly. A step is a block of choices — which rule, which arguments — and the shrinker tries deleting blocks, reordering them, and simplifying the values inside them. Each candidate is re-run from scratch. If the failure still happens, the candidate becomes the new best; if not, the shrinker tries something else.

Two things make candidates fail for the wrong reason. The first is late detection: when the only assertion is at teardown, the failure depends on the accumulated state after every step, so removing an irrelevant step can change the final totals enough to hide the bug. Invariants that run after each step detect the divergence as soon as it appears, so everything after the causal step becomes deletable, and the causal step's predecessors only need to set up the state it needs. The second is dependency: a transfer that draws from accounts cannot survive the deletion of the open that created its account, so shrinking must keep the open — correctly — but a machine where every step depends on many earlier ones leaves the shrinker very little it may delete.

Designing rules that shrink well

  • Prefer invariants to end-of-run checks. Anything that can be checked after every step should be. Keep teardown for cleanup, not assertions.
  • Keep preconditions coarse. @precondition(lambda self: self.accounts_open) is fine; preconditions that depend on exact balances make many shortened sequences invalid and stall shrinking.
  • Let rules be no-ops rather than invalid. A transfer that returns False when funds are insufficient keeps the sequence valid when an earlier deposit is removed. A rule that calls assume on balance makes the shortened sequence invalid instead.
  • Use a small value space. sampled_from("abcd") for names and small integer ranges for amounts let the shrinker simplify towards 'a' and 0 or 1 quickly.
  • Give objects readable reprs. The printed sequence uses repr, and acct('a') explains itself where <AccountRef object at 0x7f…> does not.
What blocks and what enables shrinking Two columns compare machine designs. Designs that block shrinking: assertions only at teardown, fine-grained preconditions, rules that call assume, large value spaces. Designs that enable shrinking: invariants after each step, coarse preconditions, rules that become no-ops when inapplicable, small value spaces with readable reprs. Design choices that decide how far shrinking gets stalls shrinking assertions only in teardown preconditions on exact state assume() inside rules unbounded names and amounts helps shrinking invariants after every step coarse preconditions inapplicable rules become no-ops small value spaces, readable reprs
Every item on the left makes more shortened sequences invalid; every item on the right keeps them valid, so they can be tried.

Turning the output into a regression test

Once the bug is understood and fixed, the shrunk sequence should live on as a plain test. The Hypothesis database will replay it for a while, but databases get cleared, and a replayed failure says nothing to a reader about what it protects. A plain test does:

Python
def test_self_transfer_does_not_change_balance():
    ledger = Ledger()
    ledger.open("a")
    ledger.deposit("a", 10)
    assert ledger.transfer("a", "a", 1)
    assert ledger.balance("a") == 10

The translation is mechanical — each printed step becomes the corresponding call — with the assertion taken from whichever invariant failed. Naming the test after the behaviour rather than the ticket makes it self-documenting, and because it no longer depends on the machine, it survives refactors of the machine's rules. Keep the machine too: the plain test pins one sequence, and the machine keeps searching for the next.

When shrinking still leaves too much: manual reduction

Even a well-designed machine occasionally produces a counterexample of fifteen or twenty steps where only a handful look relevant. Before reading them all, it is worth a few minutes of systematic reduction by hand, because the shrinker's limits are specific and a human can step past them.

Start from the translated plain test, not the machine. Delete steps from the end first, one at a time, re-running after each deletion: the failure depends on the state at the failing step, so steps after it are irrelevant by definition, and steps just before it are the most likely to be causal. Then work backwards, deleting each earlier step and keeping the deletion if the test still fails. When a deletion makes the test error rather than fail — a transfer referencing an account that is no longer opened — replace the step with the simplest one that satisfies the dependency rather than keeping the original.

Next, simplify arguments. Replace each amount with 0 or 1, each name with 'a', and each collection with an empty one, keeping a change only if the failure survives. This is the step the shrinker does well for simple values but struggles with when values are tied together — two accounts that must share a name, an amount that must equal an earlier balance — because it simplifies one value at a time and breaks the relationship.

What remains usually points at the cause directly. It is also a strong hint about how to improve the machine: if the manual reduction removed a step the shrinker could not, the reason is usually a dependency or precondition that could be relaxed, and relaxing it makes the next failure shrink properly on its own. Over time, a machine that has been through a few rounds of this produces counterexamples that need no manual work at all, which is the point: the effort spent on one stubborn failure pays off for every failure after it, and the whole team gets short, readable reproductions without having to know how the shrinker works.

Manual reduction order Three stages are shown left to right. First, delete steps from the end, since anything after the failing step is irrelevant. Second, delete earlier steps one at a time, replacing any that are needed for dependencies with the simplest substitute. Third, simplify arguments towards zero, one, a and empty collections. Finishing what the shrinker started 1 · trim the tail steps after the failure cannot matter 2 · delete backwards keep deletions that still fail; substitute needed setup 3 · simplify values 0, 1, 'a', [] where the failure survives Whatever you removed by hand hints at a constraint to relax in the machine.
Manual reduction on the plain test is quick, and it doubles as a diagnosis of why automatic shrinking stopped.

Edge cases and failure modes

  • Shrinking takes minutes. Long runs with expensive steps shrink slowly. Lower stateful_step_count for pull-request runs and reproduce long nightly failures with the database locally.
  • Flaky shrinking. If the system under test has hidden state — a module-level cache, a real clock — shrinking can see the failure disappear and reappear. Reset all state in __init__ and inject time.
  • Readable but misleading reprs. A repr that hides a field the bug depends on sends debugging in the wrong direction. Include every field that affects behaviour.
  • Bundles and deletion. Values in a Bundle that are never drawn still keep their creating step alive if later steps might reference them; consumes and fewer bundle-producing rules help.
  • Nondeterministic rule selection in the code under test. If the system itself uses randomness, seed it from the machine so the replayed sequence behaves identically.

Frequently Asked Questions

Why is my state machine counterexample still fifty steps long? Usually because shrinking cannot delete steps without making the sequence invalid — a later rule depends on a value an earlier rule created, or a precondition blocks the shortened sequence. Checking the state earlier with invariants and reducing dependencies between rules both let Hypothesis remove more steps.

How do I turn a state machine failure into a regression test? Copy the printed step sequence into a plain test function that calls the same operations in order, or add it with @example-style replay by keeping the Hypothesis database. The plain test is clearer and survives refactors of the machine.

Does increasing stateful_step_count help or hurt shrinking? Longer runs find deeper bugs but give shrinking more to remove. Keep the default for everyday runs and raise it in a nightly profile; shrinking still works, it just takes longer on longer sequences.

← Back to Stateful and Model-Based Testing