A RuleBasedStateMachine finds a bug that no example-based test would have: a sequence of eleven operations after which the cache reports a size the model disagrees with. The report is a wall of generated calls, and the instinct is to stare at it. There is a faster method, and it starts from the bottom of the output rather than the top.
Prerequisites
- hypothesis 6.90+ with
RuleBasedStateMachinefromhypothesis.stateful. - The machine design in stateful and model-based testing.
Solution
Hypothesis prints the failing run as executable Python. That output is the artefact — read it, then run it.
Falsifying example:
state = CacheMachine()
state.put(key='0', value=0)
state.put(key='1', value=0)
state.get(key='0')
state.put(key='2', value=0)
state.teardown()
Explanation: assertion failed in invariant size_never_exceeds_capacity
Read it from the bottom. The invariant names what broke; the last rule before it names when. Here the third put evicted an entry, but the earlier get refreshed recency in the real cache and not in the model — so the model evicted the wrong key.
Convert it into a plain regression test immediately, before fixing anything:
def test_lru_refresh_on_get_regression():
cache = LruCache(capacity=3)
cache.put("0", 0)
cache.put("1", 0)
cache.get("0") # refreshes recency: the step the model missed
cache.put("2", 0)
assert cache.keys() == ["1", "0", "2"]
That test is deterministic, fast, and survives every future change to the machine. The state machine's job was to find the sequence; keeping the sequence is your job.
Why this works
A state machine run is a sequence of rule invocations chosen by Hypothesis, with invariants evaluated after each one. When an invariant fails, everything before it succeeded — so the state was consistent until the last rule ran. That gives an exact attribution that a single large assertion at the end of a run cannot: the failure is localised to one operation applied to one known state. The printed sequence is generated from the same recorded choices, which is why it reproduces deterministically outside Hypothesis.
Making the model talk
When model and system disagree, the report shows the sequence but not the two states. Adding a __repr__ to the model, and printing it from the failing invariant, turns a fifteen-call reproduction into an immediate answer:
from hypothesis.stateful import RuleBasedStateMachine, invariant, rule
class CacheMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.real = LruCache(capacity=3)
self.model = {}
self.order = []
def __repr__(self):
return f"<model keys={self.order} real={self.real.keys()}>"
@invariant()
def keys_agree(self):
assert self.real.keys() == self.order, (
f"divergence: real={self.real.keys()} model={self.order}"
)
Putting the two sequences in the assertion message is the single highest-value line in a state machine. The failure then reads real=['1','0','2'] model=['1','2','0'], which names the operation that ordered them differently without any further investigation.
Two related habits help. Keep invariants small and numerous rather than one large one, so the name of the failing invariant is itself informative. And assert the cheap structural properties — sizes, key sets, monotonic counters — rather than deep equality, because a diff of two large structures is harder to read than a single mismatched count.
When the sequence will not shrink
A twenty-call counterexample means the shrinker could not remove steps, and there are two usual reasons.
Preconditions that reject shortened sequences. @precondition(lambda self: self.model) prevents a rule from running on an empty model, which is correct — but if most rules are gated this way, a shortened sequence becomes unrunnable and the shrinker discards it. Keep preconditions as permissive as correctness allows, and prefer rules that handle the empty case gracefully over rules that refuse to run.
State that leaks between runs. The shrinker replays candidate sequences in the same process. If the machine's __init__ does not fully reset the system under test — a module-level registry, a class attribute, a database that teardown did not clean — a shorter sequence behaves differently and the reduction fails. Implement teardown() on the machine, not in a fixture, so it runs after every sequence rather than once per test.
class ApiMachine(RuleBasedStateMachine):
def teardown(self):
# Runs after EVERY sequence, including every shrink candidate.
for order_id in self.created:
self.client.delete(f"/orders/{order_id}")
self.created.clear()
Instrumenting the machine for faster diagnosis
A state machine that prints nothing until it fails makes every failure a reconstruction exercise. Three cheap instruments turn the report into a narrative.
Log each rule as it runs. Hypothesis prints the final sequence, but seeing the run unfold — with the state after each step — shows where the divergence began rather than where it was noticed:
from hypothesis.stateful import RuleBasedStateMachine, rule
class CacheMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.log: list[str] = []
@rule(key=st.text(min_size=1, max_size=2), value=st.integers())
def put(self, key, value):
self.real.put(key, value)
self.model[key] = value
self.log.append(f"put({key!r}, {value}) -> {sorted(self.model)}")
def teardown(self):
if self.log and len(self.log) < 40:
print("\n".join(self.log)) # printed by pytest only on failure
Assert the cheap invariant often and the expensive one rarely. @invariant() runs after every rule, so a full deep-equality check between model and system on every step is slow enough to reduce the number of sequences explored. Check sizes and key sets every step, and full contents in teardown.
Name the bundle contents. When rules consume values produced by other rules, printing the bundle at each step turns v3 in the report into something meaningful — usually the exact aliasing bug the machine found.
The combination costs a few lines and converts the typical debugging session from "reconstruct fifteen calls by hand" into "read the log until the two states diverge".
Edge cases and failure modes
- Bundles print as
v1,v2placeholders. Those are references to values produced by earlier rules; the report is still executable because the variables are assigned in the printed code. - An exception inside a rule stops the sequence there. The report shows the rules up to and including the raising one, which is exactly the attribution you want — no invariant needed.
stateful_step_countbounds the sequence length. A bug that needs more operations than the limit will never be found; raise it rather than raisingmax_exampleswhen the bug is depth-related.- Invariants run on the initial state too, before any rule, so an invariant that assumes setup has happened fails immediately and confusingly.
- Machines are not thread-safe by design. A rule that spawns threads makes the sequence non-deterministic and the shrink useless; model concurrency as explicit rules instead.
- A
Flakyreport from a state machine means the same sequence produced different outcomes — almost always leaked state, and almost always fixed by a realteardown. One structural note about where machines belong in a suite. A state machine is slower than an ordinary test by one or two orders of magnitude, because each example is a sequence of operations and there are many examples. Mark them and exclude them from the pre-commit run, then give them a generous budget in a scheduled job — the same split recommended for expensive properties. A machine running twenty-five examples on every pull request is both too slow to be welcome and too shallow to find anything.
Frequently Asked Questions
How do I turn a state machine failure into a normal test? Copy the printed call sequence into a plain test function. Hypothesis prints it as executable Python, so pasting it produces a deterministic regression test that runs in milliseconds.
Why does the failure name an invariant rather than a rule? Because invariants run after every rule, so the first one to notice a broken state reports the failure. The rule that caused it is the last one printed before the invariant fired.
Why will my state machine failure not shrink? Usually a precondition that rejects most shortened sequences, or state that leaks between runs so a shorter sequence no longer reproduces. Both make the shrinker's candidates look healthy. Can I run a state machine under pytest-xdist? Yes — each machine test is an ordinary test item and distributes normally. The caveat is the same as for any test touching shared resources: the machine's system under test must be partitioned per worker, or two workers will drive the same database and produce failures that are artefacts of the parallelism rather than of the model.
What if the model is the thing that is wrong? That is a normal outcome and a useful one: a divergence proves the two implementations disagree, not which one is correct. Read the specification before changing either. A model corrected to match a buggy system is worse than no model, because every future run now agrees with the bug.
Related
- Stateful and model-based testing — designing the model these failures compare against.
- Modeling REST APIs as state machines — teardown and determinism for machines that talk to a service.
- Why Hypothesis shrinking stalls and how to fix it — the general form of an unshrinkable counterexample.
- Reproducing Hypothesis failures with @example — keeping the finding after the fix.
← Back to Stateful and Model-Based Testing