Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Hypothesis & Fuzzing

Why Hypothesis Shrinking Stalls and How to Fix It

Hypothesis reports a failure and the counterexample is a list of two hundred dictionaries. Somewhere in it is a single value that triggers the bug, and the shrinker — whose whole job is to find that value — gave up. A stalled shrink is almost never a Hypothesis problem; it is a signal that the strategy or the property is shaped in a way that hides information from the search.

Prerequisites

Solution

Three causes account for nearly every stall, and each has a structural fix.

1. Rejection sampling. assume() and .filter() discard whole examples. The shrinker generates a simpler candidate, the filter rejects it, and the shrinker learns nothing — so it keeps the larger input. Build the constraint into the draw instead:

Python
from hypothesis import strategies as st

# Stalls: most simplifications are rejected, so the counterexample stays large.
pairs = st.tuples(st.integers(), st.integers()).filter(lambda t: t[0] < t[1])

# Shrinks cleanly: every draw is valid by construction.
@st.composite
def ordered_pair(draw):
    low = draw(st.integers(min_value=-1000, max_value=999))
    high = draw(st.integers(min_value=low + 1, max_value=1000))
    return low, high

2. Draw order that hides structure. The shrinker simplifies draws roughly in the order they were made, so a strategy that draws contents before shape gives it nothing to pull on. Draw the size first, then the elements:

Python
@st.composite
def rows(draw):
    n = draw(st.integers(min_value=0, max_value=50))       # shrinks toward 0 first
    return [draw(st.integers(0, 10_000)) for _ in range(n)]

3. A non-deterministic property. Shrinking works by re-running the property against smaller inputs; if the property sometimes passes on an input that previously failed, the shrinker concludes the smaller input is fine and backs off. Hypothesis usually detects this and reports Flaky, but a property that depends on the clock, a shared cache or dict ordering can stall silently before that check fires.

Diagnosing a stalled shrink A decision diagram: if the property is non-deterministic fix that first, if the strategy filters heavily replace rejection with construction, and if the draw order puts contents before shape reorder the draws. Diagnosing a stalled shrink Why can the shrinker not simplify? property is flaky make it deterministic seed, clock, caches shrinking needs repeatability filters reject a lot construct instead no rejected candidates every draw is valid contents drawn first reorder the draws size, then elements structure shrinks first Hypothesis reports Flaky when it detects this, but not every case is detected.
Fix determinism before anything else: a flaky property makes every other shrinking improvement invisible.

Why this works

The shrinker operates on the sequence of choices a strategy made, not on the final value. It repeatedly proposes a lexicographically smaller choice sequence and re-runs the property; a proposal that still fails becomes the new best, and one that passes — or is rejected by a filter — is discarded. Two consequences follow directly. A filter that rejects most proposals starves the search of successful steps, and a property whose outcome varies makes previously-failing proposals look successful. Draw order matters because the sequence is simplified from the front: an early draw that controls size lets one successful step remove many elements at once.

One iteration of the shrinking loop A left-to-right loop: propose a simpler choice sequence, re-run the property, keep the candidate if it still fails, and repeat until no simplification succeeds within the budget. One iteration of the shrinking loop propose simpler lexicographic step re-run property must still fail keep or discard best-so-far repeat until no progress
Every filter rejection and every flaky pass looks identical to the loop: a step that did not work.

Measuring whether shrinking worked

--hypothesis-show-statistics reports the shrink phase directly, and the numbers say whether the search was productive:

Bash
$ pytest tests/test_totals.py --hypothesis-show-statistics -q
  - during shrink phase (2.31 seconds):
    - Typical runtimes: < 1ms
    - 412 shrinks succeeded, 1897 shrinks failed
  - Highest target scores: none

A high ratio of failed to successful shrinks with a large final counterexample is the signature of a blocked search. The same output reports invalid examples from the generate phase, which is the direct measure of how much rejection sampling is costing:

Bash
  - during generate phase (0.94 seconds):
    - 100 passing examples, 1 failing example, 863 invalid examples

Eight hundred invalid examples for a hundred valid ones means the strategy is rejecting nearly nine draws in ten. That number is both the runtime problem and the shrinking problem, and rebuilding the strategy fixes both at once.

Making the counterexample readable

A shrink that succeeds still produces an unreadable report when the strategy returns opaque objects. Hypothesis prints the repr of what the strategy produced, so a domain object without a meaningful repr yields <Order object at 0x7f...> and the reader learns nothing.

Two fixes. Give the object a faithful repr — a dataclass gets one for free — or have the strategy return the primitive draws alongside the object so the report shows the inputs that built it:

Python
@st.composite
def order_case(draw):
    quantity = draw(st.integers(min_value=0, max_value=100))
    price = draw(st.integers(min_value=0, max_value=10_000))
    # Returning the primitives makes the failing report self-explanatory.
    return {"quantity": quantity, "price": price, "order": Order(quantity, price)}

The same principle applies to st.builds over a class you do not control: wrap it so the report names the arguments rather than the object.

Rewriting a strategy that will not shrink

The general repair has three moves, applied in order. Working through a concrete example makes the pattern reusable.

Start with a strategy that stalls: it generates a list of order lines, filters out invalid combinations, and derives the total afterwards.

Python
from hypothesis import strategies as st

# Stalls: two filters and a derived value the shrinker cannot simplify.
bad = st.lists(
    st.tuples(st.integers(), st.integers()), min_size=1
).filter(lambda rows: all(q > 0 for q, _ in rows)) \
 .filter(lambda rows: sum(q * p for q, p in rows) < 10_000)

Move one: replace filters with bounded draws. Both filters express constraints that can be built into the draw ranges, so no candidate is ever rejected.

Move two: draw the shape before the contents. The list length becomes an explicit early draw, so the shrinker can shorten the list in one step rather than element by element.

Move three: return data whose repr explains the failure. A list of tuples prints acceptably; a list of domain objects would not.

Python
@st.composite
def order_lines(draw, max_lines: int = 8, budget: int = 10_000):
    n = draw(st.integers(min_value=1, max_value=max_lines))   # shape first
    remaining = budget
    rows = []
    for _ in range(n):
        qty = draw(st.integers(min_value=1, max_value=20))     # positive by construction
        max_price = max(1, remaining // qty)
        price = draw(st.integers(min_value=1, max_value=max_price))
        remaining -= qty * price                               # budget respected by design
        rows.append((qty, price))
    return rows

Every draw is valid, the shape shrinks first, and the counterexample is a short list of small integers. The statistics confirm it: invalid examples drop to zero and the shrink phase reports a handful of successful steps rather than thousands of failed ones.

The cost is that the strategy is longer and encodes domain rules. That is the right trade — those rules were previously expressed as filters, where they cost runtime and blocked the search, and they are now expressed as bounds, where they cost nothing.

Three moves, and what each unblocks A table of three strategy rewrites - replacing filters with bounded draws, drawing shape before contents, and returning readable data - with the shrinking problem each one removes. Three moves, and what each unblocks Criterion Removes Visible as bounded draws, no filter rejected candidates invalid examples drop to zero shape drawn first element-by-element shrinking shorter counterexamples readable return value unreadable reports the failing input is legible
The first two fix the search; the third fixes what you learn from it once it succeeds.

Edge cases and failure modes

  • Flaky errors are the honest version of a stall. When Hypothesis reports one, fix the non-determinism rather than suppressing the check.
  • A target() call changes the search but not the shrink. Targeted property-based testing guides generation toward a metric; it does not make a blocked shrink succeed.
  • Recursive strategies need explicit size bounds. st.recursive without max_leaves can produce structures whose shrink space is enormous, and the phase times out before it finishes.
  • Shrinking respects the deadline. A property with a per-example deadline near its actual runtime will have shrink attempts killed, which looks like a stall and is a timeout.
  • Stateful tests shrink the rule sequence, not just the data. A stalled state-machine shrink usually means one rule has a precondition that rejects most simplified sequences — see debugging RuleBasedStateMachine failures.
  • Large max_examples does not compensate. Budget helps a search that is making progress; it does nothing for one that cannot take a successful step. A final measurement habit: record the size of the reported counterexample, not just whether the test failed. A property whose counterexamples grow over releases is a property whose strategy is drifting toward rejection sampling — usually because constraints were added as filters rather than as bounds. Noticing that trend early is far cheaper than rewriting the strategy after it has stopped producing readable failures.
Counterexample size before and after removing filters A bar chart comparing the reported counterexample size for four strategy variants: two chained filters, one filter, bounded draws, and bounded draws with the shape drawn first. Counterexample size before and after removing filters two chained filters 312 elements one filter 138 elements bounded draws 28 elements bounded, shape first 6 elements Sizes from one strategy rewritten in four stages; the absolute numbers are illustrative.
Each removed rejection point makes the shrinker’s steps succeed more often, and the reported counterexample shrinks accordingly.

Frequently Asked Questions

Why is my counterexample a 300-element list? Because something blocked the shrinker: usually a filter that rejects the simpler candidates it tries, or a property that fails non-deterministically so a smaller failing input looks like a pass.

Does assume() hurt shrinking? Yes, when it rejects a large fraction of draws. Every rejected candidate is a shrink attempt that produced no information, so heavy use of assume turns shrinking into a random walk.

Can I make Hypothesis shrink harder? Increasing max_examples gives the shrinker more budget, but the usual problem is structural rather than budgetary. Rebuilding the strategy so invalid values cannot be drawn is far more effective than more attempts. Does target() help a stalled shrink? No. Targeted property-based testing biases generation toward inputs that maximise a metric, which finds different bugs; shrinking is a separate phase with its own rules. A stalled shrink is fixed in the strategy, not in the search objective.

Should I ever accept a large counterexample? Only briefly. A large counterexample still identifies a real bug, so fix the bug — but treat the size as a defect in the strategy and address it before the next failure, because the next one will be equally unreadable and may not be as easy to interpret.

← Back to Advanced Property-Based Testing