Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Hypothesis & Fuzzing

Reproducing Hypothesis Failures with @example

Hypothesis reports a failing input, you fix the code, and a week later the same class of bug reappears in a form nobody remembers testing. The counterexample was random, the run that found it is gone, and nothing in the repository records what was learned. Three mechanisms — the example database, @example, and @reproduce_failure — turn a one-off discovery into a permanent regression test, and they serve different purposes.

Prerequisites

Solution

Pin the example. The durable answer is an @example decorator carrying the shrunk input. It runs before any generated data on every execution, forever, and it is reviewable in a diff:

Python
from hypothesis import example, given, strategies as st

@given(st.integers(min_value=0), st.integers(min_value=0))
@example(0, 0)                       # the empty-basket bug, found 2026-03-14
@example(1, 2 ** 31)                 # the overflow that reached production
def test_total_never_negative(price, quantity):
    assert total(price, quantity) >= 0

The comment matters as much as the values. A pinned example without provenance looks like an arbitrary case and gets deleted in a cleanup; one that names the bug it encodes survives.

Replay while debugging. With print_blob=True, a failure prints a decorator that replays that exact example without regenerating anything — the fastest possible edit-run loop while fixing the code:

Python
from hypothesis import reproduce_failure

@reproduce_failure("6.92.1", b"AXicY2BgYGAAAAAFAAE=")
@given(st.integers(), st.integers())
def test_total_never_negative(price, quantity):
    ...

Delete it once the fix lands. The blob encodes Hypothesis's internal representation and does not survive a library upgrade, so it is a debugging aid rather than a regression test.

Let the database do the rest. Hypothesis writes every failing example to .hypothesis/examples/ and replays entries for that test first on the next run. Locally this is automatic; in CI it only works if the directory is cached between runs.

YAML
- uses: actions/cache@v4
  with:
    path: .hypothesis
    key: hypothesis-${{ github.ref }}-${{ github.run_id }}
    restore-keys: hypothesis-${{ github.ref }}-
Three mechanisms, three jobs A table of the three reproduction mechanisms - the example decorator, the reproduce_failure blob and the example database - describing what each stores, how long it survives, and what it is for. Three mechanisms, three jobs Criterion Stores Survives @example(...) explicit input values forever, in the repo @reproduce_failure(...) an encoded internal blob one library version the example database every recent failure while the cache lives
Only the first is a regression test; the other two are convenience layers over a search that is random by design.

Why this works

Hypothesis runs a test in phases: explicit examples first, then entries from the database for that test's identity, then generated data, then shrinking. @example inserts into the first phase, so it runs even when generation is disabled entirely. The database keys on a hash of the test's source and name, which is why renaming a test loses its history — and why the entries are pruned when the test disappears. @reproduce_failure bypasses all of it, decoding a serialized draw sequence and replaying exactly those choices, which is why it is precise and why it breaks across versions.

The order Hypothesis runs its phases in A left-to-right sequence of the phases: explicit examples from decorators, replayed entries from the example database, newly generated data, and finally shrinking of any failure found. The order Hypothesis runs its phases in explicit @example first reuse database entries generate new random data shrink minimise a failure phases=[Phase.explicit, Phase.reuse] gives a fast replay-only run for pre-merge checks.
Pinned examples always run first, which is what makes them a regression test rather than a lucky draw.

Making CI runs reproducible

Randomness is the point of property-based testing, but a pull-request pipeline benefits from stability. Two settings give it without giving up the search.

derandomize=True derives the seed from the test's identity, so the same test generates the same examples on every run. A pull request then fails or passes consistently, and a failure reproduces exactly when a colleague checks out the branch. The cost is that the pipeline stops exploring: it will never find a bug the fixed seed does not reach.

The usual arrangement runs both. Pull requests use a derandomised profile for stability; a scheduled job runs a randomised, high-example profile whose job is to explore, and any counterexample it finds is turned into an @example by the engineer who fixes it.

Python
from hypothesis import settings, Phase

settings.register_profile("ci", derandomize=True, max_examples=100, print_blob=True)
settings.register_profile("nightly", max_examples=2000, deadline=None)
settings.register_profile("replay", phases=[Phase.explicit, Phase.reuse])

replay is worth having as a third profile: it runs only pinned examples and database entries, in seconds, which makes it a good pre-commit hook.

Naming and keeping counterexamples

A pinned example is documentation, and it decays like documentation unless it is written for a reader.

State what the example encodes, not what the values are — # regression: quantity at INT32_MAX overflowed the total beats # large quantity. Keep the example next to the property it falsifies rather than in a separate regression module, so a change to the property surfaces the example in the same diff. And when a fix makes an example structurally impossible — the parameter no longer exists — delete it deliberately rather than adapting it, because an adapted example no longer encodes the bug it was created for.

For teams that want the history without the decorators, @example accepts .via("discovered failure") metadata, and Hypothesis's explicit phase reports which pinned examples ran. That is enough to answer "what have we learned about this function" during a review, which is the question an accumulated set of examples is actually answering.

Sharing a database across a team

The example database is per-machine by default, which means a counterexample found on CI is not replayed on a developer's laptop and vice versa. For teams that want the search to accumulate rather than restart, Hypothesis supports pluggable database backends.

Python
# conftest.py
from pathlib import Path
from hypothesis.database import (
    DirectoryBasedExampleDatabase, MultiplexedDatabase, ReadOnlyDatabase,
)
from hypothesis import settings

local = DirectoryBasedExampleDatabase(".hypothesis/examples")
shared = ReadOnlyDatabase(DirectoryBasedExampleDatabase("/mnt/team-hypothesis"))

settings.register_profile(
    "team",
    database=MultiplexedDatabase(local, shared),   # write locally, read from both
)

MultiplexedDatabase reads from every backend and writes to the writable ones, and wrapping the shared store in ReadOnlyDatabase means a developer's experiment cannot pollute it. The shared directory can be a network mount, a synced bucket, or a git repository updated by the nightly job — the interface is small enough that a custom backend is a few dozen lines.

Two cautions. Database entries are keyed on the test's source and name, so a refactor invalidates them silently — pinned @example decorators do not have this problem, which is why they remain the durable record. And the database can hold inputs derived from real data if strategies were built from production samples; treat a shared store with the same care as a cassette or a fixture file.

For most teams, caching the database in CI and pinning important counterexamples with @example is sufficient. Reach for a shared database when the search is genuinely expensive — a nightly run over thousands of examples whose findings should reach every branch immediately.

Edge cases and failure modes

  • Renaming a test loses its database history. The key includes the name, so a rename starts from scratch; pinned examples are unaffected, which is another argument for them.
  • @example values are not shrunk. They run exactly as written, so an unnecessarily large pinned example stays large in the output forever — pin the shrunk form.
  • A pinned example that no longer applies fails loudly. That is correct behaviour, and the fix is a deliberate deletion rather than a pytest.skip.
  • The database directory can grow. Entries are pruned per test, but a suite with thousands of properties accumulates; a size cap on the CI cache is sufficient.
  • derandomize and the database interact. With both active, a run replays known failures and then generates the same examples every time — reproducible, but no new coverage until the seed changes.
  • Blobs are version-locked. A @reproduce_failure left in the repository breaks the suite on the next Hypothesis upgrade, with an error that names the version mismatch. A note on review etiquette for pinned examples: when a pull request adds an @example, the reviewer should be able to answer "what bug does this encode" from the diff alone. That means the comment matters more than the values, and a pinned example without one is worth a review comment — six months later, nobody can distinguish a hard-won counterexample from a value somebody typed while experimenting, and the indistinguishable ones get deleted during cleanups.
What a pinned example should carry A stack of four attributes a good pinned example has: the shrunk values, a comment naming the bug, a date or ticket reference, and placement next to the property it falsifies. What a pinned example should carry the shrunk values the minimal form, not the original failure a comment naming the bug what it encodes, not what it contains a date or ticket provenance survives the people placement by the property the example and its claim stay together
Four small conventions are the difference between an accumulating regression record and a list of magic numbers.

Frequently Asked Questions

Why does a Hypothesis failure not reproduce on the next run? Because generation is random by default and the example database — which replays known counterexamples first — is not present. Either the directory was not cached in CI, or the failure came from a different test whose entry was pruned.

Should I commit the .hypothesis directory? No. Cache it in CI and keep it out of version control. Pin the specific counterexamples you care about with @example, which is explicit, reviewable and survives database loss.

What is reproduce_failure and when should I use it? It is a decorator carrying a base64 blob that replays one exact example. Use it while fixing a bug locally and delete it afterwards, because the encoding is tied to the Hypothesis version. Do pinned examples slow the suite down? Negligibly. Each @example is one extra execution of the property, run before generation begins, so ten pinned examples cost ten executions against a default budget of a hundred. That is a rounding error against the value of never re-losing a counterexample.

← Back to Hypothesis Framework Fundamentals