Advanced Property-Based Testing with Hypothesis

As test suites mature, example-based assertions struggle to cover edge cases in complex data transformations, distributed state machines, and boundary-heavy algorithms — the failure mode is a green suite that still ships off-by-one errors in temporal calculations, race conditions in state transitions, and silent corruption in serialization layers. Advanced property-based testing closes that gap by validating system invariants across mathematically generated input spaces. Building on the property-based and fuzz testing foundations, this guide moves from basic randomization to deterministic, production-grade generation: composite strategies, stateful machines, shrinking control, and hybrid fuzzing.

Prerequisites

• Python 3.10+ — 3.9 reached end-of-life in October 2025; modern type hints (typing.Annotated, typing.Protocol) streamline strategy inference.
• hypothesis>=6.100 — stable Phase.explain, report_multiple_bugs, and stateful APIs.
• pytest>=8.0, pytest-xdist for parallel execution, pytest-cov for coverage tracking.
• Comfort with @given, built-in strategies (st.integers(), st.text(), st.lists()), and pytest fixtures — see the Hypothesis framework fundamentals if any of those are unfamiliar.

Core concept

Two ideas separate advanced work from basic @given usage. First, valid-by-construction generation: instead of generating arbitrary data and discarding invalid examples with .filter(), you construct strategies that natively produce only valid states, keeping the rejection rate low and shrinking deterministic. Second, stateful modeling: many systems are sequences of operations over mutable state, so you model them as a RuleBasedStateMachine and let Hypothesis search the space of operation orderings for an invariant violation — the dedicated treatment of @rule, @initialize, and Bundle lives in stateful and model-based testing.

Step-by-step implementation

Step 1 — Compose valid-by-construction strategies

Naive filtering with st.integers().filter(lambda x: x > 0) generates arbitrary integers, discards invalid ones, and retries. Once rejection climbs above ~20%, shrinking degrades exponentially and tests stall. Instead, use @st.composite with conditional branching so every generated object is structurally valid. The full mechanics — st.builds, type registration, and find() — are in generating custom strategies with hypothesis.strategies, and when one field must be drawn from a value produced by another, composing strategies with flatmap and @composite covers the draw-time chaining.

import datetime
from dataclasses import dataclass
from typing import Literal
from hypothesis import given, settings, assume
from hypothesis import strategies as st

@dataclass
class Transaction:
    transaction_id: str
    amount: float
    currency: Literal["USD", "EUR", "GBP"]
    timestamp: datetime.datetime
    status: Literal["pending", "completed", "failed"]

@st.composite
def valid_transactions(draw: st.DrawFn) -> Transaction:
    currency = draw(st.sampled_from(["USD", "EUR", "GBP"]))
    status = draw(st.sampled_from(["pending", "completed", "failed"]))
    # Valid-by-construction: completed transactions must carry a positive amount
    low = 0.01 if status == "completed" else 0.0
    amount = draw(st.floats(min_value=low, max_value=1_000_000.0, allow_nan=False))
    # Fixed bounds keep the strategy deterministic — never call datetime.now() here
    ts = draw(st.datetimes(min_value=datetime.datetime(2020, 1, 1),
                           max_value=datetime.datetime(2030, 1, 1)))
    assume(amount >= low)  # cheap guard for the rare float rounding edge
    return Transaction(
        transaction_id=f"TXN-{draw(st.text(min_size=8, max_size=12, alphabet='0123456789ABCDEF'))}",
        amount=amount, currency=currency, timestamp=ts, status=status,
    )

@given(valid_transactions())
@settings(max_examples=200)
def test_transaction_invariants(txn: Transaction) -> None:
    assert txn.amount >= 0.0
    if txn.status == "completed":
        assert txn.amount > 0.0  # business invariant

Constructing objects from constrained primitives lets the shrinking algorithm reduce failing examples to minimal counterexamples in milliseconds rather than minutes, because every candidate it explores is already valid. Note the fixed st.datetimes bounds: a strategy that reaches for datetime.now() becomes non-reproducible, the same class of leak that controlling time and randomness in tests removes from example-based suites — generation must be a pure function of the seed.

Step 2 — Model stateful systems with RuleBasedStateMachine

Stateless tests validate pure functions; stateful tests validate sequences of operations over mutable state. Declare rules as transitions, gate them with precondition, and assert system-wide guarantees with @invariant(). Hypothesis generates operation sequences and re-checks invariants after every step.

from typing import Any
from hypothesis.stateful import RuleBasedStateMachine, rule, invariant
import hypothesis.strategies as st

class TransactionalKVStore:
    def __init__(self) -> None:
        self._data: dict[str, Any] = {}
        self._buffer: dict[str, Any] = {}
        self._in_tx = False

    def begin(self) -> None:
        self._in_tx, self._buffer = True, {}

    def put(self, key: str, value: Any) -> None:
        (self._buffer if self._in_tx else self._data)[key] = value

    def commit(self) -> None:
        if self._in_tx:
            self._data.update(self._buffer); self._buffer.clear(); self._in_tx = False

    def rollback(self) -> None:
        if self._in_tx:
            self._buffer.clear(); self._in_tx = False

class KVStateMachine(RuleBasedStateMachine):
    def __init__(self) -> None:
        super().__init__()
        self.store = TransactionalKVStore()
        self.committed: dict[str, Any] = {}   # model of what MUST survive

    @rule()
    def begin_tx(self) -> None:
        self.store.begin()

    @rule(key=st.text(min_size=1, max_size=8), value=st.integers())
    def put_value(self, key: str, value: int) -> None:
        self.store.put(key, value)
        if not self.store._in_tx:
            self.committed[key] = value

    @rule()
    def commit_tx(self) -> None:
        if self.store._in_tx:
            self.committed.update(self.store._buffer)
        self.store.commit()

    @rule()
    def rollback_tx(self) -> None:
        self.store.rollback()

    @invariant()
    def committed_data_never_lost(self) -> None:
        for key, value in self.committed.items():
            assert self.store._data.get(key) == value

TestKVStore = KVStateMachine.TestCase

The committed dict is a model — an independent re-implementation of the contract — and the invariant checks the real store against it. Edge case: rules without a precondition may fire in illegal orders (a commit before any begin), which is exactly the kind of sequence you want explored, provided the implementation tolerates it.

Step 3 — Control shrinking and analyze counterexamples

When a stateful or composite test fails, the shrinking algorithm minimizes the failing input. Complex traces can produce opaque counterexamples, so capture every violation and replay deterministically.

from hypothesis import settings, given, strategies as st

@settings(report_multiple_bugs=True)   # surface all invariant violations, not just the first
@given(st.lists(st.integers(), max_size=10))
def test_with_full_report(xs: list[int]) -> None:
    assert sum(xs) == sum(reversed(xs))

On failure, Hypothesis prints a @reproduce_failure(...) decorator encoding the exact seed and trace. Copy it into an isolated unit test to verify a fix without re-running the full generation cycle. During initial isolation, restrict phases with @settings(phases=[Phase.generate]) to skip shrinking and avoid CI timeouts, then re-enable shrinking for production runs.

Step 4 — Bridge into fuzzing across C-extension boundaries

Pure Python property tests cannot safely exercise C-extensions or memory-managed libraries. Serialize Hypothesis-generated inputs into byte buffers and feed them to a coverage-guided fuzzer such as Atheris to catch segmentation faults and undefined behavior.

import struct
from hypothesis import given, settings, Phase
import hypothesis.strategies as st

def native_parse_payload(data: bytes) -> None:
    if len(data) < 4:
        raise ValueError("header too short")
    length = struct.unpack_from(">I", data, 0)[0]
    if length > len(data) - 4:
        raise BufferError("payload length mismatch")

@given(st.binary(min_size=4, max_size=256))
@settings(max_examples=500, phases=[Phase.generate])
def test_native_boundary(data: bytes) -> None:
    try:
        native_parse_payload(data)
    except (ValueError, BufferError):
        pass  # documented validation errors
    except Exception as exc:  # anything else is a boundary violation
        raise AssertionError(f"native boundary violation: {exc}") from exc

In a real Atheris setup, atheris.Setup(sys.argv, native_parse_payload); atheris.Fuzz() registers the same target for coverage-guided mutation. Enforce execution limits on native calls with signal.alarm() or threading.Timer. This pattern pairs naturally with the isolation techniques in autospec strict mocking when the target has impure dependencies.

Step 5 — Register CI profiles and parallelize

Scale max_examples by pipeline stage and isolate the database under pytest-xdist.

import os
from hypothesis import settings, Phase, Verbosity

settings.register_profile("ci_pr", max_examples=50, deadline=500,
                          phases=[Phase.generate, Phase.shrink, Phase.explain],
                          verbosity=Verbosity.quiet, database=None)
settings.register_profile("ci_nightly", max_examples=1000, deadline=2000,
                          database=".hypothesis/examples")
settings.load_profile("ci_pr" if os.getenv("CI") else "ci_nightly")

Under pytest -n auto, the bundled hypothesis.extra.pytestplugin appends a worker ID to each database path, preventing SQLite locking. The full latency-reduction playbook is in reducing Hypothesis test execution time.

Verification

Confirm the setup behaves before trusting it in CI:

• Run pytest --hypothesis-show-statistics. Each property should report a low rejection rate (aim under 15%) and a healthy Generate/Shrink split. A high "events" count of filtered draws signals an over-constrained strategy.
• Force a failure (negate an assertion) and confirm Hypothesis shrinks to a minimal counterexample and prints a @reproduce_failure decorator.
• For the state machine, inspect the printed trace on failure — it should be the shortest call sequence that breaks the invariant, with arguments already minimized.
• Run twice with the same seed (pytest --hypothesis-seed=0) and confirm identical generation, proving determinism.

Troubleshooting

Choosing properties that actually catch bugs

The hardest part of property-based testing is not the API — it is deciding what to assert when the input is unknown. Four property shapes cover most real code, and reaching for them in order is faster than inventing one from scratch.

Round trips. Anything with an encode and a decode, a serialise and a parse, a write and a read, has a property for free: decoding an encoded value returns the original. Round trips catch the largest class of real defects because they exercise both directions against arbitrary data.

from hypothesis import given, strategies as st
from myapp.wire import encode, decode

@given(st.dictionaries(st.text(min_size=1), st.integers()))
def test_encode_decode_round_trip(payload):
    assert decode(encode(payload)) == payload      # no example list to maintain

Invariants. Some statement is true of the output regardless of input: a sorted list is ordered and has the same multiset of elements; a fee is never negative; a rendered template contains no unresolved placeholders. Invariants are weaker than round trips but apply to code that has no inverse.

Oracles. When a slow, obviously correct implementation exists, assert the fast one agrees with it. This is the highest-value property for optimisation work: the naive version is the specification, and any divergence is a bug in the new code.

@given(st.lists(st.integers()))
def test_fast_median_matches_naive(xs):
    if not xs:
        return
    naive = sorted(xs)[len(xs) // 2]
    assert fast_median(xs) == naive                 # the oracle is the spec

Metamorphic relations. When neither an inverse nor an oracle exists, relate two runs to each other. Adding an item to a cart must not decrease the total; searching a superset must return at least as many results; re-running an idempotent operation must change nothing. These catch bugs that no single-run assertion can see, because the defect lives in the relationship rather than in one output.

The failure mode to avoid is the tautological property — one that reimplements the code under test in the assertion. If the test computes the expected value with the same algorithm, it will agree with any bug the implementation contains. A quick check: could this property fail if the implementation were replaced by a subtly wrong one? If not, it is measuring nothing.

Two operational notes. Keep each property focused on one claim so a failure names the broken behaviour, and let Hypothesis generate the shape as well as the values — a list strategy with min_size=0 finds the empty-input bug that a hand-written example list almost always omits. And when a property is expensive, reduce max_examples for the local run and restore it in CI rather than deleting the property; a property that runs a hundred times nightly is worth far more than one that was too slow to keep.

Budgeting property tests in CI

Property tests are the only tests in a suite whose cost is a dial rather than a constant, and treating that dial as configuration keeps them from being deleted the first time someone optimises the pipeline.

Three profiles cover most projects. A dev profile with max_examples=10 and the deadline disabled keeps the local edit-run loop fast. A ci profile with the default hundred examples and derandomize=False gives every pull request fresh inputs. A nightly profile with max_examples=2000 and a generous deadline does the real searching, on a schedule where a twenty-minute run is acceptable.

# conftest.py
from hypothesis import HealthCheck, Phase, settings

settings.register_profile("dev", max_examples=10, deadline=None)
settings.register_profile("ci", max_examples=100, print_blob=True)
settings.register_profile("nightly", max_examples=2000, deadline=None,
                          suppress_health_check=[HealthCheck.too_slow])
settings.register_profile("replay", phases=[Phase.explicit, Phase.reuse])

Select with --hypothesis-profile=nightly or the HYPOTHESIS_PROFILE environment variable. The replay profile is the one teams forget: it runs only explicit @example cases and the stored database entries, which makes it perfect for a fast pre-merge job that re-checks every previously found counterexample without generating anything new.

Persisting the example database is what turns the nightly run into an asset. Point HYPOTHESIS_DATABASE_FILE at a directory that CI caches between runs, and a counterexample found at 3am is replayed against every subsequent pull request until it is fixed. Without that cache, each run starts from scratch and the same bug is rediscovered — or missed — at random.

Finally, keep print_blob=True on in CI. When a property fails, Hypothesis prints a base64 blob that reproduces the exact failing example via @reproduce_failure; pasting that into a local test is far faster than trying to recreate the input by hand from the printed repr. One more consideration when writing properties for numeric code: floating point. Equality assertions over floats fail on arbitrary generated input for reasons that have nothing to do with the code under test, so state the property in terms of a tolerance (math.isclose) or in terms of an ordering that survives rounding. Hypothesis will also generate nan and both infinities unless the strategy excludes them with allow_nan=False, allow_infinity=False, and deciding explicitly whether those belong in the domain is part of specifying the function rather than a workaround.

Properties also need a review convention, because a reviewer cannot check a generated test by reading its inputs. State the claim in the test name and repeat it as the assertion message: test_fee_never_exceeds_order_total with assert fee <= total, f"fee {fee} exceeded total {total}" is reviewable at a glance, where test_fee_property with a bare assert is not. The name is also what appears in CI output when the property fails at 3am, and a name that states the broken claim saves the on-call engineer from reading the strategy first.

Frequently Asked Questions

How do I prevent property-based tests from slowing down my CI pipeline? Register environment-aware settings profiles that scale max_examples by stage (50 for PRs, 500+ for nightly builds), run with pytest-xdist, cache the .hypothesis database, and avoid filter-heavy strategies. Monitor with --hypothesis-show-statistics and tune deadline accordingly.

When should I use assume() instead of strategy filter()? Use assume() inside the test to reject rare invalid inputs early while preserving shrinking efficiency. Avoid .filter() on strategies, which forces the generator to retry until a valid example appears and degrades exponentially as rejection climbs. Prefer valid-by-construction @composite strategies for complex constraints.

How do I reproduce a failing stateful test in production? Hypothesis prints a @reproduce_failure decorator encoding the exact seed and execution trace. Copy it into the test to replay the failure. Cache and commit the .hypothesis database so the minimal example survives across machines and CI runners.

Can I combine Hypothesis with coverage-guided fuzzers like Atheris? Yes. Use Hypothesis to generate structured, type-safe inputs, then serialize them to byte buffers for consumption by Atheris. Hypothesis provides semantic correctness while Atheris drives low-level memory and boundary testing of C-extensions and parsers.

How does shrinking work for a RuleBasedStateMachine? When a stateful test fails, Hypothesis shrinks the entire sequence of rule calls, not just individual argument values. It removes steps, reorders where legal, and minimizes each remaining argument, producing the shortest operation trace that still violates an invariant.

Related guides

• The reusable generator patterns behind Step 1 are detailed in generating custom strategies with hypothesis.strategies, and interdependent fields are covered in composing strategies with flatmap and @composite.
• Go deeper on the machine in Step 2 with stateful and model-based testing.
• For the underlying @given, assume, and shrinking model, revisit the Hypothesis framework fundamentals.
• When these tests get slow, reducing Hypothesis test execution time isolates the bottleneck phase.
• Combine generative inputs with matrix-driven cases using advanced parametrization techniques in the pytest track.
• Isolate impure dependencies in fuzz targets with autospec strict mocking.
• Why Hypothesis shrinking stalls and how to fix it — rejection filters, draw order, and the counterexample nobody can read.

← Back to Property-Based & Fuzz Testing Strategies