Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Hypothesis & Fuzzing

Reducing Hypothesis Test Execution Time in CI

Your Hypothesis suite has crept from seconds to minutes, and the CI dashboard shows it eating the pipeline budget — so someone reaches for max_examples=10 or deadline=None, which makes the clock look better while silently gutting failure detection. The disciplined fix is diagnostic-first: measure which of the three phases (generation, execution, shrinking) is actually slow, then apply the targeted remedy. Within the broader property-based and fuzz testing approach, speed work means aligning strategy complexity, phase execution, and database caching with your system's real computational boundaries — never disabling Hypothesis features wholesale.

Prerequisites

  • hypothesis>=6.100, pytest>=8.0, pytest-xdist, and pytest-profiling for flame graphs.
  • A test invoked with --hypothesis-show-statistics (add it to addopts in pytest.ini).

Solution

Start by reading the phase breakdown, then bound complexity and swap .filter() for assume().

Bash
# Phase-level timing + a flame graph of where CPU goes
pytest --profile -k test_property --hypothesis-show-statistics

The statistics output reports Generate vs Shrink time. If Generate dominates, your strategies produce oversized or deeply nested objects — bound them. If Shrink dominates, the test body has expensive side effects that rerun on every minimization step.

Python
from hypothesis import given, settings, assume
import hypothesis.strategies as st

# Bound every collection and numeric range so generation stays cheap
@given(st.lists(st.text(max_size=20), max_size=10))
@settings(max_examples=200)
def test_bounded(data: list[str]) -> None:
    assert len("".join(data)) <= 200

# Inefficient: filter() generates a full example, checks, discards, retries
@given(st.integers().filter(lambda x: x % 2 == 0))
def test_slow_filter(x: int) -> None:
    assert x // 2 == x / 2

# Optimized: assume() rejects at the byte-stream level, letting the engine backtrack
@given(st.integers())
def test_fast_assume(x: int) -> None:
    assume(x % 2 == 0)
    assert x // 2 == x / 2

Recursive strategies are the highest combinatorial risk; always bound depth explicitly.

Python
from hypothesis import given, settings
import hypothesis.strategies as st

@st.composite
def bounded_tree(draw, max_depth: int = 3):
    depth = draw(st.integers(0, max_depth))
    if depth == 0:
        return None
    return {"value": draw(st.integers(-100, 100)),
            "left": draw(bounded_tree(max_depth=depth - 1)),
            "right": draw(bounded_tree(max_depth=depth - 1))}

@given(bounded_tree(max_depth=4))
@settings(max_examples=150)
def test_tree(tree: dict | None) -> None:
    pass  # traversal executes predictably because depth is capped

Finally, persist the database and parallelize. Caching .hypothesis/examples/ across runs replays known failures first instead of rediscovering them, and pytest -n auto spreads execution across workers — the pytest-xdist vs pytest-parallel performance comparison covers which runner isolates the example database most cleanly under load.

YAML
# .github/workflows/test.yml
steps:
  - uses: actions/checkout@v4
  - uses: actions/setup-python@v5
    with: { python-version: "3.12" }
  - uses: actions/cache@v4
    with:
      path: .hypothesis/examples
      key: hypothesis-db-${{ runner.os }}-${{ hashFiles('**/requirements.txt') }}
  - run: pip install -r requirements.txt pytest pytest-xdist hypothesis
  - run: pytest -n auto --hypothesis-profile ci
    env: { HYPOTHESIS_PROFILE: ci }
Diagnose the slow phase, then apply the matched fix A single example flows through three phases shown as a bar: Generate (widest), Execute (with a per-execution deadline bracket), and Shrink (dashed, only on failure). An arrow drops from each phase to its targeted remedy. If Generate dominates, bound collection and recursion sizes and swap filter for assume. If Execute is slow, the deadline flags it and you profile the test body before raising the limit. If Shrink dominates, keep the test body cheap because it reruns on every minimization step. The split is read from pytest hypothesis-show-statistics. Diagnose the slow phase, then fix that phase Generate draw an example Execute run property Shrink only on failure deadline = per execution If Generate dominates Bound max_size / max_value Cap recursion depth Swap .filter() for assume() rejection at the byte stream, not a discarded object If Execute is slow DeadlineExceeded flags it Profile the body first Fix O(n²) logic raise the deadline only after the body is cheap If Shrink dominates Keep the test body cheap Move side effects out Cache fixtures the body reruns on every minimization step Read the split from: pytest --hypothesis-show-statistics
Read the Generate, Execute, and Shrink split from statistics, then fix the phase that dominates: bound strategies and swap filter for assume when Generate leads, profile the body when Execute trips the deadline, and keep the body cheap when Shrink leads because it reruns on every minimization step.

Runtime splits into four parts, and only two of them respond to lowering max_examples.

Where the time in a property test goes A bar chart splitting property-test runtime across four contributors: running the property body once per example, generating the data, shrinking after a failure, and fixed per-test setup such as fixtures and imports. Where the time in a property test goes property body x examples ~55% data generation ~25% shrinking a failure ~15% fixed per-test setup ~5% Split measured with --durations and a profiler on a mid-sized suite.
Lowering max_examples scales the first two bars only; a slow fixture or an expensive strategy setup is paid once per test either way.

Why this works

assume() integrates with Hypothesis's internal byte-stream allocation, so a rejection lets the engine backtrack and adjust generation parameters immediately, whereas .filter() generates a full object, evaluates the predicate, and discards everything on failure — benchmarks show 60–80% less generation overhead for constrained spaces. Bounding collection and recursion sizes caps the work both generation and shrinking must do per example. The example database turns repeat failures into instant replays, and worker-isolated databases let pytest-xdist cut wall-clock time without SQLite locking corruption.

Edge cases and failure modes

  • Deadline breach masking O(n²) logic — a DeadlineExceeded often means the assertion itself is slow on large input; profile the body before raising the threshold, and distinguish it from the flaky Hypothesis health-check failures that HealthCheck.too_slow raises during generation rather than execution.
  • pytest-xdist SQLite locking — workers sharing one database file corrupt serialization; rely on hypothesis.extra.pytestplugin worker suffixing or database=None.
  • Disabling shrinking permanently — phases=[Phase.generate] is fine for smoke tests but turns production failures into opaque stack traces; re-enable Phase.shrink for real pipelines.
  • Stale database skewing generation — months-old examples bias toward outdated edge cases; prune the directory on a retention schedule.
  • st.from_type() on complex Pydantic/SQLAlchemy models — recursive resolution generates oversized objects; override with explicit st.builds(Model, field=st.integers(min_value=0, max_value=10)), applying the bounding patterns from generating custom strategies with hypothesis.strategies.

Cutting cost without cutting coverage

The reflex when property tests get slow is to lower max_examples everywhere. That works, and it also throws away the search that made the tests worth having. Four cheaper reductions come first.

Stop generating what you do not vary. A strategy that draws a large object when the property only reads one field is paying for the whole object every example. Draw the field, and construct the rest as a constant outside the test.

Python
from hypothesis import given, strategies as st

CONSTANT_ORDER = build_order(lines=3)      # built once, at import time

@given(st.integers(min_value=0, max_value=10 ** 6))
def test_fee_is_monotonic(amount):
    # No per-example object construction: only the varying value is drawn.
    assert fee(CONSTANT_ORDER, amount) >= 0

Move setup out of the example loop. Anything inside the test body runs once per example. A database connection, a parsed schema, or a compiled regex belongs in a module-scoped fixture; with a hundred examples that is a hundredfold saving on a single line of setup.

Bound the collection sizes. st.lists(st.text()) generates lists of unbounded strings, and the average example is far larger than the bug you are hunting requires. max_size=10 on the list and max_size=32 on the text usually loses nothing and cuts generation time sharply — most defects that appear with ten elements also appear with three.

Split the slow properties out. A suite where three properties account for most of the runtime should run those three under a separate marker on a nightly schedule with a higher example count, and keep the fast majority in the pull-request run. This is the opposite of lowering the dial: the slow properties get more search, not less, and the fast feedback loop stays fast.

Bash
$ pytest -m "not slow_property" --hypothesis-profile=ci        # per pull request
$ pytest -m slow_property --hypothesis-profile=nightly         # scheduled, 2000 examples

Only after those four should max_examples come down, and even then not uniformly: lower it on properties whose domain is small enough to be covered quickly, and leave it high on the ones that search a large space. A round-trip property over four-field records genuinely benefits from a thousand examples; a property over a five-member enum is exhausted after twenty.

One measurement to take before changing anything: run with --durations=10 and the ci profile, then again with max_examples=1. The difference is the part of the runtime that scales with example count, and the remainder is fixed cost that no dial will touch. Teams routinely discover the fixed cost is the larger half, at which point the fix is a fixture change rather than a settings change.

Which reduction applies to a slow property A decision diagram: if runtime is dominated by fixed setup, move it into a fixture; if it scales with example count and the domain is small, lower max_examples; and if the domain is large, split the property into a nightly job with a higher count instead. Which reduction applies to a slow property What dominates the measured runtime? fixed setup move to a fixture runs once, not per example largest common win scales, small domain lower max_examples exhausted quickly anyway safe reduction scales, large domain split to nightly more search, off the path keep PR runs fast
Measure first with max_examples=1: the residue is fixed cost, and no example-count change will touch it.

Frequently Asked Questions

Why does my Hypothesis test run much slower on CI than locally? CI environments usually lack the .hypothesis/examples cache, forcing full regeneration and shrinking on every run. Cache the .hypothesis directory in the pipeline and standardize its path with DirectoryBasedExampleDatabase so failing examples replay across runners.

Can I safely use pytest-xdist with Hypothesis? Yes, but workers must not share one SQLite-backed database. The bundled hypothesis.extra.pytestplugin appends a worker ID to the database path automatically; otherwise use database=None for stateless parallel runs.

How do I tell whether slowness comes from generation or my test logic? Run pytest --hypothesis-show-statistics and read the Generate versus Shrink ratio. If Generate dominates, your strategies produce oversized objects; if Shrink dominates, the test body has expensive side effects that run on every minimization step.

Is it safe to disable shrinking to speed up tests? Only for exploratory smoke tests. Disabling Phase.shrink sacrifices minimal reproducers and turns actionable failures into opaque stack traces. Use phases=[Phase.generate] temporarily and re-enable shrinking for production pipelines. Does the example database make repeated runs faster or slower? Faster in practice. Replaying stored counterexamples costs a handful of executions at the start of a run, and it removes the far larger cost of rediscovering a known bug by search. The database only grows meaningfully when tests keep failing, and Hypothesis prunes entries whose test no longer exists — so cache the directory in CI and leave it alone.

Is derandomize=True a good way to make runs faster? It makes runs repeatable, not faster: the same seed produces the same examples every time, which removes cross-run variance but also removes the fresh search that finds new bugs. Use it for a pre-merge job where reproducibility matters and keep a randomised nightly run alongside it, so the suite still explores while pull requests stay deterministic.

← Back to Hypothesis Framework Fundamentals