Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Hypothesis & Fuzzing

Choosing max_examples and deadline Settings

Two settings decide how hard a Hypothesis test works and when it gives up: max_examples, the number of valid examples to try, and deadline, the longest a single example may take. Both have sensible defaults — 100 examples, 200 milliseconds — and both are routinely changed for the wrong reasons. Teams lower max_examples to make a slow suite pass the time budget, or disable deadlines because they flake, without asking what each setting actually buys.

The better approach treats them as policy. The number of examples reflects how much confidence a given run should purchase, which differs between a developer's edit-test loop, a pull-request pipeline and a nightly search. The deadline reflects whether per-example performance is part of the contract. Set both per environment through profiles, and override on individual tests only where there is a specific reason.

Prerequisites

Solution

Python
# conftest.py
import os
from datetime import timedelta

from hypothesis import HealthCheck, settings

settings.register_profile("dev", max_examples=25)
settings.register_profile("ci", max_examples=100, deadline=timedelta(milliseconds=800))
settings.register_profile(
    "nightly",
    max_examples=5_000,
    deadline=None,
    suppress_health_check=[HealthCheck.too_slow],
)
settings.load_profile(os.environ.get("HYPOTHESIS_PROFILE", "dev"))
Python
# test_pricing.py — a per-test override with a stated reason
from hypothesis import given, settings, strategies as st

@settings(max_examples=500)      # discount rules interact; bugs found past 100 before
@given(st.lists(st.integers(1, 10_000), max_size=20), st.sampled_from(RULES))
def test_total_never_negative(prices, rule):
    assert apply(rule, prices) >= 0
Bash
pytest -q --hypothesis-show-statistics tests/test_pricing.py
Budgets per environment Three profiles are shown as columns. The dev profile runs 25 examples for quick feedback with the default deadline. The CI profile runs 100 examples with a generous deadline. The nightly profile runs 5000 examples with no deadline and the too-slow health check suppressed. One suite, three levels of effort dev max_examples = 25 default deadline seconds per run ci max_examples = 100 deadline = 800 ms minutes per pipeline nightly max_examples = 5000 deadline = None too_slow suppressed deep search, off the path
The test code is identical in every column; only the loaded profile changes how much searching it does.

Why this works

max_examples counts valid examples — inputs that reached the end of the test body without being rejected by assume or a filter. Hypothesis spends those examples deliberately: early ones are small and simple, later ones grow, and it mixes in values from its database and from boundary heuristics. Returns diminish, but not uniformly. Tests over small input spaces (a boolean and an enum) saturate quickly, and extra examples buy nothing; tests over nested structures with interacting fields keep finding new behaviour for thousands of examples. That is why a single global number is a compromise, and why the deep search belongs in a scheduled run where time is cheap.

The deadline is enforced per example, in wall-clock time. When an example exceeds it, Hypothesis re-runs that example; if the second run is fast, the failure is reported as flaky because the input was not reliably slow. This two-step check filters most one-off noise but not sustained contention, where every run is slow. Hence the CI profile's larger deadline: it keeps the check for pathological slowdowns — an accidental quadratic path on long lists — while ignoring the ordinary slowness of a shared runner.

Reading the statistics before choosing numbers

Guessing a deadline is the usual source of flaky timing failures. --hypothesis-show-statistics prints, for each test, the number of passing, failing and invalid examples, the typical runtime range, and the fraction of time spent generating data rather than running the test. That output answers the two questions that matter.

The typical runtime tells you where to set the deadline. A test whose examples take 5–40 ms is well inside the default 200 ms, and failures there point at real slow inputs. A test whose examples take 150–400 ms will trip the default constantly; set its deadline to a few multiples of the upper end, or None if its speed is not something anyone would act on.

The count of invalid examples tells you whether max_examples is being spent well. If a test reports 100 passing and 900 invalid examples, most of the generation budget is wasted on rejected inputs, and raising max_examples makes it worse. Fix the strategy first, as described in using assume without tripping the filter health check, then decide the budget.

Placing a deadline against measured runtimes A horizontal time axis shows the typical example runtime band between 5 and 40 milliseconds. The default deadline at 200 milliseconds sits well above it, leaving room for noise. A pathological slow input at 900 milliseconds exceeds the deadline and is the failure the deadline exists to catch. Deadlines belong well above the typical band 0 1 s typical 5–40 ms deadline 200 ms headroom for noisy runners quadratic input, 900 ms the failure worth reporting
A deadline set from the measured band catches genuinely pathological inputs and ignores the jitter of a busy machine.

Edge cases and failure modes

  • Lowering max_examples to hit a time budget. This silently reduces confidence everywhere. Find the few expensive tests with --durations and address them individually.
  • Deadlines on tests with expensive setup. Fixture work does not count, but work inside the test body — building a database session per example — does. Move shared setup out of the example or raise the deadline for that test.
  • First-example cost. Imports and caches warmed inside the first example make it slow. Hypothesis tolerates this better than it used to, but warming in a fixture removes the question.
  • Profiles loaded too late. load_profile must run before tests are collected; placing it in the root conftest.py is enough.
  • Per-test overrides without a reason. A bare @settings(max_examples=1000) six months later is unexplained cost. Comment the reason, as in the example above.

When a slow example is itself the bug

Deadlines are often discussed purely as a source of flakiness, but the reason they exist is that property tests are unusually good at finding inputs that make code slow. A parser that is linear on realistic input can be quadratic on a string of ten thousand opening brackets; a regex can backtrack catastrophically on a crafted pattern; a sorting routine can hit its worst case on already-sorted data. Example-based tests rarely contain those inputs because nobody writes them by hand. Hypothesis generates them routinely.

When a deadline failure reproduces — the re-run is also slow — treat it like any other falsifying example. Shrink it, look at the input, and ask whether the complexity is acceptable. Often the answer is that the code has an accidental quadratic path, and the fix is a better data structure. Sometimes the answer is that the input is legitimately expensive and far outside what production sees; then bound the strategy (max_size) with a comment explaining why, rather than removing the deadline for all inputs.

That distinction is why the nightly profile disables the deadline while the CI profile keeps a generous one. The nightly search is looking for correctness bugs deep in the input space and should not be derailed by slowness it cannot act on. The CI run, over smaller inputs, can afford to treat a sudden tenfold slowdown as a signal worth a human's attention.

Diminishing returns and where the curve bends

It helps to picture what extra examples buy. Hypothesis front-loads its most productive work: the first handful of examples include the simplest values of each type, boundary values the strategies know about, and anything replayed from the database. Most shallow bugs — an empty list, a zero, a missing key — fall out in the first few dozen examples. After that, each additional example explores a slightly different corner of a large space, and the probability that any single one finds a new bug drops steadily.

The shape of that curve depends on the test. For a function of one small integer, the curve flattens almost immediately, and 100 examples is already more than enough. For a function over a nested structure with several interacting fields, the curve stays steep much longer, because bugs need a specific combination — a list of at least three items, one of which is negative, with a discount rule that applies to the second. Those are the tests worth a raised per-test budget, and the ones a nightly run at thousands of examples keeps finding things in.

A useful practice is to record, when a nightly run finds a bug, how many examples it took. If bugs keep turning up well past the pull-request budget for a particular test, that test deserves a higher per-test setting in CI too. If the nightly run has not found anything new in a given test for months, its nightly budget can shrink and the time can go to tests still producing findings.

Returns from additional examples Two curves plot bugs found against number of examples. A simple input space curve rises steeply and flattens within about one hundred examples. A structured input space curve keeps rising well past one thousand examples, which is where a larger budget pays off. Where extra examples still pay examples run bugs found simple input space structured input space CI budget
Raise budgets where the curve is still climbing; for flat curves, more examples only add runtime.

Frequently Asked Questions

What is a good default for max_examples? Hypothesis's default of 100 is a reasonable pull-request budget for most tests. Use fewer locally for fast feedback, and several thousand in a scheduled job for deep search. Raise it per test only where the input space is large and bugs are known to hide deep.

Should I disable the Hypothesis deadline? Disable it where timing is not something the test cares about and the environment is noisy, such as CI under parallel load. Keep it, generously set, where a sudden slowdown on some inputs would itself be a bug worth reporting.

Why does Hypothesis report a deadline failure as flaky? Because it re-runs the failing example to confirm it and the second run was fast. The example did not consistently exceed the deadline, so the failure is timing noise rather than a property of the input.

← Back to Hypothesis Framework Fundamentals