Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Hypothesis & Fuzzing

Generating DataFrames and Arrays with Hypothesis Extras

Numerical code breaks on inputs nobody writes by hand: an empty array, a single row, a column of identical values, a float that is exactly representable in one dtype and not another, a NaN in the one position the algorithm did not expect. Example-based tests cover the shapes the author thought of; Hypothesis's extra.numpy and extra.pandas modules generate arrays and frames across shapes, dtypes and element values, and then shrink any failure to the smallest one that still breaks.

The strategies are declarative — dtype, shape bounds, element strategy — and most of the skill is in constraining them to the inputs the code is supposed to handle while leaving enough variety to find the ones it mishandles. A deliberate policy on NaN and infinity, chosen per property, is the single most important decision. Get it wrong in one direction and every property has to hedge against values that make it meaningless; get it wrong in the other and the suite never sees the missing values real data is full of.

Prerequisites

Solution

Python
import numpy as np
import pandas as pd
from hypothesis import given, settings, strategies as st
from hypothesis.extra import numpy as npst
from hypothesis.extra import pandas as pdst

finite = st.floats(min_value=-1e6, max_value=1e6, allow_nan=False, allow_infinity=False)


@given(npst.arrays(dtype=np.float64,
                   shape=npst.array_shapes(min_dims=1, max_dims=2, max_side=20),
                   elements=finite))
def test_normalise_preserves_shape_and_is_bounded(arr):
    out = normalise(arr)
    assert out.shape == arr.shape
    assert np.all((out >= 0.0) & (out <= 1.0) | np.isnan(out))


orders = pdst.data_frames(
    columns=[
        pdst.column("order_id", dtype=str, elements=st.text(min_size=1, max_size=8),
                    unique=True),
        pdst.column("amount_minor", dtype=np.int64,
                    elements=st.integers(min_value=0, max_value=10_000_000)),
        pdst.column("currency", elements=st.sampled_from(["GBP", "USD", "JPY"])),
    ],
    index=pdst.range_indexes(min_size=0, max_size=30),   # includes the empty frame
)


@settings(max_examples=150)
@given(orders)
def test_totals_by_currency_match_a_plain_python_sum(df):
    result = totals_by_currency(df)
    for currency, group in df.groupby("currency"):
        assert result[currency] == int(group["amount_minor"].sum())
The three decisions in an array strategy An array strategy is built from three choices. The dtype decides representable values and overflow behaviour. The shape strategy bounds the number of dimensions and the side length, including empty and single-element arrays. The element strategy bounds values and sets the policy for NaN and infinity. Together they define the input space the property is claimed over. dtype × shape × elements = the claimed input space dtype float64, float32, int64 … decides precision and overflow behaviour match production data shape min_dims, max_dims, max_side — includes empty and single-element arrays keep sides small elements bounds, and a deliberate NaN / infinity policy per property the decision that matters most
Each choice is a claim about what the code accepts. Making them explicit in the strategy documents the function's contract as well as testing it.

Why this works

npst.arrays draws a shape from the shape strategy, then fills an array of that shape with values from the element strategy, cast to the dtype. Because shape and elements are both strategies, shrinking reduces both: a failure on a 17×9 array of large floats shrinks to the smallest shape and simplest values that still fail — often a 1×1 array containing zero, or an empty array, which points straight at the bug.

pdst.data_frames does the same per column and builds the frame through pandas, so the result has real dtypes and a real index. Declaring columns explicitly makes generated frames schema-valid by construction, and range_indexes(min_size=0) guarantees the empty frame is among the inputs — the case most groupby and aggregation code gets wrong first.

Edge cases and failure modes

  • Unbounded floats. st.floats() includes values near 1e308 whose sums overflow to infinity. Bound them to the range the code handles.
  • NaN excluded everywhere. Excluding NaN in every strategy hides the bugs where real data contains it. Keep at least one property that allows it.
  • Large frames. Big max_size values make each example and every shrink step slow. Twenty or thirty rows find nearly all bugs.
  • Object dtypes. Columns without a dtype default to object, which pandas treats differently from typed columns. Give every column an explicit dtype matching production.
  • Comparing floats exactly. A reference computed in a different order can differ in the last bit. Use np.allclose or math.isclose with a tolerance justified by the algorithm.

Dtype-specific surprises generation finds

A large share of the bugs these strategies find are not logic errors but dtype behaviour the author did not expect, and knowing the usual suspects makes the shrunk counterexamples quicker to read.

Integer overflow without an error. NumPy integer arithmetic wraps silently: np.int32(2**31 - 1) + 1 is a large negative number, not an exception. A sum over a generated int32 column with large values produces a negative total, and the counterexample is typically two elements near the dtype's maximum. The fix is an explicit wider dtype for the accumulation, and the test is what reveals that one is needed.

Float precision across dtypes. Values exactly representable in float64 may round in float32, so a round trip through a float32 column changes them. Properties comparing before and after a cast need a tolerance, or a strategy restricted to values representable in the narrower type — st.floats(width=32) generates exactly those.

Signed zero and NaN comparisons. -0.0 == 0.0 is true but they sort and hash differently in some contexts; NaN != NaN breaks any equality-based deduplication. A generated column containing both zeros or a NaN will find every place that assumes otherwise.

Nullable versus NumPy dtypes in pandas. Int64 with capital I supports missing values, int64 does not, and a column that acquires a missing value is silently upcast to float64. Generating frames with an occasional None in an integer column exposes the upcast, which usually shows up downstream as a comparison failing on 1.0 != 1.

Dtype behaviours generated inputs expose Four cards. Integer overflow wraps silently to a negative value. Float32 rounding changes values that were exact in float64. Signed zero and NaN break equality-based logic. Nullable pandas integers upcast to float when a missing value appears. Each is found by generation and typically shrinks to a two-element example. Most shrunk counterexamples are one of these silent integer overflow int32 max + 1 → large negative accumulate in a wider dtype float32 rounding exact in float64, not in float32 st.floats(width=32) or a tolerance -0.0 and NaN NaN != NaN breaks dedup use isnan-aware comparisons nullable int upcast int64 + missing → float64 use Int64 or handle NaN
Recognising the pattern in a two-element counterexample usually identifies the fix before any debugging starts.

Reference implementations as the property

The hardest part of property-testing numerical code is stating a property at all — "the output is correct" is not checkable without an oracle. The most productive oracle is a slow, obvious implementation of the same computation written in plain Python, with the vectorised or optimised version tested for agreement against it on generated inputs.

That is the pattern in the DataFrame test above: totals_by_currency might use a vectorised groupby with categorical dtypes and a pre-sorted index, while the check uses a loop over groups that nobody could get wrong. Any disagreement is a bug in the fast version — or occasionally a precision question worth understanding — and Hypothesis shrinks it to the smallest frame that disagrees, typically two or three rows that make the cause obvious.

The reference need not be efficient, and should not be clever. Its only job is to be obviously correct for small inputs, which is exactly the size Hypothesis generates and shrinks to. The same idea applies to any optimised numerical routine: a moving average against an explicit window loop, a matrix operation against nested loops, a custom aggregation against its textbook definition. It is the property-based equivalent of a golden reference, without the need to store golden outputs.

Testing an optimised routine against a plain reference Generated frames are fed to both the optimised vectorised implementation and a slow plain Python reference. Their results are compared, and any disagreement is shrunk by Hypothesis to the smallest frame that still disagrees, usually a handful of rows that reveal the bug. The obvious version is the oracle generated frame optimised routine vectorised groupby plain reference a loop over groups compare shrink any mismatch
The reference only has to be obviously right on small inputs — exactly the size Hypothesis generates and shrinks to.

Keeping generation affordable

Array and frame strategies are the most expensive in Hypothesis, and a suite that adopts them carelessly can add minutes to every run. Three habits keep the cost proportionate to the value.

Bound sizes aggressively. Bugs in numerical code almost always reproduce on tiny inputs — the shrinker proves this every time it reduces a failure to a 2×1 array — so max_side=20 or a thirty-row frame finds essentially everything a thousand-row one would, at a fraction of the cost. Large-input behaviour such as performance or memory belongs in a benchmark, not a property test. Keeping the two concerns separate keeps both fast.

Generate only the columns the property reads. A frame with twelve columns where the property touches two spends most of its time building data nobody checks. Declaring the two, and filling the rest with constants in the function under test's fixture if it requires them, keeps each example cheap.

Use profiles to scale the budget. A development profile with fifty examples keeps the edit loop fast; the CI profile runs a few hundred; a nightly profile runs thousands with larger shapes. Nothing about the tests changes between them — not the strategies, not the assertions — only the search budget, which is the approach set out in Hypothesis integration with pytest and frameworks.

Frequently Asked Questions

Should generated floats include NaN and infinity? Decide per property. A property about shape or dtype holds with NaN present and should include it; a property about sums, means or sorting usually needs finite values and should exclude them explicitly with allow_nan=False and allow_infinity=False. Test the non-finite behaviour in its own property rather than weakening every assertion.

Why is DataFrame generation slow? Because every example builds a full frame through pandas, and shrinking rebuilds many more. Keep row counts small with max_size, restrict columns to what the property needs, and prefer generating arrays and constructing the frame yourself when the schema is simple.

How do I generate a DataFrame that matches a real schema? Declare each column with hypothesis.extra.pandas.column, giving its dtype and an element strategy that respects the domain — non-negative amounts, a fixed set of currency codes — and optionally unique=True. The resulting frames are valid by construction.

← Back to Designing Strategies for Domain Data