Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Hypothesis & Fuzzing

Writing Metamorphic Properties

Property-based testing is easiest when there is an oracle — a simple, obviously correct way to compute the expected answer. Many interesting systems have no such oracle. What is the correct ranking for a search query? The right output of a physics simulation? The exact prediction of a trained model? Nobody can compute these independently, which leaves example-based tests that check a handful of hand-verified cases and little else.

Metamorphic testing sidesteps the missing oracle. Instead of checking one output against the truth, it checks how two outputs relate when the inputs are related in a known way. You may not know the correct ranking for a query, but you know that shuffling the documents in the index should not change it, that adding an unrelated document should not push a perfect match out of the results, and that searching for a term twice should give the same answer as searching once. Each of those is a property Hypothesis can test across thousands of generated inputs.

Prerequisites

Solution

Python
# search.py — the system under test (abridged)
def search(index: list[Doc], query: str, k: int = 10) -> list[str]:
    """Return ids of the top-k documents for query."""
Python
# test_search_metamorphic.py
from hypothesis import given, strategies as st

words = st.sampled_from(["alpha", "beta", "gamma", "delta", "omega"])
docs = st.builds(Doc, id=st.uuids().map(str), text=st.lists(words, min_size=1).map(" ".join))
indexes = st.lists(docs, min_size=1, max_size=30, unique_by=lambda d: d.id)

@given(indexes, words, st.randoms())
def test_order_of_index_does_not_matter(index, query, rnd):
    shuffled = index[:]
    rnd.shuffle(shuffled)
    assert set(search(index, query)) == set(search(shuffled, query))

@given(indexes, words)
def test_irrelevant_document_does_not_displace_results(index, query):
    noise = Doc(id="noise", text="zzz unrelated")
    before = search(index, query)
    after = search(index + [noise], query)
    assert "noise" not in after
    assert after == before

@given(indexes, words)
def test_adding_an_exact_match_puts_it_in_results(index, query):
    exact = Doc(id="exact", text=" ".join([query] * 5))
    assert "exact" in search(index + [exact], query)
A metamorphic relation A base input goes through the system under test to produce output A. A transformed input, derived from the base input by a known transformation such as shuffling, goes through the same system to produce output B. The test checks a relation between A and B, such as equality of result sets, instead of comparing either against a known correct answer. Compare two runs, not one run with the truth base input transformed input shuffle / add noise / scale system under test system under test relation(A, B) equal · subset · scaled
The transformation is chosen so its effect on the output is known, even when the output itself is not.

Why this works

A bug in a complex system rarely respects every symmetry of the problem. An off-by-one in pagination makes results depend on where a document sits in the index, which the shuffle relation catches. A scoring bug that rewards document length breaks the irrelevant-document relation, because the noise document shouldn't outrank anything. A tokeniser that drops the last word breaks the exact-match relation. None of those tests knows the correct ranking for any query, yet together they pin down a lot of what "correct" means.

Hypothesis contributes what it always does — many inputs, chosen to include edge cases such as single-document indexes, duplicate texts and queries that match nothing — and, crucially, shrinking. When a relation fails, the minimal counterexample is usually small enough to reason about directly: two documents, one query, a shuffle that swaps them. That turns "rankings are sometimes unstable" into a concrete reproduction.

A catalogue of useful relations

Relations come in families, and working through the families is the fastest way to find properties for a new system.

  • Invariance. Some transformation should not change the output at all: permuting input order, renaming irrelevant identifiers, adding whitespace to source code before compiling it, converting units consistently in both input and expected output.
  • Equivariance. The output should transform in step with the input: rotating an image rotates detected bounding boxes; scaling all prices by two scales the total by two; translating every point translates the centroid.
  • Monotonicity. Changing the input in one direction moves the output in a known direction: adding a matching term never lowers a document's score; adding an item to a cart never lowers the subtotal; tightening a filter never returns more rows.
  • Inclusion. One output contains the other: results for a AND b are a subset of results for a; a query with a larger k returns a superset of the smaller one's results.
  • Composition. Doing something in two steps equals doing it in one: applying two discounts in sequence equals applying their combined rate; merge(merge(a, b), c) == merge(a, merge(b, c)).
Families of metamorphic relations Five cards list relation families with an example of each: invariance, where shuffling input leaves output unchanged; equivariance, where scaling prices scales the total; monotonicity, where adding a matching term never lowers a score; inclusion, where results for a and b are a subset of results for a; and composition, where two sequential discounts equal their combined rate. Five places to look for a relation invariance shuffle input output unchanged equivariance scale prices x2 total x2 monotonicity add matching term score never drops inclusion a AND b subset of a composition two steps equal one step
For a new system, ask each question in turn; most systems yield at least three independent relations.

Numerical code: relations with tolerance

Metamorphic testing is particularly effective for numerical code, where exact oracles are expensive and floating-point makes equality fragile. The relations are the same; the comparison needs a tolerance.

Python
import math
from hypothesis import given, strategies as st

finite = st.floats(-1e6, 1e6, allow_nan=False, allow_infinity=False)

@given(st.lists(finite, min_size=1, max_size=50), st.floats(0.5, 4.0))
def test_mean_scales_with_input(xs, c):
    assert math.isclose(mean([x * c for x in xs]), c * mean(xs), rel_tol=1e-9, abs_tol=1e-6)

@given(st.lists(finite, min_size=1, max_size=50), finite)
def test_mean_shifts_with_input(xs, d):
    assert math.isclose(mean([x + d for x in xs]), mean(xs) + d, rel_tol=1e-9, abs_tol=1e-3)

The tolerances matter. Relative tolerance handles large magnitudes; absolute tolerance handles results near zero, where relative error blows up. Bounding the strategy's range keeps the relation meaningful — with values near the limits of float, scaling can overflow and the relation becomes a test of IEEE 754 rather than of your code. The floats and NaN guide covers the edge cases in depth.

Checking that the relations actually catch bugs

A metamorphic suite gives a reassuring number of green ticks, and it is fair to ask whether those ticks mean anything. The cheapest way to find out is to break the code on purpose and watch which relations notice. This is mutation testing done by hand, and for a new set of relations it is worth an afternoon.

Pick a few plausible bugs — the kinds a real change might introduce — and apply each one temporarily:

  • sort results by insertion order instead of score for ties, which the shuffle relation should catch;
  • add document length to the score, which the irrelevant-document relation should catch;
  • drop the last token of the query, which the exact-match relation should catch;
  • return only the first k - 1 results, which the inclusion relation between different k values should catch.

Run the property tests against each mutant. A mutant that no relation kills points to a gap: either a missing relation, or a strategy that never generates the inputs where the bug matters. The fix for the second case is usually in the generator — the length bug might only show when documents differ in length by a lot, so the text strategy needs a wider max_size.

For larger codebases, a tool such as mutmut automates the same idea across many small syntactic changes. The output is noisier than a hand-picked list, but it scales, and a surviving mutant in scoring code is a direct prompt for the next relation to write. What matters is closing the loop: metamorphic relations are hypotheses about how bugs show themselves, and like any hypothesis they are worth testing.

Which relation kills which mutant A grid pairs four deliberate bugs with the relations that detect them. The tie-order bug is caught by the shuffle relation, the length-bias bug by the irrelevant-document relation, the dropped-token bug by the exact-match relation, and the short-results bug by the inclusion relation. Any row without a catching relation marks a gap. Every deliberate bug should turn something red mutant killed by ties ordered by insertion shuffle invariance score rewards document length irrelevant document last query token dropped exact match included returns k - 1 results inclusion across k
A mutant that survives every relation is a precise description of the next property to write.

Where metamorphic properties sit in a suite

Metamorphic properties are not a replacement for example tests; they are the layer that example tests cannot provide. A good suite for a search service keeps a small set of golden examples — "this query on this fixture index returns these ids in this order" — that document intended behaviour and catch gross regressions. The metamorphic layer then checks the symmetries across thousands of generated indexes the golden set never covers.

The two layers fail differently, and that is useful. A golden test failing says behaviour changed; someone must decide whether the change is intended and update the expectation. A metamorphic test failing says behaviour became inconsistent, which is almost never intended and needs no debate about expectations. That makes metamorphic failures cheap to triage, and it is why teams that adopt them tend to keep adding relations.

Edge cases and failure modes

  • Relations that are accidentally true. A function that always returns an empty list satisfies invariance, inclusion and monotonicity. Pair metamorphic properties with at least one property that forces non-trivial output, such as the exact-match test above.
  • Ties breaking relations. If two documents score equally, shuffling can legitimately change their order. Compare sets, or define a deterministic tie-break in the system and test that.
  • Transformations that change meaning. Adding a "noise" document containing a query word is not noise. Generate transformed data from a disjoint vocabulary.
  • Non-determinism. Randomised algorithms need a seeded RNG, or a relation that holds in distribution rather than per call.
  • Expensive systems. Each property runs the system at least twice per example. Lower max_examples for heavy systems and push deep search to a nightly profile.

Frequently Asked Questions

What is a metamorphic property? A metamorphic property relates the outputs of two calls on related inputs, instead of checking one output against a known answer. For example, reordering the documents in a search index should not change the set of results for a query.

When should I use metamorphic testing instead of an oracle? When you cannot cheaply compute the correct answer — search ranking, machine-learning inference, numerical solvers, compilers. If a simple reference implementation exists, comparing against it is usually stronger.

How many metamorphic relations do I need? Several, each catching a different class of bug. A single relation is easy to satisfy by accident; three or four independent relations covering ordering, scaling, irrelevant data and composition constrain the implementation much more.

← Back to Advanced Property-Based Testing