[{"data":1,"prerenderedAt":893},["ShallowReactive",2],{"page-\u002Fproperty-based-fuzz-testing-strategies\u002Fadvanced-property-based-testing\u002Fwriting-metamorphic-properties\u002F":3},{"id":4,"title":5,"body":6,"description":859,"extension":860,"meta":861,"navigation":96,"path":889,"seo":890,"stem":891,"__hash__":892},"content\u002Fproperty-based-fuzz-testing-strategies\u002Fadvanced-property-based-testing\u002Fwriting-metamorphic-properties\u002Findex.md","Writing Metamorphic Properties",{"type":7,"value":8,"toc":847},"minimark",[9,13,16,21,44,48,77,226,369,373,376,379,383,386,433,530,534,537,598,610,614,617,620,641,647,654,734,738,741,744,748,784,788,794,800,806,810,838,843],[10,11,12],"p",{},"Property-based testing is easiest when there is an oracle — a simple, obviously correct way to compute the expected answer. Many interesting systems have no such oracle. What is the correct ranking for a search query? The right output of a physics simulation? The exact prediction of a trained model? Nobody can compute these independently, which leaves example-based tests that check a handful of hand-verified cases and little else.",[10,14,15],{},"Metamorphic testing sidesteps the missing oracle. Instead of checking one output against the truth, it checks how two outputs relate when the inputs are related in a known way. You may not know the correct ranking for a query, but you know that shuffling the documents in the index should not change it, that adding an unrelated document should not push a perfect match out of the results, and that searching for a term twice should give the same answer as searching once. Each of those is a property Hypothesis can test across thousands of generated inputs.",[17,18,20],"h2",{"id":19},"prerequisites","Prerequisites",[22,23,24,36],"ul",{},[25,26,27,31,32,35],"li",{},[28,29,30],"code",{},"hypothesis >= 6.100",", ",[28,33,34],{},"pytest >= 8.0",".",[25,37,38,39,35],{},"Comfort writing strategies, as in ",[40,41,43],"a",{"href":42},"\u002Fproperty-based-fuzz-testing-strategies\u002Fadvanced-property-based-testing\u002F","Advanced property-based testing",[17,45,47],{"id":46},"solution","Solution",[49,50,55],"pre",{"className":51,"code":52,"language":53,"meta":54,"style":54},"language-python shiki shiki-themes github-light github-dark","# search.py — the system under test (abridged)\ndef search(index: list[Doc], query: str, k: int = 10) -> list[str]:\n    \"\"\"Return ids of the top-k documents for query.\"\"\"\n","python","",[28,56,57,65,71],{"__ignoreMap":54},[58,59,62],"span",{"class":60,"line":61},"line",1,[58,63,64],{},"# search.py — the system under test (abridged)\n",[58,66,68],{"class":60,"line":67},2,[58,69,70],{},"def search(index: list[Doc], query: str, k: int = 10) -> list[str]:\n",[58,72,74],{"class":60,"line":73},3,[58,75,76],{},"    \"\"\"Return ids of the top-k documents for query.\"\"\"\n",[49,78,80],{"className":51,"code":79,"language":53,"meta":54,"style":54},"# test_search_metamorphic.py\nfrom hypothesis import given, strategies as st\n\nwords = st.sampled_from([\"alpha\", \"beta\", \"gamma\", \"delta\", \"omega\"])\ndocs = st.builds(Doc, id=st.uuids().map(str), text=st.lists(words, min_size=1).map(\" \".join))\nindexes = st.lists(docs, min_size=1, max_size=30, unique_by=lambda d: d.id)\n\n@given(indexes, words, st.randoms())\ndef test_order_of_index_does_not_matter(index, query, rnd):\n    shuffled = index[:]\n    rnd.shuffle(shuffled)\n    assert set(search(index, query)) == set(search(shuffled, query))\n\n@given(indexes, words)\ndef test_irrelevant_document_does_not_displace_results(index, query):\n    noise = Doc(id=\"noise\", text=\"zzz unrelated\")\n    before = search(index, query)\n    after = search(index + [noise], query)\n    assert \"noise\" not in after\n    assert after == before\n\n@given(indexes, words)\ndef test_adding_an_exact_match_puts_it_in_results(index, query):\n    exact = Doc(id=\"exact\", text=\" \".join([query] * 5))\n    assert \"exact\" in search(index + [exact], query)\n",[28,81,82,87,92,98,104,110,116,121,127,133,139,145,151,156,162,168,174,180,186,192,198,203,208,214,220],{"__ignoreMap":54},[58,83,84],{"class":60,"line":61},[58,85,86],{},"# test_search_metamorphic.py\n",[58,88,89],{"class":60,"line":67},[58,90,91],{},"from hypothesis import given, strategies as st\n",[58,93,94],{"class":60,"line":73},[58,95,97],{"emptyLinePlaceholder":96},true,"\n",[58,99,101],{"class":60,"line":100},4,[58,102,103],{},"words = st.sampled_from([\"alpha\", \"beta\", \"gamma\", \"delta\", \"omega\"])\n",[58,105,107],{"class":60,"line":106},5,[58,108,109],{},"docs = st.builds(Doc, id=st.uuids().map(str), text=st.lists(words, min_size=1).map(\" \".join))\n",[58,111,113],{"class":60,"line":112},6,[58,114,115],{},"indexes = st.lists(docs, min_size=1, max_size=30, unique_by=lambda d: d.id)\n",[58,117,119],{"class":60,"line":118},7,[58,120,97],{"emptyLinePlaceholder":96},[58,122,124],{"class":60,"line":123},8,[58,125,126],{},"@given(indexes, words, st.randoms())\n",[58,128,130],{"class":60,"line":129},9,[58,131,132],{},"def test_order_of_index_does_not_matter(index, query, rnd):\n",[58,134,136],{"class":60,"line":135},10,[58,137,138],{},"    shuffled = index[:]\n",[58,140,142],{"class":60,"line":141},11,[58,143,144],{},"    rnd.shuffle(shuffled)\n",[58,146,148],{"class":60,"line":147},12,[58,149,150],{},"    assert set(search(index, query)) == set(search(shuffled, query))\n",[58,152,154],{"class":60,"line":153},13,[58,155,97],{"emptyLinePlaceholder":96},[58,157,159],{"class":60,"line":158},14,[58,160,161],{},"@given(indexes, words)\n",[58,163,165],{"class":60,"line":164},15,[58,166,167],{},"def test_irrelevant_document_does_not_displace_results(index, query):\n",[58,169,171],{"class":60,"line":170},16,[58,172,173],{},"    noise = Doc(id=\"noise\", text=\"zzz unrelated\")\n",[58,175,177],{"class":60,"line":176},17,[58,178,179],{},"    before = search(index, query)\n",[58,181,183],{"class":60,"line":182},18,[58,184,185],{},"    after = search(index + [noise], query)\n",[58,187,189],{"class":60,"line":188},19,[58,190,191],{},"    assert \"noise\" not in after\n",[58,193,195],{"class":60,"line":194},20,[58,196,197],{},"    assert after == before\n",[58,199,201],{"class":60,"line":200},21,[58,202,97],{"emptyLinePlaceholder":96},[58,204,206],{"class":60,"line":205},22,[58,207,161],{},[58,209,211],{"class":60,"line":210},23,[58,212,213],{},"def test_adding_an_exact_match_puts_it_in_results(index, query):\n",[58,215,217],{"class":60,"line":216},24,[58,218,219],{},"    exact = Doc(id=\"exact\", text=\" \".join([query] * 5))\n",[58,221,223],{"class":60,"line":222},25,[58,224,225],{},"    assert \"exact\" in search(index + [exact], query)\n",[227,228,231,365],"figure",{"className":229},[230],"diagram",[232,233,240,241,240,245,240,249,240,267,240,275,240,285,240,293,240,299,240,305,240,309,240,315,240,322,240,326,240,330,240,332,240,334,240,339,240,342,240,347,240,350,240,356,240,361],"svg",{"viewBox":234,"role":235,"ariaLabelledBy":236,"xmlns":239},"0 0 800 246","img",[237,238],"mm-t","mm-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[242,243,244],"title",{"id":237},"A metamorphic relation",[246,247,248],"desc",{"id":238},"A base input goes through the system under test to produce output A. A transformed input, derived from the base input by a known transformation such as shuffling, goes through the same system to produce output B. The test checks a relation between A and B, such as equality of result sets, instead of comparing either against a known correct answer.",[250,251,252,253,240],"defs",{},"\n    ",[254,255,262],"marker",{"id":256,"viewBox":257,"refX":258,"refY":259,"markerWidth":260,"markerHeight":260,"orient":261},"mm-a","0 0 10 10","9","5","7","auto-start-reverse",[263,264],"path",{"d":265,"fill":266},"M0 0 L10 5 L0 10 z","#81b29a",[268,269],"rect",{"x":270,"y":270,"width":271,"height":272,"rx":273,"fill":274},"0","800","246","14","#fffdf8",[276,277,284],"text",{"x":278,"y":279,"textAnchor":280,"fontSize":281,"fontWeight":282,"fill":283},"400","28","middle","15.5","700","#3d405b","Compare two runs, not one run with the truth",[268,286],{"x":287,"y":288,"width":289,"height":290,"rx":258,"fill":291,"stroke":283,"strokeWidth":292},"30","54","170","44","#f4f1de","1.5",[276,294,298],{"x":295,"y":296,"textAnchor":280,"fontSize":297,"fill":283},"115","81","12","base input",[268,300],{"x":287,"y":301,"width":289,"height":290,"rx":258,"fill":302,"stroke":303,"strokeWidth":304},"162","#f7f0da","#f2cc8f","1.8",[276,306,308],{"x":295,"y":307,"textAnchor":280,"fontSize":297,"fill":283},"189","transformed input",[60,310],{"x1":295,"y1":311,"x2":295,"y2":312,"stroke":266,"strokeWidth":313,"markerEnd":314},"100","158","1.6","url(#mm-a)",[276,316,321],{"x":317,"y":318,"fontSize":319,"fill":320},"124","134","10.5","#8a5a00","shuffle \u002F add noise \u002F scale",[268,323],{"x":324,"y":288,"width":289,"height":290,"rx":258,"fill":325,"stroke":266,"strokeWidth":304},"300","#e6f0ea",[276,327,329],{"x":328,"y":296,"textAnchor":280,"fontSize":297,"fill":283},"385","system under test",[268,331],{"x":324,"y":301,"width":289,"height":290,"rx":258,"fill":325,"stroke":266,"strokeWidth":304},[276,333,329],{"x":328,"y":307,"textAnchor":280,"fontSize":297,"fill":283},[60,335],{"x1":336,"y1":337,"x2":338,"y2":337,"stroke":266,"strokeWidth":313,"markerEnd":314},"204","76","296",[60,340],{"x1":336,"y1":341,"x2":338,"y2":341,"stroke":266,"strokeWidth":313,"markerEnd":314},"184",[60,343],{"x1":344,"y1":337,"x2":345,"y2":346,"stroke":266,"strokeWidth":313,"markerEnd":314},"474","560","118",[60,348],{"x1":344,"y1":341,"x2":345,"y2":349,"stroke":266,"strokeWidth":313,"markerEnd":314},"142",[268,351],{"x":352,"y":311,"width":353,"height":354,"rx":355,"fill":283},"566","208","60","11",[276,357,360],{"x":358,"y":359,"textAnchor":280,"fontSize":297,"fontWeight":282,"fill":274},"670","126","relation(A, B)",[276,362,364],{"x":358,"y":363,"textAnchor":280,"fontSize":355,"fill":274},"146","equal · subset · scaled",[366,367,368],"figcaption",{},"The transformation is chosen so its effect on the output is known, even when the output itself is not.",[17,370,372],{"id":371},"why-this-works","Why this works",[10,374,375],{},"A bug in a complex system rarely respects every symmetry of the problem. An off-by-one in pagination makes results depend on where a document sits in the index, which the shuffle relation catches. A scoring bug that rewards document length breaks the irrelevant-document relation, because the noise document shouldn't outrank anything. A tokeniser that drops the last word breaks the exact-match relation. None of those tests knows the correct ranking for any query, yet together they pin down a lot of what \"correct\" means.",[10,377,378],{},"Hypothesis contributes what it always does — many inputs, chosen to include edge cases such as single-document indexes, duplicate texts and queries that match nothing — and, crucially, shrinking. When a relation fails, the minimal counterexample is usually small enough to reason about directly: two documents, one query, a shuffle that swaps them. That turns \"rankings are sometimes unstable\" into a concrete reproduction.",[17,380,382],{"id":381},"a-catalogue-of-useful-relations","A catalogue of useful relations",[10,384,385],{},"Relations come in families, and working through the families is the fastest way to find properties for a new system.",[22,387,388,395,401,407,424],{},[25,389,390,394],{},[391,392,393],"strong",{},"Invariance."," Some transformation should not change the output at all: permuting input order, renaming irrelevant identifiers, adding whitespace to source code before compiling it, converting units consistently in both input and expected output.",[25,396,397,400],{},[391,398,399],{},"Equivariance."," The output should transform in step with the input: rotating an image rotates detected bounding boxes; scaling all prices by two scales the total by two; translating every point translates the centroid.",[25,402,403,406],{},[391,404,405],{},"Monotonicity."," Changing the input in one direction moves the output in a known direction: adding a matching term never lowers a document's score; adding an item to a cart never lowers the subtotal; tightening a filter never returns more rows.",[25,408,409,412,413,416,417,419,420,423],{},[391,410,411],{},"Inclusion."," One output contains the other: results for ",[28,414,415],{},"a AND b"," are a subset of results for ",[28,418,40],{},"; a query with a larger ",[28,421,422],{},"k"," returns a superset of the smaller one's results.",[25,425,426,429,430,35],{},[391,427,428],{},"Composition."," Doing something in two steps equals doing it in one: applying two discounts in sequence equals applying their combined rate; ",[28,431,432],{},"merge(merge(a, b), c) == merge(a, merge(b, c))",[227,434,436,527],{"className":435},[230],[232,437,240,442,240,445,240,448,240,451,240,454,240,460,240,464,240,468,240,473,240,476,240,480,240,483,240,486,240,491,240,495,240,498,240,502,240,505,240,508,240,510,240,513,240,517,240,521,240,524],{"viewBox":438,"role":235,"ariaLabelledBy":439,"xmlns":239},"0 0 800 256",[440,441],"mmc-t","mmc-d",[242,443,444],{"id":440},"Families of metamorphic relations",[246,446,447],{"id":441},"Five cards list relation families with an example of each: invariance, where shuffling input leaves output unchanged; equivariance, where scaling prices scales the total; monotonicity, where adding a matching term never lowers a score; inclusion, where results for a and b are a subset of results for a; and composition, where two sequential discounts equal their combined rate.",[268,449],{"x":270,"y":270,"width":271,"height":450,"rx":273,"fill":274},"256",[276,452,453],{"x":278,"y":279,"textAnchor":280,"fontSize":281,"fontWeight":282,"fill":283},"Five places to look for a relation",[268,455],{"x":456,"y":457,"width":458,"height":459,"rx":355,"fill":325,"stroke":266,"strokeWidth":304},"20","50","144","180",[276,461,463],{"x":462,"y":337,"textAnchor":280,"fontSize":297,"fontWeight":282,"fill":283},"92","invariance",[276,465,467],{"x":462,"y":466,"textAnchor":280,"fontSize":319,"fill":283},"110","shuffle input",[276,469,472],{"x":462,"y":470,"textAnchor":280,"fontSize":319,"fill":471},"130","#2a5f49","output unchanged",[268,474],{"x":475,"y":457,"width":458,"height":459,"rx":355,"fill":302,"stroke":303,"strokeWidth":304},"176",[276,477,479],{"x":478,"y":337,"textAnchor":280,"fontSize":297,"fontWeight":282,"fill":283},"248","equivariance",[276,481,482],{"x":478,"y":466,"textAnchor":280,"fontSize":319,"fill":283},"scale prices x2",[276,484,485],{"x":478,"y":470,"textAnchor":280,"fontSize":319,"fill":320},"total x2",[268,487],{"x":488,"y":457,"width":458,"height":459,"rx":355,"fill":489,"stroke":490,"strokeWidth":304},"332","#fbe9e3","#e07a5f",[276,492,494],{"x":493,"y":337,"textAnchor":280,"fontSize":297,"fontWeight":282,"fill":283},"404","monotonicity",[276,496,497],{"x":493,"y":466,"textAnchor":280,"fontSize":319,"fill":283},"add matching term",[276,499,501],{"x":493,"y":470,"textAnchor":280,"fontSize":319,"fill":500},"#8f3d22","score never drops",[268,503],{"x":504,"y":457,"width":458,"height":459,"rx":355,"fill":325,"stroke":266,"strokeWidth":304},"488",[276,506,507],{"x":345,"y":337,"textAnchor":280,"fontSize":297,"fontWeight":282,"fill":283},"inclusion",[276,509,415],{"x":345,"y":466,"textAnchor":280,"fontSize":319,"fill":283},[276,511,512],{"x":345,"y":470,"textAnchor":280,"fontSize":319,"fill":471},"subset of a",[268,514],{"x":515,"y":457,"width":516,"height":459,"rx":355,"fill":302,"stroke":303,"strokeWidth":304},"644","136",[276,518,520],{"x":519,"y":337,"textAnchor":280,"fontSize":297,"fontWeight":282,"fill":283},"712","composition",[276,522,523],{"x":519,"y":466,"textAnchor":280,"fontSize":319,"fill":283},"two steps",[276,525,526],{"x":519,"y":470,"textAnchor":280,"fontSize":319,"fill":320},"equal one step",[366,528,529],{},"For a new system, ask each question in turn; most systems yield at least three independent relations.",[17,531,533],{"id":532},"numerical-code-relations-with-tolerance","Numerical code: relations with tolerance",[10,535,536],{},"Metamorphic testing is particularly effective for numerical code, where exact oracles are expensive and floating-point makes equality fragile. The relations are the same; the comparison needs a tolerance.",[49,538,540],{"className":51,"code":539,"language":53,"meta":54,"style":54},"import math\nfrom hypothesis import given, strategies as st\n\nfinite = st.floats(-1e6, 1e6, allow_nan=False, allow_infinity=False)\n\n@given(st.lists(finite, min_size=1, max_size=50), st.floats(0.5, 4.0))\ndef test_mean_scales_with_input(xs, c):\n    assert math.isclose(mean([x * c for x in xs]), c * mean(xs), rel_tol=1e-9, abs_tol=1e-6)\n\n@given(st.lists(finite, min_size=1, max_size=50), finite)\ndef test_mean_shifts_with_input(xs, d):\n    assert math.isclose(mean([x + d for x in xs]), mean(xs) + d, rel_tol=1e-9, abs_tol=1e-3)\n",[28,541,542,547,551,555,560,564,569,574,579,583,588,593],{"__ignoreMap":54},[58,543,544],{"class":60,"line":61},[58,545,546],{},"import math\n",[58,548,549],{"class":60,"line":67},[58,550,91],{},[58,552,553],{"class":60,"line":73},[58,554,97],{"emptyLinePlaceholder":96},[58,556,557],{"class":60,"line":100},[58,558,559],{},"finite = st.floats(-1e6, 1e6, allow_nan=False, allow_infinity=False)\n",[58,561,562],{"class":60,"line":106},[58,563,97],{"emptyLinePlaceholder":96},[58,565,566],{"class":60,"line":112},[58,567,568],{},"@given(st.lists(finite, min_size=1, max_size=50), st.floats(0.5, 4.0))\n",[58,570,571],{"class":60,"line":118},[58,572,573],{},"def test_mean_scales_with_input(xs, c):\n",[58,575,576],{"class":60,"line":123},[58,577,578],{},"    assert math.isclose(mean([x * c for x in xs]), c * mean(xs), rel_tol=1e-9, abs_tol=1e-6)\n",[58,580,581],{"class":60,"line":129},[58,582,97],{"emptyLinePlaceholder":96},[58,584,585],{"class":60,"line":135},[58,586,587],{},"@given(st.lists(finite, min_size=1, max_size=50), finite)\n",[58,589,590],{"class":60,"line":141},[58,591,592],{},"def test_mean_shifts_with_input(xs, d):\n",[58,594,595],{"class":60,"line":147},[58,596,597],{},"    assert math.isclose(mean([x + d for x in xs]), mean(xs) + d, rel_tol=1e-9, abs_tol=1e-3)\n",[10,599,600,601,604,605,609],{},"The tolerances matter. Relative tolerance handles large magnitudes; absolute tolerance handles results near zero, where relative error blows up. Bounding the strategy's range keeps the relation meaningful — with values near the limits of ",[28,602,603],{},"float",", scaling can overflow and the relation becomes a test of IEEE 754 rather than of your code. The ",[40,606,608],{"href":607},"\u002Fproperty-based-fuzz-testing-strategies\u002Fadvanced-property-based-testing\u002Ftesting-numeric-code-with-floats-and-nan-edge-cases\u002F","floats and NaN guide"," covers the edge cases in depth.",[17,611,613],{"id":612},"checking-that-the-relations-actually-catch-bugs","Checking that the relations actually catch bugs",[10,615,616],{},"A metamorphic suite gives a reassuring number of green ticks, and it is fair to ask whether those ticks mean anything. The cheapest way to find out is to break the code on purpose and watch which relations notice. This is mutation testing done by hand, and for a new set of relations it is worth an afternoon.",[10,618,619],{},"Pick a few plausible bugs — the kinds a real change might introduce — and apply each one temporarily:",[22,621,622,625,628,631],{},[25,623,624],{},"sort results by insertion order instead of score for ties, which the shuffle relation should catch;",[25,626,627],{},"add document length to the score, which the irrelevant-document relation should catch;",[25,629,630],{},"drop the last token of the query, which the exact-match relation should catch;",[25,632,633,634,637,638,640],{},"return only the first ",[28,635,636],{},"k - 1"," results, which the inclusion relation between different ",[28,639,422],{}," values should catch.",[10,642,643,644,35],{},"Run the property tests against each mutant. A mutant that no relation kills points to a gap: either a missing relation, or a strategy that never generates the inputs where the bug matters. The fix for the second case is usually in the generator — the length bug might only show when documents differ in length by a lot, so the text strategy needs a wider ",[28,645,646],{},"max_size",[10,648,649,650,653],{},"For larger codebases, a tool such as ",[28,651,652],{},"mutmut"," automates the same idea across many small syntactic changes. The output is noisier than a hand-picked list, but it scales, and a surviving mutant in scoring code is a direct prompt for the next relation to write. What matters is closing the loop: metamorphic relations are hypotheses about how bugs show themselves, and like any hypothesis they are worth testing.",[227,655,657,731],{"className":656},[230],[232,658,240,663,240,666,240,669,240,672,240,675,240,681,240,686,240,690,240,694,240,698,240,701,240,704,240,708,240,711,240,714,240,718,240,721,240,724,240,728],{"viewBox":659,"role":235,"ariaLabelledBy":660,"xmlns":239},"0 0 800 236",[661,662],"mmk-t","mmk-d",[242,664,665],{"id":661},"Which relation kills which mutant",[246,667,668],{"id":662},"A grid pairs four deliberate bugs with the relations that detect them. The tie-order bug is caught by the shuffle relation, the length-bias bug by the irrelevant-document relation, the dropped-token bug by the exact-match relation, and the short-results bug by the inclusion relation. Any row without a catching relation marks a gap.",[268,670],{"x":270,"y":270,"width":271,"height":671,"rx":273,"fill":274},"236",[276,673,674],{"x":278,"y":279,"textAnchor":280,"fontSize":281,"fontWeight":282,"fill":283},"Every deliberate bug should turn something red",[268,676],{"x":677,"y":678,"width":679,"height":680,"rx":260,"fill":283},"26","46","748","32",[276,682,685],{"x":678,"y":683,"fontSize":684,"fontWeight":282,"fill":274},"67","11.5","mutant",[276,687,689],{"x":688,"y":683,"fontSize":684,"fontWeight":282,"fill":274},"420","killed by",[268,691],{"x":677,"y":692,"width":679,"height":680,"rx":693,"fill":291},"82","6",[276,695,697],{"x":678,"y":696,"fontSize":355,"fill":283},"103","ties ordered by insertion",[276,699,700],{"x":688,"y":696,"fontSize":355,"fill":471},"shuffle invariance",[268,702],{"x":677,"y":346,"width":679,"height":680,"rx":693,"fill":274,"stroke":703},"rgba(61,64,91,0.14)",[276,705,707],{"x":678,"y":706,"fontSize":355,"fill":283},"139","score rewards document length",[276,709,710],{"x":688,"y":706,"fontSize":355,"fill":471},"irrelevant document",[268,712],{"x":677,"y":713,"width":679,"height":680,"rx":693,"fill":291},"154",[276,715,717],{"x":678,"y":716,"fontSize":355,"fill":283},"175","last query token dropped",[276,719,720],{"x":688,"y":716,"fontSize":355,"fill":471},"exact match included",[268,722],{"x":677,"y":723,"width":679,"height":680,"rx":693,"fill":274,"stroke":703},"190",[276,725,727],{"x":678,"y":726,"fontSize":355,"fill":283},"211","returns k - 1 results",[276,729,730],{"x":688,"y":726,"fontSize":355,"fill":471},"inclusion across k",[366,732,733],{},"A mutant that survives every relation is a precise description of the next property to write.",[17,735,737],{"id":736},"where-metamorphic-properties-sit-in-a-suite","Where metamorphic properties sit in a suite",[10,739,740],{},"Metamorphic properties are not a replacement for example tests; they are the layer that example tests cannot provide. A good suite for a search service keeps a small set of golden examples — \"this query on this fixture index returns these ids in this order\" — that document intended behaviour and catch gross regressions. The metamorphic layer then checks the symmetries across thousands of generated indexes the golden set never covers.",[10,742,743],{},"The two layers fail differently, and that is useful. A golden test failing says behaviour changed; someone must decide whether the change is intended and update the expectation. A metamorphic test failing says behaviour became inconsistent, which is almost never intended and needs no debate about expectations. That makes metamorphic failures cheap to triage, and it is why teams that adopt them tend to keep adding relations.",[17,745,747],{"id":746},"edge-cases-and-failure-modes","Edge cases and failure modes",[22,749,750,756,762,768,774],{},[25,751,752,755],{},[391,753,754],{},"Relations that are accidentally true."," A function that always returns an empty list satisfies invariance, inclusion and monotonicity. Pair metamorphic properties with at least one property that forces non-trivial output, such as the exact-match test above.",[25,757,758,761],{},[391,759,760],{},"Ties breaking relations."," If two documents score equally, shuffling can legitimately change their order. Compare sets, or define a deterministic tie-break in the system and test that.",[25,763,764,767],{},[391,765,766],{},"Transformations that change meaning."," Adding a \"noise\" document containing a query word is not noise. Generate transformed data from a disjoint vocabulary.",[25,769,770,773],{},[391,771,772],{},"Non-determinism."," Randomised algorithms need a seeded RNG, or a relation that holds in distribution rather than per call.",[25,775,776,779,780,783],{},[391,777,778],{},"Expensive systems."," Each property runs the system at least twice per example. Lower ",[28,781,782],{},"max_examples"," for heavy systems and push deep search to a nightly profile.",[17,785,787],{"id":786},"frequently-asked-questions","Frequently Asked Questions",[10,789,790,793],{},[391,791,792],{},"What is a metamorphic property?","\nA metamorphic property relates the outputs of two calls on related inputs, instead of checking one output against a known answer. For example, reordering the documents in a search index should not change the set of results for a query.",[10,795,796,799],{},[391,797,798],{},"When should I use metamorphic testing instead of an oracle?","\nWhen you cannot cheaply compute the correct answer — search ranking, machine-learning inference, numerical solvers, compilers. If a simple reference implementation exists, comparing against it is usually stronger.",[10,801,802,805],{},[391,803,804],{},"How many metamorphic relations do I need?","\nSeveral, each catching a different class of bug. A single relation is easy to satisfy by accident; three or four independent relations covering ordering, scaling, irrelevant data and composition constrain the implementation much more.",[17,807,809],{"id":808},"related","Related",[22,811,812,818,825,831],{},[25,813,814,817],{},[40,815,816],{"href":42},"Advanced Property-Based Testing"," — property patterns beyond round-trips.",[25,819,820,824],{},[40,821,823],{"href":822},"\u002Fproperty-based-fuzz-testing-strategies\u002Fadvanced-property-based-testing\u002Fround-trip-properties-for-serializers-and-parsers\u002F","Round-Trip Properties for Serializers and Parsers"," — the other oracle-free workhorse.",[25,826,827,830],{},[40,828,829],{"href":607},"Testing Numeric Code with Floats and NaN Edge Cases"," — tolerances and special values.",[25,832,833,837],{},[40,834,836],{"href":835},"\u002Fproperty-based-fuzz-testing-strategies\u002Fstateful-and-model-based-testing\u002F","Stateful and Model-Based Testing"," — when a simple model can serve as the oracle.",[10,839,840,841],{},"← Back to ",[40,842,816],{"href":42},[844,845,846],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":54,"searchDepth":67,"depth":67,"links":848},[849,850,851,852,853,854,855,856,857,858],{"id":19,"depth":67,"text":20},{"id":46,"depth":67,"text":47},{"id":371,"depth":67,"text":372},{"id":381,"depth":67,"text":382},{"id":532,"depth":67,"text":533},{"id":612,"depth":67,"text":613},{"id":736,"depth":67,"text":737},{"id":746,"depth":67,"text":747},{"id":786,"depth":67,"text":787},{"id":808,"depth":67,"text":809},"Test code with no easy oracle by relating outputs across transformed inputs: permutation, scaling, adding irrelevant data, and composition — with Hypothesis examples for search, ranking and numerics.","md",{"slug":862,"type":863,"breadcrumb":864,"datePublished":865,"dateModified":865,"faq":866,"howto":873},"writing-metamorphic-properties","article","Metamorphic properties","2026-09-18",[867,869,871],{"q":792,"a":868},"A metamorphic property relates the outputs of two calls on related inputs, instead of checking one output against a known answer. For example, reordering the documents in a search index should not change the set of results for a query.",{"q":798,"a":870},"When you cannot cheaply compute the correct answer — search ranking, machine-learning inference, numerical solvers, compilers. If a simple reference implementation exists, comparing against it is usually stronger.",{"q":804,"a":872},"Several, each catching a different class of bug. A single relation is easy to satisfy by accident; three or four independent relations covering ordering, scaling, irrelevant data and composition constrain the implementation much more.",{"name":874,"description":875,"steps":876},"How to write metamorphic properties with Hypothesis","Identify transformations whose effect on the output is known, then test that relation across generated inputs.",[877,880,883,886],{"name":878,"text":879},"List transformations","Write down input changes with a predictable effect: reorder, duplicate, scale, add irrelevant items.",{"name":881,"text":882},"State the expected relation","For each transformation, say whether the output should be equal, a permutation, scaled, or a superset.",{"name":884,"text":885},"Generate both inputs","Draw a base input and derive the transformed input from it inside the test.",{"name":887,"text":888},"Compare the two outputs","Assert the relation, with tolerances for floating point where needed.","\u002Fproperty-based-fuzz-testing-strategies\u002Fadvanced-property-based-testing\u002Fwriting-metamorphic-properties",{"title":5,"description":859},"property-based-fuzz-testing-strategies\u002Fadvanced-property-based-testing\u002Fwriting-metamorphic-properties\u002Findex","T3x0l84gn_evQnr2sVjRg0UVjHDsJ3MOzrb8Tmdn5yQ",1789718767649]