[{"data":1,"prerenderedAt":1003},["ShallowReactive",2],{"page-\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fbisecting-test-order-dependencies\u002F":3},{"id":4,"title":5,"body":6,"description":969,"extension":970,"meta":971,"navigation":119,"path":999,"seo":1000,"stem":1001,"__hash__":1002},"content\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fbisecting-test-order-dependencies\u002Findex.md","Bisecting Test Order Dependencies",{"type":7,"value":8,"toc":958},"minimark",[9,13,21,26,52,56,260,343,464,468,471,477,482,486,489,552,572,662,666,669,683,694,704,741,807,811,832,839,850,854,890,894,900,910,916,920,949,954],[10,11,12],"p",{},"Order-dependent tests are the most frustrating kind of flaky test, because nothing about them is random once you know the order. Test B passes alone, passes in the usual order, and fails whenever test A happens to run first. A leaves something behind — a module-level cache it populated, an environment variable it set directly, a patched function it never restored, a row in a shared database — and B trips over it. Randomised ordering, parallel runs and newly added tests all rearrange the sequence, so the failure appears and disappears for reasons that look like chance.",[10,14,15,16,20],{},"The way out is mechanical. Get a reproducible order that fails. Confirm the victim passes alone. Then bisect the tests that ran before it: run half of them, then the victim, and keep whichever half still produces the failure. With a few hundred predecessors, eight or nine runs find the polluter. The tool ",[17,18,19],"code",{},"detect-test-pollution"," automates exactly this.",[22,23,25],"h2",{"id":24},"prerequisites","Prerequisites",[27,28,29,44],"ul",{},[30,31,32,35,36,39,40,43],"li",{},[17,33,34],{},"pytest >= 8.0",", ",[17,37,38],{},"pytest-randomly >= 3.15",", optionally ",[17,41,42],{},"detect-test-pollution >= 1.2",".",[30,45,46,47,43],{},"Background from ",[48,49,51],"a",{"href":50},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002F","Debugging tests in CI and containers",[22,53,55],{"id":54},"solution","Solution",[57,58,63],"pre",{"className":59,"code":60,"language":61,"meta":62,"style":62},"language-bash shiki shiki-themes github-light github-dark","# 1. Reproduce the failing order. pytest-randomly prints the seed:\n#    Using --randomly-seed=2918476\npytest -p randomly --randomly-seed=2918476 -v 2>&1 | tee run.log\n\n# 2. Save the node ids that ran before the victim, in order.\nVICTIM=\"tests\u002Ftest_billing.py::test_invoice_total\"\ngrep -E ' (PASSED|FAILED|ERROR|SKIPPED)' run.log | awk '{print $1}' \\\n  | awk -v v=\"$VICTIM\" '$0 == v {exit} {print}' > before.txt\n\n# 3. Confirm the victim passes alone.\npytest \"$VICTIM\"\n\n# 4. Let detect-test-pollution bisect the predecessors automatically.\ndetect-test-pollution --failing-test \"$VICTIM\" --testids-file before.txt\n#   ...\n#   the polluting test is: tests\u002Ftest_settings.py::test_override_currency\n","bash","",[17,64,65,74,80,114,121,127,140,166,194,199,205,218,223,229,248,254],{"__ignoreMap":62},[66,67,70],"span",{"class":68,"line":69},"line",1,[66,71,73],{"class":72},"sJ8bj","# 1. Reproduce the failing order. pytest-randomly prints the seed:\n",[66,75,77],{"class":68,"line":76},2,[66,78,79],{"class":72},"#    Using --randomly-seed=2918476\n",[66,81,83,87,91,95,98,101,105,108,111],{"class":68,"line":82},3,[66,84,86],{"class":85},"sScJk","pytest",[66,88,90],{"class":89},"sj4cs"," -p",[66,92,94],{"class":93},"sZZnC"," randomly",[66,96,97],{"class":89}," --randomly-seed=2918476",[66,99,100],{"class":89}," -v",[66,102,104],{"class":103},"szBVR"," 2>&1",[66,106,107],{"class":103}," |",[66,109,110],{"class":85}," tee",[66,112,113],{"class":93}," run.log\n",[66,115,117],{"class":68,"line":116},4,[66,118,120],{"emptyLinePlaceholder":119},true,"\n",[66,122,124],{"class":68,"line":123},5,[66,125,126],{"class":72},"# 2. Save the node ids that ran before the victim, in order.\n",[66,128,130,134,137],{"class":68,"line":129},6,[66,131,133],{"class":132},"sVt8B","VICTIM",[66,135,136],{"class":103},"=",[66,138,139],{"class":93},"\"tests\u002Ftest_billing.py::test_invoice_total\"\n",[66,141,143,146,149,152,155,157,160,163],{"class":68,"line":142},7,[66,144,145],{"class":85},"grep",[66,147,148],{"class":89}," -E",[66,150,151],{"class":93}," ' (PASSED|FAILED|ERROR|SKIPPED)'",[66,153,154],{"class":93}," run.log",[66,156,107],{"class":103},[66,158,159],{"class":85}," awk",[66,161,162],{"class":93}," '{print $1}'",[66,164,165],{"class":89}," \\\n",[66,167,169,172,174,176,179,182,185,188,191],{"class":68,"line":168},8,[66,170,171],{"class":103},"  |",[66,173,159],{"class":85},[66,175,100],{"class":89},[66,177,178],{"class":93}," v=\"",[66,180,181],{"class":132},"$VICTIM",[66,183,184],{"class":93},"\"",[66,186,187],{"class":93}," '$0 == v {exit} {print}'",[66,189,190],{"class":103}," >",[66,192,193],{"class":93}," before.txt\n",[66,195,197],{"class":68,"line":196},9,[66,198,120],{"emptyLinePlaceholder":119},[66,200,202],{"class":68,"line":201},10,[66,203,204],{"class":72},"# 3. Confirm the victim passes alone.\n",[66,206,208,210,213,215],{"class":68,"line":207},11,[66,209,86],{"class":85},[66,211,212],{"class":93}," \"",[66,214,181],{"class":132},[66,216,217],{"class":93},"\"\n",[66,219,221],{"class":68,"line":220},12,[66,222,120],{"emptyLinePlaceholder":119},[66,224,226],{"class":68,"line":225},13,[66,227,228],{"class":72},"# 4. Let detect-test-pollution bisect the predecessors automatically.\n",[66,230,232,234,237,239,241,243,246],{"class":68,"line":231},14,[66,233,19],{"class":85},[66,235,236],{"class":89}," --failing-test",[66,238,212],{"class":93},[66,240,181],{"class":132},[66,242,184],{"class":93},[66,244,245],{"class":89}," --testids-file",[66,247,193],{"class":93},[66,249,251],{"class":68,"line":250},15,[66,252,253],{"class":72},"#   ...\n",[66,255,257],{"class":68,"line":256},16,[66,258,259],{"class":72},"#   the polluting test is: tests\u002Ftest_settings.py::test_override_currency\n",[57,261,263],{"className":59,"code":262,"language":61,"meta":62,"style":62},"# Manual bisection, when you want to see each step.\nsplit -n l\u002F2 before.txt half_\npytest -p no:randomly $(cat half_aa) \"$VICTIM\"   # fails? polluter is in half_aa\npytest -p no:randomly $(cat half_ab) \"$VICTIM\"   # else it is in half_ab — repeat\n",[17,264,265,270,287,317],{"__ignoreMap":62},[66,266,267],{"class":68,"line":69},[66,268,269],{"class":72},"# Manual bisection, when you want to see each step.\n",[66,271,272,275,278,281,284],{"class":68,"line":76},[66,273,274],{"class":85},"split",[66,276,277],{"class":89}," -n",[66,279,280],{"class":93}," l\u002F2",[66,282,283],{"class":93}," before.txt",[66,285,286],{"class":93}," half_\n",[66,288,289,291,293,296,299,302,305,308,310,312,314],{"class":68,"line":82},[66,290,86],{"class":85},[66,292,90],{"class":89},[66,294,295],{"class":93}," no:randomly",[66,297,298],{"class":132}," $(",[66,300,301],{"class":85},"cat",[66,303,304],{"class":93}," half_aa",[66,306,307],{"class":132},") ",[66,309,184],{"class":93},[66,311,181],{"class":132},[66,313,184],{"class":93},[66,315,316],{"class":72},"   # fails? polluter is in half_aa\n",[66,318,319,321,323,325,327,329,332,334,336,338,340],{"class":68,"line":116},[66,320,86],{"class":85},[66,322,90],{"class":89},[66,324,295],{"class":93},[66,326,298],{"class":132},[66,328,301],{"class":85},[66,330,331],{"class":93}," half_ab",[66,333,307],{"class":132},[66,335,184],{"class":93},[66,337,181],{"class":132},[66,339,184],{"class":93},[66,341,342],{"class":72},"   # else it is in half_ab — repeat\n",[344,345,348,460],"figure",{"className":346},[347],"diagram",[349,350,357,358,357,362,357,366,357,374,357,384,357,390,357,399,357,406,357,413,357,417,357,424,357,426,357,430,357,434,357,436,357,440,357,448,357,454,357,456],"svg",{"viewBox":351,"role":352,"ariaLabelledBy":353,"xmlns":356},"0 0 800 256","img",[354,355],"bi-t","bi-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[359,360,361],"title",{"id":354},"Bisecting the tests that ran before the victim",[363,364,365],"desc",{"id":355},"Four rows show a bisection. The first row has 256 predecessor tests before the victim. Each subsequent row keeps the half that still makes the victim fail: 128, then 64, and so on, until after eight rounds a single polluting test remains. The number of runs grows with the logarithm of the number of predecessors.",[367,368],"rect",{"x":369,"y":369,"width":370,"height":371,"rx":372,"fill":373},"0","800","256","14","#fffdf8",[375,376,383],"text",{"x":377,"y":378,"textAnchor":379,"fontSize":380,"fontWeight":381,"fill":382},"400","28","middle","15.5","700","#3d405b","Halve until one polluter remains",[375,385,389],{"x":386,"y":387,"fontSize":388,"fill":382},"30","70","11","256 tests",[367,391],{"x":392,"y":393,"width":394,"height":395,"rx":396,"fill":397,"stroke":398},"120","56","560","22","4","#f4f1de","rgba(61,64,91,0.35)",[367,400],{"x":401,"y":393,"width":402,"height":395,"rx":396,"fill":403,"stroke":404,"strokeWidth":405},"690","80","#fbe9e3","#e07a5f","1.6",[375,407,412],{"x":408,"y":409,"textAnchor":379,"fontSize":410,"fill":411},"730","72","10.5","#8f3d22","victim",[375,414,416],{"x":386,"y":415,"fontSize":388,"fill":382},"108","128",[367,418],{"x":377,"y":419,"width":420,"height":395,"rx":396,"fill":421,"stroke":422,"strokeWidth":423},"94","280","#f7f0da","#f2cc8f","1.4",[367,425],{"x":401,"y":419,"width":402,"height":395,"rx":396,"fill":403,"stroke":404,"strokeWidth":405},[375,427,429],{"x":386,"y":428,"fontSize":388,"fill":382},"146","64",[367,431],{"x":377,"y":432,"width":433,"height":395,"rx":396,"fill":421,"stroke":422,"strokeWidth":423},"132","140",[367,435],{"x":401,"y":432,"width":402,"height":395,"rx":396,"fill":403,"stroke":404,"strokeWidth":405},[375,437,439],{"x":386,"y":438,"fontSize":388,"fill":382},"184","… 1",[367,441],{"x":442,"y":443,"width":444,"height":395,"rx":396,"fill":445,"stroke":446,"strokeWidth":447},"470","170","18","#e6f0ea","#81b29a","2",[375,449,453],{"x":450,"y":451,"fontSize":410,"fontWeight":381,"fill":452},"500","186","#2a5f49","polluter",[367,455],{"x":401,"y":443,"width":402,"height":395,"rx":396,"fill":403,"stroke":404,"strokeWidth":405},[375,457,459],{"x":377,"y":458,"textAnchor":379,"fontSize":388,"fill":382},"230","log2(256) = 8 runs, each only as long as its half",[461,462,463],"figcaption",{},"Each round keeps the half that still breaks the victim, so the number of runs grows with the logarithm of the suite size.",[22,465,467],{"id":466},"why-this-works","Why this works",[10,469,470],{},"An order dependency means there exists some test P such that running P before the victim V makes V fail. If the dependency comes from a single polluter — by far the most common case — then among any split of the predecessors into two halves, exactly one half contains P, and running that half followed by V reproduces the failure. Keeping that half and repeating halves the search space each time, so 256 candidates need about eight rounds and 4,000 need about twelve. Each round runs only the candidate half, so later rounds are fast.",[10,472,473,476],{},[17,474,475],{},"-p no:randomly"," in the manual steps matters: once you have a failing sequence, you want each bisection run to execute exactly the listed tests in exactly the listed order. pytest runs explicitly listed node ids in the order given when randomisation is off.",[10,478,479,481],{},[17,480,19],{}," performs the same algorithm, with extra care: it first confirms the victim passes alone and fails after the full list, and it handles collection of node ids for you. When it finishes, it prints a single polluting test id.",[22,483,485],{"id":484},"finding-what-the-polluter-leaves-behind","Finding what the polluter leaves behind",[10,487,488],{},"Knowing the polluter is half the job; the other half is knowing what state it leaks. Read the polluter looking for side effects that outlive it:",[27,490,491,505,518,528,542],{},[30,492,493,497,498,501,502,43],{},[494,495,496],"strong",{},"Environment variables"," set with ",[17,499,500],{},"os.environ[...] = ..."," or ",[17,503,504],{},"os.putenv",[30,506,507,510,511,514,515,43],{},[494,508,509],{},"Module attributes"," assigned directly — ",[17,512,513],{},"settings.CURRENCY = \"EUR\""," — or functions replaced without ",[17,516,517],{},"monkeypatch",[30,519,520,523,524,527],{},[494,521,522],{},"Caches"," filled as a side effect: ",[17,525,526],{},"functools.lru_cache"," on configuration loaders, class-level registries, memoised clients.",[30,529,530,533,534,537,538,541],{},[494,531,532],{},"Process state",": the working directory changed with ",[17,535,536],{},"os.chdir",", the default timezone or locale changed, ",[17,539,540],{},"sys.path"," extended.",[30,543,544,547,548,551],{},[494,545,546],{},"External state",": database rows committed outside a rolled-back transaction, files written outside ",[17,549,550],{},"tmp_path",", messages left on a queue.",[10,553,554,555,35,558,561,562,35,565,35,568,571],{},"A quick confirmation is to diff state around the polluter. Snapshot ",[17,556,557],{},"dict(os.environ)",[17,559,560],{},"os.getcwd()"," and the relevant module attributes before and after running it alone; whatever differs is the leak. The fix is almost always to route the change through a fixture that restores it — ",[17,563,564],{},"monkeypatch.setenv",[17,566,567],{},"monkeypatch.setattr",[17,569,570],{},"monkeypatch.chdir",", a cache-clearing teardown, or a transaction that rolls back — so the test can make whatever change it needs without leaving it for the next one.",[344,573,575,659],{"className":574},[347],[349,576,357,581,357,584,357,587,357,590,357,593,357,599,357,604,357,607,357,611,357,615,357,618,357,622,357,625,357,628,357,631,357,635,357,638,357,641,357,645,357,648,357,652,357,656],{"viewBox":577,"role":352,"ariaLabelledBy":578,"xmlns":356},"0 0 800 236",[579,580],"bil-t","bil-d",[359,582,583],{"id":579},"Common leaks and their restoring replacements",[363,585,586],{"id":580},"A table maps common leaks to fixes. Setting os.environ directly is replaced by monkeypatch.setenv. Assigning module attributes is replaced by monkeypatch.setattr. os.chdir is replaced by monkeypatch.chdir. Filled lru_cache is cleared in fixture teardown with cache_clear. Committed database rows are avoided with a transaction that rolls back.",[367,588],{"x":369,"y":369,"width":370,"height":589,"rx":372,"fill":373},"236",[375,591,592],{"x":377,"y":378,"textAnchor":379,"fontSize":380,"fontWeight":381,"fill":382},"Every leak has a restoring twin",[367,594],{"x":595,"y":596,"width":597,"height":386,"rx":598,"fill":382},"26","44","748","7",[375,600,603],{"x":601,"y":429,"fontSize":602,"fontWeight":381,"fill":373},"46","11.5","leak",[375,605,606],{"x":377,"y":429,"fontSize":602,"fontWeight":381,"fill":373},"restoring replacement",[367,608],{"x":595,"y":609,"width":597,"height":378,"rx":610,"fill":397},"78","6",[375,612,614],{"x":601,"y":613,"fontSize":388,"fill":382},"97","os.environ[\"X\"] = …",[375,616,617],{"x":377,"y":613,"fontSize":388,"fill":452},"monkeypatch.setenv(\"X\", …)",[367,619],{"x":595,"y":620,"width":597,"height":378,"rx":610,"fill":373,"stroke":621},"110","rgba(61,64,91,0.14)",[375,623,513],{"x":601,"y":624,"fontSize":388,"fill":382},"129",[375,626,627],{"x":377,"y":624,"fontSize":388,"fill":452},"monkeypatch.setattr(settings, \"CURRENCY\", \"EUR\")",[367,629],{"x":595,"y":630,"width":597,"height":378,"rx":610,"fill":397},"142",[375,632,634],{"x":601,"y":633,"fontSize":388,"fill":382},"161","os.chdir(path)",[375,636,637],{"x":377,"y":633,"fontSize":388,"fill":452},"monkeypatch.chdir(path)",[367,639],{"x":595,"y":640,"width":597,"height":378,"rx":610,"fill":373,"stroke":621},"174",[375,642,644],{"x":601,"y":643,"fontSize":388,"fill":382},"193","@lru_cache filled as a side effect",[375,646,647],{"x":377,"y":643,"fontSize":388,"fill":452},"load_config.cache_clear() in teardown",[367,649],{"x":595,"y":650,"width":597,"height":651,"rx":610,"fill":397},"206","24",[375,653,655],{"x":601,"y":654,"fontSize":388,"fill":382},"223","committed database rows",[375,657,658],{"x":377,"y":654,"fontSize":388,"fill":452},"per-test transaction, rolled back",[461,660,661],{},"Fix the polluter, not the victim: the victim was relying, correctly, on a clean starting state.",[22,663,665],{"id":664},"preventing-order-dependencies-from-accumulating","Preventing order dependencies from accumulating",[10,667,668],{},"Bisection finds one polluter at a time. Keeping new ones out is cheaper, and three habits do most of the work.",[10,670,671,674,675,678,679,682],{},[494,672,673],{},"Randomise order everywhere."," ",[17,676,677],{},"pytest-randomly"," shuffles modules, classes and tests on every run and reseeds ",[17,680,681],{},"random"," per test. Installed as a development dependency, it makes order dependencies surface on the laptop of whoever introduces them, usually within a few runs, instead of months later in CI. The seed line at the top of the output makes every failure reproducible, so a shuffled failure costs one extra command rather than an investigation.",[10,684,685,688,689,693],{},[494,686,687],{},"Run each test in isolation occasionally."," A scheduled job that runs every test file on its own — or every test, for small suites — catches the opposite problem: tests that pass only ",[690,691,692],"em",{},"because"," an earlier test set something up. Those are order dependencies too, just with the roles reversed, and random ordering finds them less reliably because the helpful predecessor often still runs first by chance.",[10,695,696,699,700,703],{},[494,697,698],{},"Make global state visible."," An autouse fixture that snapshots ",[17,701,702],{},"os.environ",", the working directory and a short list of known application globals before each test, and compares after, turns silent leaks into immediate failures naming the test that leaked and the key that changed. It is a few lines of code and removes the need for bisection in most future cases, because the polluter reports itself.",[57,705,709],{"className":706,"code":707,"language":708,"meta":62,"style":62},"language-python shiki shiki-themes github-light github-dark","@pytest.fixture(autouse=True)\ndef _no_leaked_state():\n    env, cwd = dict(os.environ), os.getcwd()\n    yield\n    assert dict(os.environ) == env, \"test leaked environment changes\"\n    assert os.getcwd() == cwd, \"test changed the working directory\"\n","python",[17,710,711,716,721,726,731,736],{"__ignoreMap":62},[66,712,713],{"class":68,"line":69},[66,714,715],{},"@pytest.fixture(autouse=True)\n",[66,717,718],{"class":68,"line":76},[66,719,720],{},"def _no_leaked_state():\n",[66,722,723],{"class":68,"line":82},[66,724,725],{},"    env, cwd = dict(os.environ), os.getcwd()\n",[66,727,728],{"class":68,"line":116},[66,729,730],{},"    yield\n",[66,732,733],{"class":68,"line":123},[66,734,735],{},"    assert dict(os.environ) == env, \"test leaked environment changes\"\n",[66,737,738],{"class":68,"line":129},[66,739,740],{},"    assert os.getcwd() == cwd, \"test changed the working directory\"\n",[344,742,744,804],{"className":743},[347],[349,745,357,750,357,753,357,756,357,759,357,762,357,767,357,772,357,774,357,778,357,782,357,785,357,788,357,791,357,794,357,798,357,801],{"viewBox":746,"role":352,"ariaLabelledBy":747,"xmlns":356},"0 0 800 226",[748,749],"bip-t","bip-d",[359,751,752],{"id":748},"Three guards against order dependencies",[363,754,755],{"id":749},"Three cards describe guards. Random ordering with pytest-randomly surfaces polluters locally with a reproducible seed. Periodic isolated runs catch tests that only pass because of a helpful predecessor. A state-snapshot fixture fails the leaking test itself, naming what changed, so no bisection is needed.",[367,757],{"x":369,"y":369,"width":370,"height":758,"rx":372,"fill":373},"226",[375,760,761],{"x":377,"y":378,"textAnchor":379,"fontSize":380,"fontWeight":381,"fill":382},"Catch the next polluter without bisecting",[367,763],{"x":595,"y":764,"width":589,"height":765,"rx":766,"fill":421,"stroke":422,"strokeWidth":447},"50","152","12",[375,768,771],{"x":769,"y":609,"textAnchor":379,"fontSize":770,"fontWeight":381,"fill":382},"144","12.5","random order",[375,773,677],{"x":769,"y":620,"textAnchor":379,"fontSize":410,"fill":382},[375,775,777],{"x":769,"y":443,"textAnchor":379,"fontSize":410,"fontWeight":381,"fill":776},"#8a5a00","found locally, seed printed",[367,779],{"x":780,"y":764,"width":589,"height":765,"rx":766,"fill":397,"stroke":382,"strokeWidth":781},"282","1.5",[375,783,784],{"x":377,"y":609,"textAnchor":379,"fontSize":770,"fontWeight":381,"fill":382},"isolated runs",[375,786,787],{"x":377,"y":620,"textAnchor":379,"fontSize":410,"fill":382},"each file on its own",[375,789,790],{"x":377,"y":443,"textAnchor":379,"fontSize":410,"fontWeight":381,"fill":382},"catches helpful predecessors",[367,792],{"x":793,"y":764,"width":589,"height":765,"rx":766,"fill":445,"stroke":446,"strokeWidth":447},"538",[375,795,797],{"x":796,"y":609,"textAnchor":379,"fontSize":770,"fontWeight":381,"fill":382},"656","state snapshot",[375,799,800],{"x":796,"y":620,"textAnchor":379,"fontSize":410,"fill":382},"autouse before\u002Fafter check",[375,802,803],{"x":796,"y":443,"textAnchor":379,"fontSize":410,"fontWeight":381,"fill":452},"the polluter fails itself",[461,805,806],{},"With a snapshot fixture in place, the next leak fails in the test that caused it, not in some unrelated victim.",[22,808,810],{"id":809},"a-worked-example-the-currency-that-stuck","A worked example: the currency that stuck",[10,812,813,814,817,818,821,822,825,826,828,829,43],{},"In one suite, ",[17,815,816],{},"test_invoice_total"," failed roughly one run in six under random ordering, always asserting that a total of ",[17,819,820],{},"12.50"," was ",[17,823,824],{},"1250",". Re-running with the printed seed reproduced it every time, and the test passed on its own. ",[17,827,19],{}," took the 312 tests that preceded it and, after nine rounds, named ",[17,830,831],{},"tests\u002Ftest_settings.py::test_override_currency",[10,833,834,835,838],{},"The polluter was three lines long. It assigned ",[17,836,837],{},"settings.CURRENCY = \"JPY\""," directly to check that yen amounts render without decimals, asserted on the rendering, and ended. Japanese yen has no minor unit, so every later test in the same process that formatted money did so without decimals — including the invoice test, which then compared a string without a decimal point against its expectation.",[10,840,841,842,845,846,849],{},"The fix replaced the assignment with ",[17,843,844],{},"monkeypatch.setattr(settings, \"CURRENCY\", \"JPY\")",", which restores the original value at teardown. The invoice test was left untouched, because it was correct: it assumed the default currency, which is what the application uses. After the change, the same seed passed, and a hundred further random orders passed too. The state-snapshot fixture above was added in the same change, extended to compare ",[17,847,848],{},"settings.CURRENCY",", so that the next test to assign settings directly fails immediately and names itself. The whole investigation, from first reproduction to merged fix, took under an hour — most of it reading the polluter — because the bisection itself was automatic.",[22,851,853],{"id":852},"edge-cases-and-failure-modes","Edge cases and failure modes",[27,855,856,862,872,878,884],{},[30,857,858,861],{},[494,859,860],{},"Two polluters needed together."," Occasionally the victim fails only after two tests both run. Bisection stalls because neither half alone reproduces; run the halves combined in different pairs, or use a delta-debugging approach.",[30,863,864,867,868,871],{},[494,865,866],{},"The victim pollutes itself."," Running the same test twice in one process fails the second time. ",[17,869,870],{},"pytest --count=2"," from pytest-repeat on the victim alone detects this.",[30,873,874,877],{},[494,875,876],{},"Order-dependent collection."," Some leaks happen at import time, during collection, not in test bodies. Bisection over test ids still finds the module whose import pollutes.",[30,879,880,883],{},[494,881,882],{},"Failures only under xdist."," Replay the worker's sequence serially first; see the xdist guide below.",[30,885,886,889],{},[494,887,888],{},"Fixing the victim instead."," Making the victim reset global state hides the polluter and leaves every other test exposed. Fix the source.",[22,891,893],{"id":892},"frequently-asked-questions","Frequently Asked Questions",[10,895,896,899],{},[494,897,898],{},"What is a test order dependency?","\nA test whose outcome depends on which tests ran before it in the same process, because an earlier test changed shared state — a global, an environment variable, a cache, the working directory, a database row — and did not restore it.",[10,901,902,905,906,909],{},[494,903,904],{},"How do I reproduce a random-order failure?","\npytest-randomly prints the seed at the top of each run. Re-run with ",[17,907,908],{},"-p randomly --randomly-seed=\u003Cseed>"," to get the same order, then save the failing sequence of node ids.",[10,911,912,915],{},[494,913,914],{},"Is there a tool that finds the polluting test automatically?","\nYes. detect-test-pollution takes the failing test and a list of candidate tests, then bisects the candidates automatically until it finds the single test that makes the failing one fail when run before it.",[22,917,919],{"id":918},"related","Related",[27,921,922,928,935,942],{},[30,923,924,927],{},[48,925,926],{"href":50},"Debugging Tests in CI and Containers"," — CI-only failure strategy.",[30,929,930,934],{},[48,931,933],{"href":932},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fdebugging-a-test-that-only-fails-under-xdist\u002F","Debugging a Test That Only Fails Under xdist"," — parallel neighbours.",[30,936,937,941],{},[48,938,940],{"href":939},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Freproducing-ci-only-test-failures-locally\u002F","Reproducing CI-Only Test Failures Locally"," — matching the CI environment.",[30,943,944,948],{},[48,945,947],{"href":946},"\u002Fadvanced-pytest-architecture-configuration\u002Fmastering-pytest-fixtures\u002F","Mastering Pytest Fixtures"," — teardown that restores state.",[10,950,951,952],{},"← Back to ",[48,953,926],{"href":50},[955,956,957],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":62,"searchDepth":76,"depth":76,"links":959},[960,961,962,963,964,965,966,967,968],{"id":24,"depth":76,"text":25},{"id":54,"depth":76,"text":55},{"id":466,"depth":76,"text":467},{"id":484,"depth":76,"text":485},{"id":664,"depth":76,"text":665},{"id":809,"depth":76,"text":810},{"id":852,"depth":76,"text":853},{"id":892,"depth":76,"text":893},{"id":918,"depth":76,"text":919},"Find the test that breaks another only when it runs first: reproducing an order with pytest-randomly seeds, bisecting the preceding tests, detect-test-pollution, and fixing the leaked state.","md",{"slug":972,"type":973,"breadcrumb":974,"datePublished":975,"dateModified":975,"faq":976,"howto":983},"bisecting-test-order-dependencies","article","Order dependencies","2026-09-18",[977,979,981],{"q":898,"a":978},"A test whose outcome depends on which tests ran before it in the same process, because an earlier test changed shared state — a global, an environment variable, a cache, the working directory, a database row — and did not restore it.",{"q":904,"a":980},"pytest-randomly prints the seed at the top of each run. Re-run with -p randomly --randomly-seed=\u003Cseed> to get the same order, then save the failing sequence of node ids.",{"q":914,"a":982},"Yes. detect-test-pollution takes the failing test and a list of candidate tests, then bisects the candidates automatically until it finds the single test that makes the failing one fail when run before it.",{"name":984,"description":985,"steps":986},"How to bisect a test order dependency","Reproduce the failing order, confirm the victim passes alone, then halve the preceding tests until one polluter remains.",[987,990,993,996],{"name":988,"text":989},"Reproduce the order","Re-run with the same randomisation seed and save the node ids up to the failing test.",{"name":991,"text":992},"Confirm the victim passes alone","Run the failing test by itself; if it passes, an earlier test is interfering.",{"name":994,"text":995},"Bisect the predecessors","Run half of the preceding tests followed by the victim; keep whichever half still causes the failure.",{"name":997,"text":998},"Fix the polluter","Find the state the polluter leaves behind and restore it with monkeypatch or a teardown.","\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fbisecting-test-order-dependencies",{"title":5,"description":969},"systematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fbisecting-test-order-dependencies\u002Findex","AVwJeV4z_RzyZsWE6MUHcfazx2kn4ubLNRNJ37Gpbw8",1789718765723]