[{"data":1,"prerenderedAt":1005},["ShallowReactive",2],{"page-\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fdebugging-a-test-that-only-fails-under-xdist\u002F":3},{"id":4,"title":5,"body":6,"description":968,"extension":969,"meta":970,"navigation":128,"path":1001,"seo":1002,"stem":1003,"__hash__":1004},"content\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fdebugging-a-test-that-only-fails-under-xdist\u002Findex.md","Debugging a Test That Only Fails Under xdist",{"type":7,"value":8,"toc":957},"minimark",[9,18,21,26,52,56,214,372,498,502,516,523,545,549,560,566,644,648,651,677,683,692,701,711,786,790,808,833,837,883,887,893,909,915,919,948,953],[10,11,12,13,17],"p",{},"A test that passes on its own and in a serial run, and fails only with ",[14,15,16],"code",{},"pytest -n auto",", is telling you it depends on something the parallel run changes. The list of such things is short. Two workers can reach for the same external resource at the same moment — a file at a fixed path, a TCP port, a database or schema name, a cache directory. A session-scoped fixture that \"runs once\" actually runs once per worker, so setup that assumes exclusivity collides with itself. And the tests that share a worker process are a different, partly random set from the serial run, so leaked global state from a neighbour now reaches a test it never reached before.",[10,19,20],{},"Each cause has a recognisable signature and a direct fix. The hard part is reproduction: the failure depends on timing and on which tests landed on which worker. Logging the worker assignment and replaying one worker's sequence serially turns most xdist-only failures into ordinary, deterministic ones.",[22,23,25],"h2",{"id":24},"prerequisites","Prerequisites",[27,28,29,44],"ul",{},[30,31,32,35,36,39,40,43],"li",{},[14,33,34],{},"pytest >= 8.0",", ",[14,37,38],{},"pytest-xdist >= 3.5",", optionally ",[14,41,42],{},"filelock",".",[30,45,46,47,43],{},"Background from ",[48,49,51],"a",{"href":50},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002F","Debugging tests in CI and containers",[22,53,55],{"id":54},"solution","Solution",[57,58,63],"pre",{"className":59,"code":60,"language":61,"meta":62,"style":62},"language-bash shiki shiki-themes github-light github-dark","# 1. See which worker ran what, in a stable order.\npytest -n 4 -v -p no:randomly 2>&1 | tee xdist.log\n#   [gw2] PASSED tests\u002Ftest_export.py::test_writes_report\n#   [gw2] FAILED tests\u002Ftest_export.py::test_report_is_valid_csv\n\n# 2. Replay gw2's sequence serially.\ngrep '^\\[gw2\\]' xdist.log | awk '{print $3}' > gw2.txt\npytest -p no:randomly $(cat gw2.txt)\n\n# 3. Or test for a resource collision: run the one test on many workers at once.\npytest -n 8 --count=16 tests\u002Ftest_export.py::test_report_is_valid_csv   # pytest-repeat\n","bash","",[14,64,65,74,111,117,123,130,136,162,184,189,195],{"__ignoreMap":62},[66,67,70],"span",{"class":68,"line":69},"line",1,[66,71,73],{"class":72},"sJ8bj","# 1. See which worker ran what, in a stable order.\n",[66,75,77,81,85,88,91,94,98,102,105,108],{"class":68,"line":76},2,[66,78,80],{"class":79},"sScJk","pytest",[66,82,84],{"class":83},"sj4cs"," -n",[66,86,87],{"class":83}," 4",[66,89,90],{"class":83}," -v",[66,92,93],{"class":83}," -p",[66,95,97],{"class":96},"sZZnC"," no:randomly",[66,99,101],{"class":100},"szBVR"," 2>&1",[66,103,104],{"class":100}," |",[66,106,107],{"class":79}," tee",[66,109,110],{"class":96}," xdist.log\n",[66,112,114],{"class":68,"line":113},3,[66,115,116],{"class":72},"#   [gw2] PASSED tests\u002Ftest_export.py::test_writes_report\n",[66,118,120],{"class":68,"line":119},4,[66,121,122],{"class":72},"#   [gw2] FAILED tests\u002Ftest_export.py::test_report_is_valid_csv\n",[66,124,126],{"class":68,"line":125},5,[66,127,129],{"emptyLinePlaceholder":128},true,"\n",[66,131,133],{"class":68,"line":132},6,[66,134,135],{"class":72},"# 2. Replay gw2's sequence serially.\n",[66,137,139,142,145,148,150,153,156,159],{"class":68,"line":138},7,[66,140,141],{"class":79},"grep",[66,143,144],{"class":96}," '^\\[gw2\\]'",[66,146,147],{"class":96}," xdist.log",[66,149,104],{"class":100},[66,151,152],{"class":79}," awk",[66,154,155],{"class":96}," '{print $3}'",[66,157,158],{"class":100}," >",[66,160,161],{"class":96}," gw2.txt\n",[66,163,165,167,169,171,175,178,181],{"class":68,"line":164},8,[66,166,80],{"class":79},[66,168,93],{"class":83},[66,170,97],{"class":96},[66,172,174],{"class":173},"sVt8B"," $(",[66,176,177],{"class":79},"cat",[66,179,180],{"class":96}," gw2.txt",[66,182,183],{"class":173},")\n",[66,185,187],{"class":68,"line":186},9,[66,188,129],{"emptyLinePlaceholder":128},[66,190,192],{"class":68,"line":191},10,[66,193,194],{"class":72},"# 3. Or test for a resource collision: run the one test on many workers at once.\n",[66,196,198,200,202,205,208,211],{"class":68,"line":197},11,[66,199,80],{"class":79},[66,201,84],{"class":83},[66,203,204],{"class":83}," 8",[66,206,207],{"class":83}," --count=16",[66,209,210],{"class":96}," tests\u002Ftest_export.py::test_report_is_valid_csv",[66,212,213],{"class":72},"   # pytest-repeat\n",[57,215,219],{"className":216,"code":217,"language":218,"meta":62,"style":62},"language-python shiki shiki-themes github-light github-dark","# Parallel-safe replacements for the usual collisions.\nimport socket\nimport pytest\nfrom filelock import FileLock\n\n@pytest.fixture\ndef report_path(tmp_path):\n    return tmp_path \u002F \"report.csv\"            # not Path(\"\u002Ftmp\u002Freport.csv\")\n\n@pytest.fixture\ndef free_port():\n    with socket.socket() as s:\n        s.bind((\"127.0.0.1\", 0))\n        return s.getsockname()[1]              # not a hard-coded 8080\n\n@pytest.fixture(scope=\"session\")\ndef db_name(worker_id):\n    return f\"test_app_{worker_id}\"             # gw0, gw1… or \"master\" without xdist\n\n@pytest.fixture(scope=\"session\")\ndef shared_fixture_data(tmp_path_factory, worker_id):\n    root = tmp_path_factory.getbasetemp().parent   # shared across workers\n    marker = root \u002F \"fixture-data.ready\"\n    with FileLock(str(root \u002F \"fixture-data.lock\")):\n        if not marker.exists():\n            build_expensive_fixture_data(root \u002F \"fixture-data\")\n            marker.touch()\n    return root \u002F \"fixture-data\"\n","python",[14,220,221,226,231,236,241,245,250,255,260,264,268,273,279,285,291,296,302,308,314,319,324,330,336,342,348,354,360,366],{"__ignoreMap":62},[66,222,223],{"class":68,"line":69},[66,224,225],{},"# Parallel-safe replacements for the usual collisions.\n",[66,227,228],{"class":68,"line":76},[66,229,230],{},"import socket\n",[66,232,233],{"class":68,"line":113},[66,234,235],{},"import pytest\n",[66,237,238],{"class":68,"line":119},[66,239,240],{},"from filelock import FileLock\n",[66,242,243],{"class":68,"line":125},[66,244,129],{"emptyLinePlaceholder":128},[66,246,247],{"class":68,"line":132},[66,248,249],{},"@pytest.fixture\n",[66,251,252],{"class":68,"line":138},[66,253,254],{},"def report_path(tmp_path):\n",[66,256,257],{"class":68,"line":164},[66,258,259],{},"    return tmp_path \u002F \"report.csv\"            # not Path(\"\u002Ftmp\u002Freport.csv\")\n",[66,261,262],{"class":68,"line":186},[66,263,129],{"emptyLinePlaceholder":128},[66,265,266],{"class":68,"line":191},[66,267,249],{},[66,269,270],{"class":68,"line":197},[66,271,272],{},"def free_port():\n",[66,274,276],{"class":68,"line":275},12,[66,277,278],{},"    with socket.socket() as s:\n",[66,280,282],{"class":68,"line":281},13,[66,283,284],{},"        s.bind((\"127.0.0.1\", 0))\n",[66,286,288],{"class":68,"line":287},14,[66,289,290],{},"        return s.getsockname()[1]              # not a hard-coded 8080\n",[66,292,294],{"class":68,"line":293},15,[66,295,129],{"emptyLinePlaceholder":128},[66,297,299],{"class":68,"line":298},16,[66,300,301],{},"@pytest.fixture(scope=\"session\")\n",[66,303,305],{"class":68,"line":304},17,[66,306,307],{},"def db_name(worker_id):\n",[66,309,311],{"class":68,"line":310},18,[66,312,313],{},"    return f\"test_app_{worker_id}\"             # gw0, gw1… or \"master\" without xdist\n",[66,315,317],{"class":68,"line":316},19,[66,318,129],{"emptyLinePlaceholder":128},[66,320,322],{"class":68,"line":321},20,[66,323,301],{},[66,325,327],{"class":68,"line":326},21,[66,328,329],{},"def shared_fixture_data(tmp_path_factory, worker_id):\n",[66,331,333],{"class":68,"line":332},22,[66,334,335],{},"    root = tmp_path_factory.getbasetemp().parent   # shared across workers\n",[66,337,339],{"class":68,"line":338},23,[66,340,341],{},"    marker = root \u002F \"fixture-data.ready\"\n",[66,343,345],{"class":68,"line":344},24,[66,346,347],{},"    with FileLock(str(root \u002F \"fixture-data.lock\")):\n",[66,349,351],{"class":68,"line":350},25,[66,352,353],{},"        if not marker.exists():\n",[66,355,357],{"class":68,"line":356},26,[66,358,359],{},"            build_expensive_fixture_data(root \u002F \"fixture-data\")\n",[66,361,363],{"class":68,"line":362},27,[66,364,365],{},"            marker.touch()\n",[66,367,369],{"class":68,"line":368},28,[66,370,371],{},"    return root \u002F \"fixture-data\"\n",[373,374,377,494],"figure",{"className":375},[376],"diagram",[378,379,386,387,386,391,386,395,386,403,386,413,386,423,386,429,386,435,386,439,386,445,386,450,386,455,386,458,386,462,386,465,386,469,386,472,386,477,386,481,386,485,386,488,386,491],"svg",{"viewBox":380,"role":381,"ariaLabelledBy":382,"xmlns":385},"0 0 800 256","img",[383,384],"xd-t","xd-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[388,389,390],"title",{"id":383},"Three causes of xdist-only failures",[392,393,394],"desc",{"id":384},"Three columns describe causes. Resource collisions: two workers use the same fixed path, port or database at the same moment; signature is failures on different tests each run. Per-worker session fixtures: setup that assumes it runs once runs once per worker and collides with itself; signature is errors in fixture setup. Neighbour state: a test leaks global state to whichever test follows it on the same worker; signature is reproducible by replaying the worker's sequence.",[396,397],"rect",{"x":398,"y":398,"width":399,"height":400,"rx":401,"fill":402},"0","800","256","14","#fffdf8",[404,405,412],"text",{"x":406,"y":407,"textAnchor":408,"fontSize":409,"fontWeight":410,"fill":411},"400","28","middle","15.5","700","#3d405b","What parallelism changes, and how each shows up",[396,414],{"x":415,"y":416,"width":417,"height":418,"rx":419,"fill":420,"stroke":421,"strokeWidth":422},"26","50","236","184","12","#fbe9e3","#e07a5f","2",[404,424,428],{"x":425,"y":426,"textAnchor":408,"fontSize":427,"fontWeight":410,"fill":411},"144","78","12.5","resource collision",[404,430,434],{"x":431,"y":432,"fontSize":433,"fill":411},"44","108","11","same path · port · db name",[404,436,438],{"x":431,"y":437,"fontSize":433,"fill":411},"130","at the same moment",[404,440,444],{"x":431,"y":441,"fontSize":442,"fill":443},"176","10.5","#8f3d22","different test fails each run",[404,446,449],{"x":431,"y":447,"fontSize":433,"fontWeight":410,"fill":448},"210","#2a5f49","fix: tmp_path, port 0",[396,451],{"x":452,"y":416,"width":417,"height":418,"rx":419,"fill":453,"stroke":454,"strokeWidth":422},"282","#f7f0da","#f2cc8f",[404,456,457],{"x":406,"y":426,"textAnchor":408,"fontSize":427,"fontWeight":410,"fill":411},"session fixture x N",[404,459,461],{"x":460,"y":432,"fontSize":433,"fill":411},"300","\"once\" means once per worker",[404,463,464],{"x":460,"y":437,"fontSize":433,"fill":411},"setup collides with itself",[404,466,468],{"x":460,"y":441,"fontSize":442,"fill":467},"#8a5a00","errors in fixture setup",[404,470,471],{"x":460,"y":447,"fontSize":433,"fontWeight":410,"fill":448},"fix: worker_id, FileLock",[396,473],{"x":474,"y":416,"width":417,"height":418,"rx":419,"fill":475,"stroke":476,"strokeWidth":422},"538","#e6f0ea","#81b29a",[404,478,480],{"x":479,"y":426,"textAnchor":408,"fontSize":427,"fontWeight":410,"fill":411},"656","neighbour state",[404,482,484],{"x":483,"y":432,"fontSize":433,"fill":411},"556","new set of tests share",[404,486,487],{"x":483,"y":437,"fontSize":433,"fill":411},"a process with it",[404,489,490],{"x":483,"y":441,"fontSize":442,"fill":448},"replays serially",[404,492,493],{"x":483,"y":447,"fontSize":433,"fontWeight":410,"fill":448},"fix: reset globals",[495,496,497],"figcaption",{},"The signature usually identifies the column: nondeterministic victims suggest a collision, setup errors a fixture, and reproducible replays a neighbour.",[22,499,501],{"id":500},"why-this-works","Why this works",[10,503,504,507,508,511,512,515],{},[14,505,506],{},"-v"," output under xdist prefixes each result with the worker id, and ",[14,509,510],{},"-p no:randomly"," removes one source of variation so the same command produces the same assignment more often. The default ",[14,513,514],{},"--dist load"," scheduler hands out tests in small chunks, so assignment still depends on timing, but within one run the log records exactly which tests each worker ran and in which order. Replaying that sequence serially recreates the neighbour relationships without the parallelism. If the failure reproduces, the cause is state left by an earlier test on the same worker; if it does not, the cause needs concurrency — a resource collision.",[10,517,518,519,522],{},"Running a single test many times across many workers isolates collisions. A test that uses ",[14,520,521],{},"\u002Ftmp\u002Freport.csv"," will pass alone and fail as soon as two copies run at the same moment, because one worker reads the file while another truncates it. A test that is safe passes all sixteen repetitions.",[10,524,525,526,529,530,35,533,536,537,540,541,544],{},"The ",[14,527,528],{},"worker_id"," fixture, provided by xdist, returns ",[14,531,532],{},"gw0",[14,534,535],{},"gw1"," and so on — or ",[14,538,539],{},"master"," when xdist is not active — which makes it the natural key for per-worker resources. ",[14,542,543],{},"tmp_path_factory.getbasetemp().parent"," is a directory shared by all workers in one run, which gives the file lock and the ready-marker somewhere common to live.",[22,546,548],{"id":547},"recognising-the-neighbour-case","Recognising the neighbour case",[10,550,551,552,555,556,559],{},"When replaying a worker's sequence reproduces the failure, bisect it. Remove the first half of the preceding tests and re-run; if it still fails, the culprit is in the second half. Within a handful of runs you have a pair: test A, then test B fails. What A leaves behind is typically one of a small set: a module-level cache or singleton it populated, an environment variable it set with ",[14,553,554],{},"os.environ"," instead of ",[14,557,558],{},"monkeypatch",", a patched attribute it never restored, the current working directory it changed, or a logging handler it attached.",[10,561,562,563,565],{},"The fix belongs in A, not B. Replace direct mutation with ",[14,564,558],{}," so pytest restores it, clear caches in a fixture teardown, and treat any global state the application exposes as something tests must reset. Once fixed, the pair passes in both orders, and the suite no longer depends on which tests happen to share a process.",[373,567,569,641],{"className":568},[376],[378,570,386,575,386,578,386,581,386,584,386,587,386,593,386,599,386,604,386,609,386,613,386,616,386,620,386,624,386,627,386,630,386,633,386,637],{"viewBox":571,"role":381,"ariaLabelledBy":572,"xmlns":385},"0 0 800 226",[573,574],"xdn-t","xdn-d",[388,576,577],{"id":573},"A leaked global reaching a new neighbour",[392,579,580],{"id":574},"In the serial run, test A runs early and test B much later after a teardown that happens to reset the state. Under xdist, test A and test B land consecutively on worker gw2, so the global state A leaves behind reaches B directly and B fails.",[396,582],{"x":398,"y":398,"width":399,"height":583,"rx":401,"fill":402},"226",[404,585,586],{"x":406,"y":407,"textAnchor":408,"fontSize":409,"fontWeight":410,"fill":411},"Same tests, new neighbours",[404,588,592],{"x":589,"y":590,"fontSize":591,"fontWeight":410,"fill":411},"30","80","11.5","serial",[396,594],{"x":595,"y":596,"width":590,"height":415,"rx":597,"fill":453,"stroke":454,"strokeWidth":598},"120","64","5","1.6",[404,600,603],{"x":601,"y":602,"textAnchor":408,"fontSize":433,"fill":411},"160","82","A",[396,605],{"x":447,"y":596,"width":606,"height":415,"rx":597,"fill":607,"stroke":608},"360","#f4f1de","rgba(61,64,91,0.35)",[404,610,612],{"x":611,"y":602,"textAnchor":408,"fontSize":442,"fill":411},"390","200 tests, one resets the cache",[396,614],{"x":615,"y":596,"width":590,"height":415,"rx":597,"fill":475,"stroke":476,"strokeWidth":598},"580",[404,617,619],{"x":618,"y":602,"textAnchor":408,"fontSize":433,"fill":448},"620","B passes",[404,621,623],{"x":589,"y":622,"fontSize":591,"fontWeight":410,"fill":411},"150","gw2",[396,625],{"x":595,"y":626,"width":590,"height":415,"rx":597,"fill":453,"stroke":454,"strokeWidth":598},"134",[404,628,603],{"x":601,"y":629,"textAnchor":408,"fontSize":433,"fill":411},"152",[396,631],{"x":447,"y":626,"width":590,"height":415,"rx":597,"fill":420,"stroke":421,"strokeWidth":632},"1.8",[404,634,636],{"x":635,"y":629,"textAnchor":408,"fontSize":433,"fill":443},"250","B fails",[404,638,640],{"x":406,"y":639,"textAnchor":408,"fontSize":433,"fill":411},"200","Fix A's leak with monkeypatch or teardown; B was only the victim.",[495,642,643],{},"xdist did not create the bug; it removed the accidental reset that used to hide it.",[22,645,647],{"id":646},"making-a-suite-parallel-safe-by-default","Making a suite parallel-safe by default",[10,649,650],{},"Fixing xdist failures one at a time works, but a few conventions stop most of them from appearing in the first place. They are cheap to adopt and easy to enforce in review.",[10,652,653,657,658,661,662,665,666,669,670,673,674,676],{},[654,655,656],"strong",{},"No fixed paths."," Every file a test writes goes under ",[14,659,660],{},"tmp_path"," or ",[14,663,664],{},"tmp_path_factory",". A quick search for string literals starting with ",[14,667,668],{},"\u002Ftmp"," or relative paths like ",[14,671,672],{},"\"output\u002F\""," in the test directory finds most violations. Application code that writes to a configurable directory should have that directory pointed at ",[14,675,660],{}," by a fixture.",[10,678,679,682],{},[654,680,681],{},"No fixed ports."," Servers started in tests bind to port 0 and report the port the operating system chose. Clients get the port from the fixture, never from a constant.",[10,684,685,688,689,691],{},[654,686,687],{},"Per-worker names for shared services."," Databases, schemas, message-broker queues, cache key prefixes and S3 buckets used by tests include the ",[14,690,528],{},". With one PostgreSQL container per CI job, a database per worker keeps workers from seeing each other's data while sharing the expensive container.",[10,693,694,697,698,700],{},[654,695,696],{},"No unguarded global mutation."," Environment variables, module attributes, the working directory and singletons are changed through ",[14,699,558],{},", which restores them at teardown. Application-level caches get an explicit reset that an autouse fixture calls.",[10,702,703,706,707,710],{},[654,704,705],{},"Random order locally."," ",[14,708,709],{},"pytest-randomly"," in the development environment shuffles test order on every run, which surfaces neighbour dependencies on developers' machines long before xdist rearranges them in CI. When it finds one, the printed seed reproduces the order exactly.",[373,712,714,783],{"className":713},[376],[378,715,386,720,386,723,386,726,386,728,386,731,386,735,386,738,386,742,386,744,386,748,386,751,386,754,386,757,386,760,386,763,386,766,386,769,386,773,386,777,386,780],{"viewBox":716,"role":381,"ariaLabelledBy":717,"xmlns":385},"0 0 800 236",[718,719],"xdc-t","xdc-d",[388,721,722],{"id":718},"Conventions that keep tests parallel-safe",[392,724,725],{"id":719},"Five conventions are listed as cards: files under tmp_path, servers on port zero, per-worker names for databases and queues, globals changed only through monkeypatch, and random test order locally with pytest-randomly. Each prevents one class of xdist-only failure.",[396,727],{"x":398,"y":398,"width":399,"height":417,"rx":401,"fill":402},[404,729,730],{"x":406,"y":407,"textAnchor":408,"fontSize":409,"fontWeight":410,"fill":411},"Five habits, most xdist failures prevented",[396,732],{"x":733,"y":734,"width":425,"height":601,"rx":433,"fill":475,"stroke":476,"strokeWidth":632},"20","52",[404,736,660],{"x":737,"y":590,"textAnchor":408,"fontSize":591,"fontWeight":410,"fill":411},"92",[404,739,741],{"x":737,"y":740,"textAnchor":408,"fontSize":442,"fill":411},"114","no fixed files",[396,743],{"x":441,"y":734,"width":425,"height":601,"rx":433,"fill":475,"stroke":476,"strokeWidth":632},[404,745,747],{"x":746,"y":590,"textAnchor":408,"fontSize":591,"fontWeight":410,"fill":411},"248","port 0",[404,749,750],{"x":746,"y":740,"textAnchor":408,"fontSize":442,"fill":411},"no fixed ports",[396,752],{"x":753,"y":734,"width":425,"height":601,"rx":433,"fill":453,"stroke":454,"strokeWidth":632},"332",[404,755,528],{"x":756,"y":590,"textAnchor":408,"fontSize":591,"fontWeight":410,"fill":411},"404",[404,758,759],{"x":756,"y":740,"textAnchor":408,"fontSize":442,"fill":411},"dbs · queues · keys",[396,761],{"x":762,"y":734,"width":425,"height":601,"rx":433,"fill":453,"stroke":454,"strokeWidth":632},"488",[404,764,558],{"x":765,"y":590,"textAnchor":408,"fontSize":591,"fontWeight":410,"fill":411},"560",[404,767,768],{"x":765,"y":740,"textAnchor":408,"fontSize":442,"fill":411},"globals restored",[396,770],{"x":771,"y":734,"width":772,"height":601,"rx":433,"fill":420,"stroke":421,"strokeWidth":632},"644","136",[404,774,776],{"x":775,"y":590,"textAnchor":408,"fontSize":591,"fontWeight":410,"fill":411},"712","randomly",[404,778,779],{"x":775,"y":740,"textAnchor":408,"fontSize":442,"fill":411},"order bugs found",[404,781,782],{"x":775,"y":626,"textAnchor":408,"fontSize":442,"fill":411},"locally",[495,784,785],{},"The first four remove shared resources and state; the fifth finds whatever slips through, on a laptop instead of in CI.",[22,787,789],{"id":788},"when-the-failure-is-a-crash-not-an-assertion","When the failure is a crash, not an assertion",[10,791,792,793,796,797,800,801,661,804,807],{},"Sometimes the symptom under xdist is not a failing assertion but a worker dying: ",[14,794,795],{},"[gw3] node down: Not properly terminated"," followed by ",[14,798,799],{},"replacing crashed worker gw3",". xdist restarts the worker and carries on, and the test that was running is reported as failed with little detail. The usual causes are a segfault in a C extension, the out-of-memory killer reaping a worker that grew too large, or a test that calls ",[14,802,803],{},"os._exit",[14,805,806],{},"sys.exit"," in a subprocess path that turned out to be the worker itself.",[10,809,810,811,814,815,661,818,555,821,824,825,828,829,832],{},"Memory is the most common in CI. Each worker holds its own copy of everything the tests import and cache, so ",[14,812,813],{},"-n auto"," on a runner with many cores and little memory can exceed the limit even when a serial run fits comfortably. Check the runner's kernel log or the CI job's memory graph around the crash, and try ",[14,816,817],{},"-n 2",[14,819,820],{},"-n 4",[14,822,823],{},"auto",". For segfaults, enable ",[14,826,827],{},"faulthandler"," output to a file per worker — ",[14,830,831],{},"-p faulthandler"," is on by default, but its output goes to the worker's stderr, which xdist may not show — and run the crashed worker's sequence serially to reproduce it with a full stack.",[22,834,836],{"id":835},"edge-cases-and-failure-modes","Edge cases and failure modes",[27,838,839,849,861,871,877],{},[30,840,841,844,845,848],{},[654,842,843],{},"Output ordering."," Worker output interleaves; never infer order from the combined log without the ",[14,846,847],{},"[gwN]"," prefixes.",[30,850,851,860],{},[654,852,853,856,857,43],{},[14,854,855],{},"--dist loadscope"," and ",[14,858,859],{},"loadfile"," These keep a module or class on one worker, which can hide or reveal neighbour issues. Reproduce with the same mode CI uses.",[30,862,863,866,867,870],{},[654,864,865],{},"Databases."," One database per worker (",[14,868,869],{},"test_app_{worker_id}",") is simplest; per-test transactions inside it keep tests independent.",[30,872,873,876],{},[654,874,875],{},"Environment variables."," Workers inherit the parent environment at startup. Changes made by one worker are invisible to others, which can mask or create differences from serial runs.",[30,878,879,882],{},[654,880,881],{},"Coverage and plugins."," Plugins that write to fixed paths — coverage data, reports — need parallel-aware configuration; see the coverage guide below.",[22,884,886],{"id":885},"frequently-asked-questions","Frequently Asked Questions",[10,888,889,892],{},[654,890,891],{},"Why does a test pass alone but fail with pytest -n auto?","\nUnder xdist the test runs in a different process, next to different neighbours, possibly at the same moment as tests on other workers. Failures come from shared external resources such as files, ports and databases, from session fixtures running once per worker, or from state left by whichever tests shared its worker.",[10,894,895,898,899,901,902,905,906,908],{},[654,896,897],{},"How do I reproduce the exact order a worker ran tests in?","\nRun with ",[14,900,506],{}," to log which worker ran each test, or use ",[14,903,904],{},"--dist loadfile"," to keep each module's tests together on one worker. Then run that worker's tests serially in the same order with ",[14,907,510],{}," and the node ids listed explicitly.",[10,910,911,914],{},[654,912,913],{},"Do session-scoped fixtures run once under xdist?","\nNo. Each worker is a separate process with its own session, so a session fixture runs once per worker. Fixtures that create shared external resources must coordinate across workers, for example with a file lock, or be made per-worker.",[22,916,918],{"id":917},"related","Related",[27,920,921,927,934,941],{},[30,922,923,926],{},[48,924,925],{"href":50},"Debugging Tests in CI and Containers"," — CI-only failure strategy.",[30,928,929,933],{},[48,930,932],{"href":931},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fbisecting-test-order-dependencies\u002F","Bisecting Test Order Dependencies"," — finding the interfering test.",[30,935,936,940],{},[48,937,939],{"href":938},"\u002Fadvanced-pytest-architecture-configuration\u002Foptimizing-test-discovery\u002Fpytest-xdist-vs-pytest-parallel-performance-comparison\u002F","pytest-xdist vs pytest-parallel Performance"," — how xdist schedules work.",[30,942,943,947],{},[48,944,946],{"href":945},"\u002Fadvanced-pytest-architecture-configuration\u002Fcoverage-measurement-and-enforcement\u002Fwhy-pytest-cov-reports-zero-under-xdist\u002F","Why pytest-cov Reports Zero Under xdist"," — plugins and parallelism.",[10,949,950,951],{},"← Back to ",[48,952,925],{"href":50},[954,955,956],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":62,"searchDepth":76,"depth":76,"links":958},[959,960,961,962,963,964,965,966,967],{"id":24,"depth":76,"text":25},{"id":54,"depth":76,"text":55},{"id":500,"depth":76,"text":501},{"id":547,"depth":76,"text":548},{"id":646,"depth":76,"text":647},{"id":788,"depth":76,"text":789},{"id":835,"depth":76,"text":836},{"id":885,"depth":76,"text":886},{"id":917,"depth":76,"text":918},"Diagnose pytest failures that appear only with -n: reproducing a worker's exact test sequence, shared files and ports, session fixtures running per worker, global state, and making tests parallel-safe.","md",{"slug":971,"type":972,"breadcrumb":973,"datePublished":974,"dateModified":974,"faq":975,"howto":982},"debugging-a-test-that-only-fails-under-xdist","article","xdist-only failures","2026-09-18",[976,978,980],{"q":891,"a":977},"Under xdist the test runs in a different process, next to different neighbours, possibly at the same moment as tests on other workers. Failures come from shared external resources such as files, ports and databases, from session fixtures running once per worker, or from state left by whichever tests shared its worker.",{"q":897,"a":979},"Run with -v to log which worker ran each test, or use --dist loadfile to keep each module's tests together on one worker. Then run that worker's tests serially in the same order with -p no:randomly and the node ids listed explicitly.",{"q":913,"a":981},"No. Each worker is a separate process with its own session, so a session fixture runs once per worker. Fixtures that create shared external resources must coordinate across workers, for example with a file lock, or be made per-worker.",{"name":983,"description":984,"steps":985},"How to debug a test that only fails under xdist","Classify the failure as a resource collision, a per-worker fixture assumption or a neighbour interaction, and reproduce it deterministically.",[986,989,992,995,998],{"name":987,"text":988},"Log worker assignment","Run with -v and -p no:randomly to see which worker ran each test and in what order.",{"name":990,"text":991},"Check for shared resources","Look for fixed file paths, ports, database names or environment variables used by several tests.",{"name":993,"text":994},"Check session fixtures","Find session fixtures that create external state and assume they run once.",{"name":996,"text":997},"Replay the worker's sequence","Run the same tests serially in the same order to find an interfering neighbour.",{"name":999,"text":1000},"Make it parallel-safe","Use tmp_path, worker_id-derived names, ephemeral ports and file locks.","\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fdebugging-a-test-that-only-fails-under-xdist",{"title":5,"description":968},"systematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fdebugging-a-test-that-only-fails-under-xdist\u002Findex","PKfDRFgdgiJvozofSaUjJc_r_2Kp69jDCTxyG9hzvBg",1789718765724]