[{"data":1,"prerenderedAt":939},["ShallowReactive",2],{"page-\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Freproducing-ci-only-test-failures-locally\u002F":3},{"id":4,"title":5,"body":6,"description":905,"extension":906,"meta":907,"navigation":724,"path":935,"seo":936,"stem":937,"__hash__":938},"content\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Freproducing-ci-only-test-failures-locally\u002Findex.md","Reproducing CI-Only Test Failures Locally",{"type":7,"value":8,"toc":894},"minimark",[9,13,18,34,38,41,48,72,93,103,154,163,169,184,190,199,388,392,395,497,501,508,528,535,550,565,569,572,582,588,598,605,609,662,666,669,678,693,702,708,746,749,753,759,768,781,852,856,884,890],[10,11,12],"p",{},"A failure that only happens in CI is not mysterious; it is a failure whose cause is in an environment dimension you have not matched yet. There are about six such dimensions, they can be enumerated, and working through them in order turns the investigation from guesswork into a checklist that usually terminates in ten minutes.",[14,15,17],"h2",{"id":16},"prerequisites","Prerequisites",[19,20,21,31],"ul",{},[22,23,24,25,30],"li",{},"The artefact bundle from the failing run — log, junit XML and environment snapshot, as set up in ",[26,27,29],"a",{"href":28},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002F","debugging tests in CI and containers",".",[22,32,33],{},"Docker or Podman, and the CI image reference.",[14,35,37],{"id":36},"solution","Solution",[10,39,40],{},"Work down the list. Each step is one command and each eliminates a class of cause.",[10,42,43,47],{},[44,45,46],"strong",{},"1. The same image, pinned by digest."," A tag moves; a digest does not, and \"it works in the image\" is meaningless if the image is not the one CI ran.",[49,50,55],"pre",{"className":51,"code":52,"language":53,"meta":54,"style":54},"language-console shiki shiki-themes github-light github-dark","$ docker run --rm -it -v \"$PWD:\u002Fsrc\" -w \u002Fsrc \\\n    ghcr.io\u002Facme\u002Fci-python@sha256:6b1c... bash\n","console","",[56,57,58,66],"code",{"__ignoreMap":54},[59,60,63],"span",{"class":61,"line":62},"line",1,[59,64,65],{},"$ docker run --rm -it -v \"$PWD:\u002Fsrc\" -w \u002Fsrc \\\n",[59,67,69],{"class":61,"line":68},2,[59,70,71],{},"    ghcr.io\u002Facme\u002Fci-python@sha256:6b1c... bash\n",[10,73,74,77,78,81,82,81,85,88,89,92],{},[44,75,76],{},"2. The same command."," Copy it verbatim from the workflow file, including every flag. ",[56,79,80],{},"-n auto",", ",[56,83,84],{},"-p no:randomly",[56,86,87],{},"--dist loadfile"," and ",[56,90,91],{},"-o"," overrides all change behaviour, and a locally-typed approximation is the most common reason a reproduction attempt fails.",[10,94,95,98,99,102],{},[44,96,97],{},"3. The same order."," If the suite uses ",[56,100,101],{},"pytest-randomly",", the seed is in the run header — reuse it. Otherwise, extract the executed order from the junit XML and replay it:",[49,104,106],{"className":51,"code":105,"language":53,"meta":54,"style":54},"$ pytest -p randomly -p no:cacheprovider --randomly-seed=1701 -q      # same shuffle\n$ python -c \"\nimport xml.etree.ElementTree as ET\nroot = ET.parse('artifacts\u002Fjunit.xml').getroot()\nfor case in root.iter('testcase'):\n    print(f\\\"{case.get('file')}::{case.get('name')}\\\")\n\" > order.txt\n$ pytest -q $(tr '\\n' ' ' \u003C order.txt)                                # exact order\n",[56,107,108,113,118,124,130,136,142,148],{"__ignoreMap":54},[59,109,110],{"class":61,"line":62},[59,111,112],{},"$ pytest -p randomly -p no:cacheprovider --randomly-seed=1701 -q      # same shuffle\n",[59,114,115],{"class":61,"line":68},[59,116,117],{},"$ python -c \"\n",[59,119,121],{"class":61,"line":120},3,[59,122,123],{},"import xml.etree.ElementTree as ET\n",[59,125,127],{"class":61,"line":126},4,[59,128,129],{},"root = ET.parse('artifacts\u002Fjunit.xml').getroot()\n",[59,131,133],{"class":61,"line":132},5,[59,134,135],{},"for case in root.iter('testcase'):\n",[59,137,139],{"class":61,"line":138},6,[59,140,141],{},"    print(f\\\"{case.get('file')}::{case.get('name')}\\\")\n",[59,143,145],{"class":61,"line":144},7,[59,146,147],{},"\" > order.txt\n",[59,149,151],{"class":61,"line":150},8,[59,152,153],{},"$ pytest -q $(tr '\\n' ' ' \u003C order.txt)                                # exact order\n",[10,155,156,159,160,162],{},[44,157,158],{},"4. The same parallelism."," ",[56,161,80],{}," resolves to the CPU count, which differs between a laptop and a runner. Pin it to the number CI used, and pin the distribution mode too.",[10,164,165,168],{},[44,166,167],{},"5. The same resource limits."," A two-CPU, four-gigabyte runner behaves differently from a sixteen-core workstation:",[49,170,172],{"className":51,"code":171,"language":53,"meta":54,"style":54},"$ docker run --rm -it --cpus=2 --memory=4g -v \"$PWD:\u002Fsrc\" -w \u002Fsrc \\\n    ghcr.io\u002Facme\u002Fci-python@sha256:6b1c... pytest -q -n 2\n",[56,173,174,179],{"__ignoreMap":54},[59,175,176],{"class":61,"line":62},[59,177,178],{},"$ docker run --rm -it --cpus=2 --memory=4g -v \"$PWD:\u002Fsrc\" -w \u002Fsrc \\\n",[59,180,181],{"class":61,"line":68},[59,182,183],{},"    ghcr.io\u002Facme\u002Fci-python@sha256:6b1c... pytest -q -n 2\n",[10,185,186,189],{},[44,187,188],{},"6. The same locale, timezone and hash seed."," These come from the environment snapshot:",[49,191,193],{"className":51,"code":192,"language":53,"meta":54,"style":54},"$ docker run --rm -e TZ=UTC -e LC_ALL=C.UTF-8 -e PYTHONHASHSEED=0 ...\n",[56,194,195],{"__ignoreMap":54},[59,196,197],{"class":61,"line":62},[59,198,192],{},[200,201,204,384],"figure",{"className":202},[203],"diagram",[205,206,213,214,213,218,213,222,213,240,213,248,213,257,213,265,213,271,213,276,213,281,213,291,213,296,213,300,213,304,213,308,213,312,213,316,213,320,213,323,213,327,213,331,213,335,213,339,213,343,213,346,213,349,213,353,213,357,213,361,213,365,213,368,213,372,213,376,213,380],"svg",{"viewBox":207,"role":208,"ariaLabelledBy":209,"xmlns":212},"0 0 760 532","img",[210,211],"reprosteps-t","reprosteps-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[215,216,217],"title",{"id":210},"Six dimensions to match, in order",[219,220,221],"desc",{"id":211},"A vertical checklist of the six dimensions that differ between a laptop and a CI runner: the image pinned by digest, the exact command, the test order and seed, the parallel worker count, the CPU and memory limits, and the locale, timezone and hash seed.",[223,224,225,226,213],"defs",{},"\n    ",[227,228,235],"marker",{"id":229,"viewBox":230,"refX":231,"refY":232,"markerWidth":233,"markerHeight":233,"orient":234},"reprosteps-a","0 0 10 10","9","5","7","auto-start-reverse",[236,237],"path",{"d":238,"fill":239},"M0 0 L10 5 L0 10 z","#e07a5f",[241,242],"rect",{"x":243,"y":243,"width":244,"height":245,"rx":246,"fill":247},"0","760","532","14","#fffdf8",[249,250,217],"text",{"x":251,"y":252,"textAnchor":253,"fontSize":254,"fontWeight":255,"fill":256},"380","30","middle","15","700","#3d405b",[241,258],{"x":259,"y":260,"width":261,"height":262,"rx":263,"fill":247,"stroke":239,"strokeWidth":264},"34","58","320","64","12","1.8",[249,266,270],{"x":267,"y":268,"textAnchor":253,"fontSize":269,"fontWeight":255,"fill":256},"194","86","12.5","the image, by digest",[249,272,275],{"x":267,"y":273,"textAnchor":253,"fontSize":274,"fill":256},"103","11","a tag is not a version",[61,277],{"x1":267,"y1":278,"x2":267,"y2":279,"stroke":239,"strokeWidth":264,"markerEnd":280},"128","149","url(#reprosteps-a)",[241,282],{"x":283,"y":262,"width":284,"height":285,"rx":286,"fill":247,"stroke":287,"strokeWidth":288,"strokeDashArray":289},"384","346","52","10","#81b29a","1.5",[232,290],"4",[249,292,295],{"x":293,"y":294,"textAnchor":253,"fontSize":274,"fill":256},"557","93","Digests are in the run log",[241,297],{"x":259,"y":298,"width":261,"height":262,"rx":263,"fill":299,"stroke":239,"strokeWidth":264},"156","#f4f1de",[249,301,303],{"x":267,"y":302,"textAnchor":253,"fontSize":269,"fontWeight":255,"fill":256},"184","the exact command",[249,305,307],{"x":267,"y":306,"textAnchor":253,"fontSize":274,"fill":256},"201","every flag matters",[61,309],{"x1":267,"y1":310,"x2":267,"y2":311,"stroke":239,"strokeWidth":264,"markerEnd":280},"226","247",[241,313],{"x":283,"y":314,"width":284,"height":285,"rx":286,"fill":299,"stroke":287,"strokeWidth":288,"strokeDashArray":315},"162",[232,290],[249,317,319],{"x":293,"y":318,"textAnchor":253,"fontSize":274,"fill":256},"191","Copy it from the workflow file",[241,321],{"x":259,"y":322,"width":261,"height":262,"rx":263,"fill":247,"stroke":239,"strokeWidth":264},"254",[249,324,326],{"x":267,"y":325,"textAnchor":253,"fontSize":269,"fontWeight":255,"fill":256},"282","the order and seed",[249,328,330],{"x":267,"y":329,"textAnchor":253,"fontSize":274,"fill":256},"299","the most common cause",[61,332],{"x1":267,"y1":333,"x2":267,"y2":334,"stroke":239,"strokeWidth":264,"markerEnd":280},"324","345",[241,336],{"x":283,"y":337,"width":284,"height":285,"rx":286,"fill":247,"stroke":287,"strokeWidth":288,"strokeDashArray":338},"260",[232,290],[249,340,342],{"x":293,"y":341,"textAnchor":253,"fontSize":274,"fill":256},"289","Seed is printed in the header",[241,344],{"x":259,"y":345,"width":261,"height":262,"rx":263,"fill":299,"stroke":239,"strokeWidth":264},"352",[249,347,348],{"x":267,"y":251,"textAnchor":253,"fontSize":269,"fontWeight":255,"fill":256},"workers and resources",[249,350,352],{"x":267,"y":351,"textAnchor":253,"fontSize":274,"fill":256},"397","-n auto is not a number",[61,354],{"x1":267,"y1":355,"x2":267,"y2":356,"stroke":239,"strokeWidth":264,"markerEnd":280},"422","443",[241,358],{"x":283,"y":359,"width":284,"height":285,"rx":286,"fill":299,"stroke":287,"strokeWidth":288,"strokeDashArray":360},"358",[232,290],[249,362,364],{"x":293,"y":363,"textAnchor":253,"fontSize":274,"fill":256},"387","Pin both -n and --dist",[241,366],{"x":259,"y":367,"width":261,"height":262,"rx":263,"fill":247,"stroke":239,"strokeWidth":264},"450",[249,369,371],{"x":267,"y":370,"textAnchor":253,"fontSize":269,"fontWeight":255,"fill":256},"478","locale, TZ, hash seed",[249,373,375],{"x":267,"y":374,"textAnchor":253,"fontSize":274,"fill":256},"495","formatting and iteration order",[241,377],{"x":283,"y":378,"width":284,"height":285,"rx":286,"fill":247,"stroke":287,"strokeWidth":288,"strokeDashArray":379},"456",[232,290],[249,381,383],{"x":293,"y":382,"textAnchor":253,"fontSize":274,"fill":256},"485","All three come from env.txt",[385,386,387],"figcaption",{},"Work down the list and stop when the failure appears: the step that reproduced it is the diagnosis.",[14,389,391],{"id":390},"why-this-works","Why this works",[10,393,394],{},"A test failure is deterministic given the same code and the same inputs. When CI and a laptop disagree, the inputs differ — and the inputs to a test suite include far more than the source tree. Test order determines which state a test inherits from its predecessors; worker count determines which tests share a database, a port or a temporary path; CPU count and memory affect timing-sensitive assertions and garbage-collection behaviour; and locale, timezone and hash seed change formatting, sorting and dictionary iteration order. Matching all six makes the two runs the same experiment, at which point the failure either reproduces or the remaining difference is genuinely unusual.",[200,396,398,494],{"className":397},[203],[205,399,213,404,213,407,213,410,213,413,213,415,213,421,213,426,213,430,213,434,213,438,213,443,213,446,213,449,213,452,213,455,213,458,213,461,213,464,213,468,213,471,213,474,213,477,213,481,213,484,213,487,213,491],{"viewBox":400,"role":208,"ariaLabelledBy":401,"xmlns":212},"0 0 760 270",[402,403],"bisectorder-t","bisectorder-d",[215,405,406],{"id":402},"Which dimension explains which symptom",[219,408,409],{"id":403},"A table matching four CI-only symptoms - failure only in a full run, failure only under parallelism, timing assertion failures, and formatting or ordering differences - to the dimension responsible.",[241,411],{"x":243,"y":243,"width":244,"height":412,"rx":246,"fill":247},"270",[249,414,406],{"x":251,"y":252,"textAnchor":253,"fontSize":254,"fontWeight":255,"fill":256},[241,416],{"x":417,"y":285,"width":418,"height":419,"rx":286,"fill":420,"stroke":256,"strokeWidth":288},"24","712","40","#f2cc8f",[249,422,425],{"x":423,"y":424,"fontSize":263,"fontWeight":255,"fill":256},"38","76","Criterion",[249,427,429],{"x":428,"y":424,"textAnchor":253,"fontSize":263,"fontWeight":255,"fill":256},"390","Likely dimension",[249,431,433],{"x":432,"y":424,"textAnchor":253,"fontSize":263,"fontWeight":255,"fill":256},"620","Confirm by",[241,435],{"x":417,"y":436,"width":418,"height":419,"fill":299,"stroke":256,"strokeWidth":437},"92","1",[249,439,442],{"x":423,"y":440,"fontSize":441,"fill":256},"116","11.5","only in a full run",[249,444,445],{"x":428,"y":440,"textAnchor":253,"fontSize":441,"fill":256},"test order",[249,447,448],{"x":432,"y":440,"textAnchor":253,"fontSize":441,"fill":256},"replay the recorded order",[241,450],{"x":417,"y":451,"width":418,"height":419,"fill":247,"stroke":256,"strokeWidth":437},"132",[249,453,454],{"x":423,"y":298,"fontSize":441,"fill":256},"only under -n",[249,456,457],{"x":428,"y":298,"textAnchor":253,"fontSize":441,"fill":256},"worker partitioning",[249,459,460],{"x":432,"y":298,"textAnchor":253,"fontSize":441,"fill":256},"run with -n 1",[241,462],{"x":417,"y":463,"width":418,"height":419,"fill":299,"stroke":256,"strokeWidth":437},"172",[249,465,467],{"x":423,"y":466,"fontSize":441,"fill":256},"196","timing assertions",[249,469,470],{"x":428,"y":466,"textAnchor":253,"fontSize":441,"fill":256},"CPU limits",[249,472,473],{"x":432,"y":466,"textAnchor":253,"fontSize":441,"fill":256},"run with --cpus matched",[241,475],{"x":417,"y":476,"width":418,"height":419,"fill":247,"stroke":256,"strokeWidth":437},"212",[249,478,480],{"x":423,"y":479,"fontSize":441,"fill":256},"236","formatting or sorting",[249,482,483],{"x":428,"y":479,"textAnchor":253,"fontSize":441,"fill":256},"locale, hash seed",[249,485,486],{"x":432,"y":479,"textAnchor":253,"fontSize":441,"fill":256},"export the CI values",[61,488],{"x1":489,"y1":285,"x2":489,"y2":490,"stroke":256,"strokeWidth":437},"274","252",[61,492],{"x1":493,"y1":285,"x2":493,"y2":490,"stroke":256,"strokeWidth":437},"505",[385,495,496],{},"The symptom names the dimension, which is why the checklist rarely needs to be worked through in full.",[14,498,500],{"id":499},"isolating-an-order-dependency","Isolating an order dependency",[10,502,503,504,507],{},"If step three reproduces the failure, the cause is a hidden dependency between tests, and ",[56,505,506],{},"pytest"," can find it faster than reading code.",[49,509,511],{"className":51,"code":510,"language":53,"meta":54,"style":54},"$ pip install pytest-randomly            # if not already present\n$ pytest -q --randomly-seed=1701         # reproduce\n$ pytest -q --randomly-seed=1701 -p no:randomly tests\u002Ftest_a.py tests\u002Ftest_b.py\n",[56,512,513,518,523],{"__ignoreMap":54},[59,514,515],{"class":61,"line":62},[59,516,517],{},"$ pip install pytest-randomly            # if not already present\n",[59,519,520],{"class":61,"line":68},[59,521,522],{},"$ pytest -q --randomly-seed=1701         # reproduce\n",[59,524,525],{"class":61,"line":120},[59,526,527],{},"$ pytest -q --randomly-seed=1701 -p no:randomly tests\u002Ftest_a.py tests\u002Ftest_b.py\n",[10,529,530,531,534],{},"For a large suite, bisect the ordering rather than the code. ",[56,532,533],{},"pytest --lf"," reruns the last failure alone — if it passes in isolation, the dependency is confirmed. Then halve the preceding test list repeatedly until a minimal pair remains:",[49,536,538],{"className":51,"code":537,"language":53,"meta":54,"style":54},"$ pytest -q $(head -n 400 order.txt | tr '\\n' ' ')      # still fails?\n$ pytest -q $(head -n 200 order.txt | tr '\\n' ' ')      # narrow\n",[56,539,540,545],{"__ignoreMap":54},[59,541,542],{"class":61,"line":62},[59,543,544],{},"$ pytest -q $(head -n 400 order.txt | tr '\\n' ' ')      # still fails?\n",[59,546,547],{"class":61,"line":68},[59,548,549],{},"$ pytest -q $(head -n 200 order.txt | tr '\\n' ' ')      # narrow\n",[10,551,552,553,556,557,88,561,30],{},"The pair that remains is almost always one test mutating global state and another reading it — a module-level cache, an environment variable set without ",[56,554,555],{},"monkeypatch",", a registry that was appended to. The fix belongs in the first test, and the techniques are in ",[26,558,560],{"href":559},"\u002Fadvanced-mocking-test-doubles-in-python\u002Fdependency-injection-for-testability\u002Freplacing-singletons-and-module-globals-in-tests\u002F","replacing singletons and module globals in tests",[26,562,564],{"href":563},"\u002Fadvanced-mocking-test-doubles-in-python\u002Ffaking-the-filesystem-and-environment\u002Fpatching-environment-variables-with-monkeypatch-setenv\u002F","patching environment variables with monkeypatch.setenv",[14,566,568],{"id":567},"when-it-still-will-not-reproduce","When it still will not reproduce",[10,570,571],{},"Three residual causes account for most of the remainder, and each has a distinguishing test.",[10,573,574,577,578,581],{},[44,575,576],{},"Network and DNS."," CI runners resolve differently, have different egress rules, and often no outbound access at all. A test that passes locally because it silently reached the real API will fail in CI with a timeout — run locally with networking disabled (",[56,579,580],{},"docker run --network=none",") to confirm.",[10,583,584,587],{},[44,585,586],{},"Time of day and clock skew."," A test that fails only on scheduled runs is usually reading the wall clock: a date boundary, a token expiry, a business-hours check. Run with the CI job's timestamp faked to confirm.",[10,589,590,593,594,597],{},[44,591,592],{},"Filesystem semantics."," Case sensitivity, path length limits and file ordering from ",[56,595,596],{},"os.listdir"," differ between macOS, Linux and network mounts. A test asserting on a sorted directory listing may pass on one and fail on the other; run inside the Linux image rather than natively on macOS.",[10,599,600,601,30],{},"If none of these apply and the failure remains CI-only, the next tool is a remote debugger attached to the containerised run, which is covered in ",[26,602,604],{"href":603},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fattaching-debugpy-to-a-container\u002F","attaching debugpy to a container",[14,606,608],{"id":607},"edge-cases-and-failure-modes","Edge cases and failure modes",[19,610,611,617,626,632,642,656],{},[22,612,613,616],{},[44,614,615],{},"Bind-mounting the working tree changes filesystem behaviour."," Mount performance and case sensitivity differ from a copied-in tree, so a reproduction that needs the exact filesystem should build the image rather than mounting.",[22,618,619,625],{},[44,620,621,624],{},[56,622,623],{},"--randomly-seed"," only applies when the plugin is installed."," A seed pasted into a run without the plugin is silently ignored.",[22,627,628,631],{},[44,629,630],{},"Junit XML does not record execution order under xdist reliably."," Use the per-test artefact dump to record worker and start time instead.",[22,633,634,641],{},[44,635,636,637,640],{},"Matching ",[56,638,639],{},"--cpus"," does not match CPU model."," Timing-sensitive tests can still differ; treat wall-clock assertions as a defect rather than something to reproduce exactly.",[22,643,644,647,648,651,652,655],{},[44,645,646],{},"Cached CI layers hide dependency changes."," A ",[56,649,650],{},"pip install"," in a cached layer may resolve to different versions than a fresh run; compare ",[56,653,654],{},"pip freeze"," from the snapshot.",[22,657,658,661],{},[44,659,660],{},"Some failures are genuinely load-dependent."," A runner executing three jobs concurrently has different timing than an idle laptop, and the honest fix is removing the timing dependency.",[14,663,665],{"id":664},"preventing-the-next-ci-only-failure","Preventing the next CI-only failure",[10,667,668],{},"Reproducing one failure is worth doing; removing the class is worth more. Four changes narrow the gap between a laptop and a runner permanently.",[10,670,671,647,674,677],{},[44,672,673],{},"Run the container locally by default.",[56,675,676],{},"make test"," target that runs the CI image with the CI command means developers and CI execute the same thing, and the whole category of \"works on my machine\" disappears without anybody being disciplined about it.",[10,679,680,159,683,81,686,81,689,692],{},[44,681,682],{},"Pin the non-deterministic dimensions in configuration rather than in the workflow.",[56,684,685],{},"PYTHONHASHSEED",[56,687,688],{},"TZ",[56,690,691],{},"LC_ALL"," and the worker count belong in the project's test configuration, where both environments read them, rather than in a CI file only the runner sees.",[10,694,695,698,699,701],{},[44,696,697],{},"Randomise order locally, not just in CI."," If CI shuffles and laptops do not, order-dependent failures are found by CI first every time. Installing ",[56,700,101],{}," for everyone moves that discovery to the machine where debugging is easy.",[10,703,704,707],{},[44,705,706],{},"Assert the environment in a fixture."," A session-scoped check that fails fast when the timezone, locale or worker configuration is unexpected turns a subtle behavioural difference into an explicit message:",[49,709,713],{"className":710,"code":711,"language":712,"meta":54,"style":54},"language-python shiki shiki-themes github-light github-dark","import os, time, pytest\n\n@pytest.fixture(scope=\"session\", autouse=True)\ndef _assert_expected_environment():\n    assert time.tzname[0] == \"UTC\", f\"expected UTC, got {time.tzname[0]}\"\n    assert os.environ.get(\"PYTHONHASHSEED\") == \"0\", \"set PYTHONHASHSEED=0\"\n","python",[56,714,715,720,726,731,736,741],{"__ignoreMap":54},[59,716,717],{"class":61,"line":62},[59,718,719],{},"import os, time, pytest\n",[59,721,722],{"class":61,"line":68},[59,723,725],{"emptyLinePlaceholder":724},true,"\n",[59,727,728],{"class":61,"line":120},[59,729,730],{},"@pytest.fixture(scope=\"session\", autouse=True)\n",[59,732,733],{"class":61,"line":126},[59,734,735],{},"def _assert_expected_environment():\n",[59,737,738],{"class":61,"line":132},[59,739,740],{},"    assert time.tzname[0] == \"UTC\", f\"expected UTC, got {time.tzname[0]}\"\n",[59,742,743],{"class":61,"line":138},[59,744,745],{},"    assert os.environ.get(\"PYTHONHASHSEED\") == \"0\", \"set PYTHONHASHSEED=0\"\n",[10,747,748],{},"That fixture reads as bureaucratic until the first time it catches a runner image upgrade that changed the default locale — at which point it has paid for itself several times over.",[14,750,752],{"id":751},"frequently-asked-questions","Frequently Asked Questions",[10,754,755,758],{},[44,756,757],{},"What is the single most common cause of a CI-only failure?","\nTest order. CI often runs the full suite in a different order or with a different worker count than a developer running one file, and a hidden dependency between tests only shows in some orders.",[10,760,761,764,765,767],{},[44,762,763],{},"How do I run CI's exact test order locally?","\nRead the order from the junit XML or the log, then pass the node ids in that order, or reuse the same ",[56,766,101],{}," seed which is printed in the run header.",[10,769,770,773,774,776,777,780],{},[44,771,772],{},"Do resource limits matter for correctness?","\nYes. A container with two CPUs changes ",[56,775,80],{},", timing-sensitive assertions and thread scheduling, and a low memory limit changes when the allocator returns memory. Match both when reproducing.\n",[44,778,779],{},"Should I run the whole suite locally to reproduce, or just the failing test?","\nStart with the failing test alone: if it fails in isolation, the environment is the cause and the checklist above applies. If it passes alone and fails in the full run, the cause is order or shared state, and the bisection procedure is the faster path. Running the whole suite as the first move wastes several minutes on the most likely outcome, which is that the test passes.",[200,782,784,849],{"className":783},[203],[205,785,213,790,213,793,213,796,213,798,213,800,213,803,213,806,213,809,213,814,213,817,213,819,213,823,213,826,213,828,213,830,213,834,213,837,213,840,213,842,213,846],{"viewBox":786,"role":208,"ariaLabelledBy":787,"xmlns":212},"0 0 760 320",[788,789],"reprofirst-t","reprofirst-d",[215,791,792],{"id":788},"What each isolation level tells you",[219,794,795],{"id":789},"A stack of four reproduction attempts in increasing scope: the test alone, its module, the full suite serially, and the full suite in parallel, with the conclusion each supports.",[241,797],{"x":243,"y":243,"width":244,"height":261,"rx":246,"fill":247},[249,799,792],{"x":251,"y":252,"textAnchor":253,"fontSize":254,"fontWeight":255,"fill":256},[241,801],{"x":252,"y":802,"width":255,"height":285,"rx":286,"fill":299,"stroke":287,"strokeWidth":264},"56",[241,804],{"x":252,"y":802,"width":805,"height":285,"rx":290,"fill":287},"8",[249,807,808],{"x":285,"y":268,"fontSize":269,"fontWeight":255,"fill":256},"the test alone",[249,810,813],{"x":811,"y":268,"textAnchor":812,"fontSize":274,"fill":256},"714","end","fails: environment. passes: shared state",[241,815],{"x":252,"y":816,"width":255,"height":285,"rx":286,"fill":247,"stroke":420,"strokeWidth":264},"120",[241,818],{"x":252,"y":816,"width":805,"height":285,"rx":290,"fill":420},[249,820,822],{"x":285,"y":821,"fontSize":269,"fontWeight":255,"fill":256},"150","its module",[249,824,825],{"x":811,"y":821,"textAnchor":812,"fontSize":274,"fill":256},"fails: a sibling test dirtied something",[241,827],{"x":252,"y":302,"width":255,"height":285,"rx":286,"fill":299,"stroke":239,"strokeWidth":264},[241,829],{"x":252,"y":302,"width":805,"height":285,"rx":290,"fill":239},[249,831,833],{"x":285,"y":832,"fontSize":269,"fontWeight":255,"fill":256},"214","full suite, serial",[249,835,836],{"x":811,"y":832,"textAnchor":812,"fontSize":274,"fill":256},"fails: order dependence, bisect it",[241,838],{"x":252,"y":839,"width":255,"height":285,"rx":286,"fill":247,"stroke":256,"strokeWidth":264},"248",[241,841],{"x":252,"y":839,"width":805,"height":285,"rx":290,"fill":256},[249,843,845],{"x":285,"y":844,"fontSize":269,"fontWeight":255,"fill":256},"278","full suite, parallel",[249,847,848],{"x":811,"y":844,"textAnchor":812,"fontSize":274,"fill":256},"fails only here: worker partitioning",[385,850,851],{},"Each level costs more time and eliminates a different cause, so working upward finds the answer in the fewest minutes.",[14,853,855],{"id":854},"related","Related",[19,857,858,864,870,877],{},[22,859,860,863],{},[26,861,862],{"href":28},"Debugging tests in CI and containers"," — the artefact capture that makes this checklist possible.",[22,865,866,869],{},[26,867,868],{"href":603},"Attaching debugpy to a container"," — when the artefacts are not enough.",[22,871,872,876],{},[26,873,875],{"href":874},"\u002Fadvanced-pytest-architecture-configuration\u002Foptimizing-test-discovery\u002Fpytest-xdist-vs-pytest-parallel-performance-comparison\u002F","pytest-xdist vs pytest-parallel performance comparison"," — worker partitioning and why it changes outcomes.",[22,878,879,883],{},[26,880,882],{"href":881},"\u002Fadvanced-pytest-architecture-configuration\u002Foptimizing-test-discovery\u002Fdebugging-flaky-tests-with-pytest-rerunfailures\u002F","Debugging flaky tests with pytest-rerunfailures"," — separating order dependence from genuine non-determinism.",[10,885,886,887],{},"← Back to ",[26,888,889],{"href":28},"Debugging Tests in CI and Containers",[891,892,893],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":54,"searchDepth":68,"depth":68,"links":895},[896,897,898,899,900,901,902,903,904],{"id":16,"depth":68,"text":17},{"id":36,"depth":68,"text":37},{"id":390,"depth":68,"text":391},{"id":499,"depth":68,"text":500},{"id":567,"depth":68,"text":568},{"id":607,"depth":68,"text":608},{"id":664,"depth":68,"text":665},{"id":751,"depth":68,"text":752},{"id":854,"depth":68,"text":855},"Turn a CI-only failure into a local one: match the image, test order, worker count, resource limits, locale and clock, then bisect the difference systematically.","md",{"slug":908,"type":909,"breadcrumb":910,"datePublished":911,"dateModified":911,"faq":912,"howto":919},"reproducing-ci-only-test-failures-locally","article","Reproduce Locally","2026-08-01",[913,915,917],{"q":757,"a":914},"Test order. CI often runs the full suite in a different order or with a different worker count than a developer running one file, and a hidden dependency between tests only shows in some orders.",{"q":763,"a":916},"Read the order from the junit XML or the log, then pass the node ids in that order, or reuse the same pytest-randomly seed which is printed in the run header.",{"q":772,"a":918},"Yes. A container with two CPUs changes -n auto, timing-sensitive assertions and thread scheduling, and a low memory limit changes when the allocator returns memory. Match both when reproducing.",{"name":920,"description":921,"steps":922},"How to reproduce a CI-only test failure locally","Match the environment dimensions that differ until the failure appears on your machine.",[923,926,929,932],{"name":924,"text":925},"Run the same image","Pull the CI image by digest and run the same command inside it.",{"name":927,"text":928},"Match the test order and seed","Reuse the printed random seed or replay the node ids in CI order.",{"name":930,"text":931},"Match parallelism and resources","Pin the worker count and constrain CPU and memory to the runner's limits.",{"name":933,"text":934},"Bisect the remaining differences","Change one dimension at a time until the failure appears or the list is exhausted.","\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Freproducing-ci-only-test-failures-locally",{"title":5,"description":905},"systematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Freproducing-ci-only-test-failures-locally\u002Findex","W1E0kJfFQ3HYikD14atkTEpI0hRS9S5s-YzhIj0S6Hk",1785613403107]