[{"data":1,"prerenderedAt":1666},["ShallowReactive",2],{"page-\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002F":3},{"id":4,"title":5,"body":6,"description":1630,"extension":1631,"meta":1632,"navigation":457,"path":1662,"seo":1663,"stem":1664,"__hash__":1665},"content\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Findex.md","Debugging Tests in CI and Containers",{"type":7,"value":8,"toc":1614},"minimark",[9,13,18,36,40,43,213,236,378,382,392,406,426,436,543,649,653,656,669,681,690,696,748,751,830,834,839,869,872,887,891,894,905,964,970,984,994,1003,1078,1082,1190,1194,1197,1203,1209,1215,1221,1230,1233,1236,1240,1243,1262,1317,1320,1324,1330,1336,1342,1348,1352,1355,1361,1371,1381,1387,1393,1424,1427,1502,1507,1510,1530,1549,1564,1570,1574,1603,1610],[10,11,12],"p",{},"The suite is green locally and red in CI, and the failure output is a stack trace with no context. Everything that would explain it — the state of the temporary directory, the environment, the order the tests ran in — was discarded when the runner terminated. Debugging in CI is therefore less about tools than about deciding, in advance, what evidence a failure must leave behind.",[14,15,17],"h2",{"id":16},"prerequisites","Prerequisites",[19,20,21,25],"ul",{},[22,23,24],"li",{},"A CI system that can upload artefacts, and a container image you can run locally.",[22,26,27,31,32,35],{},[28,29,30],"strong",{},"pytest 7.0+",", and ",[28,33,34],{},"debugpy 1.8+"," for the remote-attach section.",[14,37,39],{"id":38},"core-concept-capture-then-compare","Core concept: capture, then compare",[10,41,42],{},"Two activities solve most CI-only failures. The first is capturing enough state at failure time that no re-run is needed. The second is comparing the CI environment with the local one, because the failure is a difference and the difference is almost always in a short list.",[44,45,50],"pre",{"className":46,"code":47,"language":48,"meta":49,"style":49},"language-yaml shiki shiki-themes github-light github-dark","- name: Run tests\n  run: |\n    pytest -q --junitxml=artifacts\u002Fjunit.xml \\\n           --basetemp=artifacts\u002Ftmp \\\n           -p no:randomly -o cache_dir=artifacts\u002Fpytest_cache\n  env:\n    PYTHONHASHSEED: \"0\"\n- name: Environment snapshot\n  if: always()\n  run: |\n    { python -VV; pip freeze; env | sort; nproc; locale; date; } > artifacts\u002Fenv.txt\n- uses: actions\u002Fupload-artifact@v4\n  if: always()\n  with: { name: test-artifacts, path: artifacts\u002F }\n","yaml","",[51,52,53,73,85,91,97,103,112,123,135,146,155,161,174,183],"code",{"__ignoreMap":49},[54,55,58,62,66,69],"span",{"class":56,"line":57},"line",1,[54,59,61],{"class":60},"sVt8B","- ",[54,63,65],{"class":64},"s9eBZ","name",[54,67,68],{"class":60},": ",[54,70,72],{"class":71},"sZZnC","Run tests\n",[54,74,76,79,81],{"class":56,"line":75},2,[54,77,78],{"class":64},"  run",[54,80,68],{"class":60},[54,82,84],{"class":83},"szBVR","|\n",[54,86,88],{"class":56,"line":87},3,[54,89,90],{"class":71},"    pytest -q --junitxml=artifacts\u002Fjunit.xml \\\n",[54,92,94],{"class":56,"line":93},4,[54,95,96],{"class":71},"           --basetemp=artifacts\u002Ftmp \\\n",[54,98,100],{"class":56,"line":99},5,[54,101,102],{"class":71},"           -p no:randomly -o cache_dir=artifacts\u002Fpytest_cache\n",[54,104,106,109],{"class":56,"line":105},6,[54,107,108],{"class":64},"  env",[54,110,111],{"class":60},":\n",[54,113,115,118,120],{"class":56,"line":114},7,[54,116,117],{"class":64},"    PYTHONHASHSEED",[54,119,68],{"class":60},[54,121,122],{"class":71},"\"0\"\n",[54,124,126,128,130,132],{"class":56,"line":125},8,[54,127,61],{"class":60},[54,129,65],{"class":64},[54,131,68],{"class":60},[54,133,134],{"class":71},"Environment snapshot\n",[54,136,138,141,143],{"class":56,"line":137},9,[54,139,140],{"class":64},"  if",[54,142,68],{"class":60},[54,144,145],{"class":71},"always()\n",[54,147,149,151,153],{"class":56,"line":148},10,[54,150,78],{"class":64},[54,152,68],{"class":60},[54,154,84],{"class":83},[54,156,158],{"class":56,"line":157},11,[54,159,160],{"class":71},"    { python -VV; pip freeze; env | sort; nproc; locale; date; } > artifacts\u002Fenv.txt\n",[54,162,164,166,169,171],{"class":56,"line":163},12,[54,165,61],{"class":60},[54,167,168],{"class":64},"uses",[54,170,68],{"class":60},[54,172,173],{"class":71},"actions\u002Fupload-artifact@v4\n",[54,175,177,179,181],{"class":56,"line":176},13,[54,178,140],{"class":64},[54,180,68],{"class":60},[54,182,145],{"class":71},[54,184,186,189,192,194,196,199,202,205,207,210],{"class":56,"line":185},14,[54,187,188],{"class":64},"  with",[54,190,191],{"class":60},": { ",[54,193,65],{"class":64},[54,195,68],{"class":60},[54,197,198],{"class":71},"test-artifacts",[54,200,201],{"class":60},", ",[54,203,204],{"class":64},"path",[54,206,68],{"class":60},[54,208,209],{"class":71},"artifacts\u002F",[54,211,212],{"class":60}," }\n",[10,214,215,218,219,201,222,201,225,201,228,231,232,235],{},[51,216,217],{},"if: always()"," is the line that matters most: without it, the upload step is skipped precisely when the tests failed. The environment snapshot is the second: ",[51,220,221],{},"python -VV",[51,223,224],{},"pip freeze",[51,226,227],{},"env",[51,229,230],{},"nproc"," and ",[51,233,234],{},"locale"," answer most \"why is CI different\" questions in one file.",[237,238,241,374],"figure",{"className":239},[240],"diagram",[242,243,250,251,250,255,250,259,250,267,250,276,250,285,250,291,250,295,250,299,250,304,250,309,250,312,250,315,250,318,250,322,250,325,250,328,250,331,250,335,250,338,250,341,250,344,250,348,250,351,250,354,250,357,250,361,250,364,250,367,250,371],"svg",{"viewBox":244,"role":245,"ariaLabelledBy":246,"xmlns":249},"0 0 760 310","img",[247,248],"cidiffs-t","cidiffs-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[252,253,254],"title",{"id":247},"What usually differs between local and CI",[256,257,258],"desc",{"id":248},"A table of five environment differences that cause CI-only failures - test order, parallel worker count, CPU and memory limits, locale and timezone, and filesystem case sensitivity - with how to detect and control each.",[260,261],"rect",{"x":262,"y":262,"width":263,"height":264,"rx":265,"fill":266},"0","760","310","14","#fffdf8",[268,269,254],"text",{"x":270,"y":271,"textAnchor":272,"fontSize":273,"fontWeight":274,"fill":275},"380","30","middle","15","700","#3d405b",[260,277],{"x":278,"y":279,"width":280,"height":281,"rx":282,"fill":283,"stroke":275,"strokeWidth":284},"24","52","712","40","10","#f2cc8f","1.5",[268,286,290],{"x":287,"y":288,"fontSize":289,"fontWeight":274,"fill":275},"38","76","12","Criterion",[268,292,294],{"x":293,"y":288,"textAnchor":272,"fontSize":289,"fontWeight":274,"fill":275},"390","Detect with",[268,296,298],{"x":297,"y":288,"textAnchor":272,"fontSize":289,"fontWeight":274,"fill":275},"620","Control with",[260,300],{"x":278,"y":301,"width":280,"height":281,"fill":302,"stroke":275,"strokeWidth":303},"92","#f4f1de","1",[268,305,308],{"x":287,"y":306,"fontSize":307,"fill":275},"116","11.5","test order",[268,310,311],{"x":293,"y":306,"textAnchor":272,"fontSize":307,"fill":275},"-p randomly, seed in log",[268,313,314],{"x":297,"y":306,"textAnchor":272,"fontSize":307,"fill":275},"fixed seed or no:randomly",[260,316],{"x":278,"y":317,"width":280,"height":281,"fill":266,"stroke":275,"strokeWidth":303},"132",[268,319,321],{"x":287,"y":320,"fontSize":307,"fill":275},"156","worker count",[268,323,324],{"x":293,"y":320,"textAnchor":272,"fontSize":307,"fill":275},"-n auto resolves differently",[268,326,327],{"x":297,"y":320,"textAnchor":272,"fontSize":307,"fill":275},"pin -n to a number",[260,329],{"x":278,"y":330,"width":280,"height":281,"fill":302,"stroke":275,"strokeWidth":303},"172",[268,332,334],{"x":287,"y":333,"fontSize":307,"fill":275},"196","CPU and memory",[268,336,337],{"x":293,"y":333,"textAnchor":272,"fontSize":307,"fill":275},"nproc, cgroup limits",[268,339,340],{"x":297,"y":333,"textAnchor":272,"fontSize":307,"fill":275},"match container resources",[260,342],{"x":278,"y":343,"width":280,"height":281,"fill":266,"stroke":275,"strokeWidth":303},"212",[268,345,347],{"x":287,"y":346,"fontSize":307,"fill":275},"236","locale and timezone",[268,349,350],{"x":293,"y":346,"textAnchor":272,"fontSize":307,"fill":275},"locale, date",[268,352,353],{"x":297,"y":346,"textAnchor":272,"fontSize":307,"fill":275},"pin LC_ALL and TZ",[260,355],{"x":278,"y":356,"width":280,"height":281,"fill":302,"stroke":275,"strokeWidth":303},"252",[268,358,360],{"x":287,"y":359,"fontSize":307,"fill":275},"276","filesystem case",[268,362,363],{"x":293,"y":359,"textAnchor":272,"fontSize":307,"fill":275},"macOS vs Linux",[268,365,366],{"x":297,"y":359,"textAnchor":272,"fontSize":307,"fill":275},"test on the CI platform",[56,368],{"x1":369,"y1":279,"x2":369,"y2":370,"stroke":275,"strokeWidth":303},"274","292",[56,372],{"x1":373,"y1":279,"x2":373,"y2":370,"stroke":275,"strokeWidth":303},"505",[375,376,377],"figcaption",{},"Each row is a one-line control; setting all five removes most of the gap between a laptop and a runner.",[14,379,381],{"id":380},"step-by-step-implementation","Step-by-step implementation",[10,383,384,387,388,391],{},[28,385,386],{},"1. Make the run reproducible."," Pin the test order seed, the hash seed and the worker count. ",[51,389,390],{},"-n auto"," resolving to 2 locally and 16 in CI is by itself enough to change which tests share a database.",[10,393,394,397,398,401,402,405],{},[28,395,396],{},"2. Keep the temporary directories."," ",[51,399,400],{},"--basetemp=artifacts\u002Ftmp"," puts every ",[51,403,404],{},"tmp_path"," under a collectable path, so the files a failing test wrote are in the artefact bundle. That single change answers most assertion failures about file contents without any further investigation.",[10,407,408,411,412,415,416,419,420,425],{},[28,409,410],{},"3. Record which test ran where."," Under ",[51,413,414],{},"pytest-xdist",", add the worker id to the junit output or log it from a hook, so a failure concentrated on one worker is visible as such rather than as random flakiness. The hook is the same ",[51,417,418],{},"pytest_runtest_makereport"," wrapper described in ",[421,422,424],"a",{"href":423},"\u002Fadvanced-pytest-architecture-configuration\u002Foptimizing-test-discovery\u002Fdebugging-flaky-tests-with-pytest-rerunfailures\u002F","debugging flaky tests with pytest-rerunfailures",".",[10,427,428,431,432,435],{},[28,429,430],{},"4. Dump state at failure."," A ",[51,433,434],{},"conftest.py"," hook that writes the failing test's locals and the environment to a file turns a stack trace into a diagnosis:",[44,437,441],{"className":438,"code":439,"language":440,"meta":49,"style":49},"language-python shiki shiki-themes github-light github-dark","# conftest.py\nimport json, os, pathlib, pytest\n\nARTIFACTS = pathlib.Path(os.environ.get(\"ARTIFACTS\", \"artifacts\"))\n\n@pytest.hookimpl(hookwrapper=True)\ndef pytest_runtest_makereport(item, call):\n    outcome = yield\n    report = outcome.get_result()\n    if report.when == \"call\" and report.failed:\n        ARTIFACTS.mkdir(parents=True, exist_ok=True)\n        target = ARTIFACTS \u002F f\"{item.name}.json\"\n        target.write_text(json.dumps({\n            \"nodeid\": item.nodeid,\n            \"worker\": os.environ.get(\"PYTEST_XDIST_WORKER\", \"master\"),\n            \"cwd\": os.getcwd(),\n            \"env\": {k: v for k, v in os.environ.items() if k.startswith(\"APP_\")},\n            \"longrepr\": str(report.longrepr)[:4000],\n        }, indent=2))\n","python",[51,442,443,448,453,459,464,468,473,478,483,488,493,498,503,508,513,519,525,531,537],{"__ignoreMap":49},[54,444,445],{"class":56,"line":57},[54,446,447],{},"# conftest.py\n",[54,449,450],{"class":56,"line":75},[54,451,452],{},"import json, os, pathlib, pytest\n",[54,454,455],{"class":56,"line":87},[54,456,458],{"emptyLinePlaceholder":457},true,"\n",[54,460,461],{"class":56,"line":93},[54,462,463],{},"ARTIFACTS = pathlib.Path(os.environ.get(\"ARTIFACTS\", \"artifacts\"))\n",[54,465,466],{"class":56,"line":99},[54,467,458],{"emptyLinePlaceholder":457},[54,469,470],{"class":56,"line":105},[54,471,472],{},"@pytest.hookimpl(hookwrapper=True)\n",[54,474,475],{"class":56,"line":114},[54,476,477],{},"def pytest_runtest_makereport(item, call):\n",[54,479,480],{"class":56,"line":125},[54,481,482],{},"    outcome = yield\n",[54,484,485],{"class":56,"line":137},[54,486,487],{},"    report = outcome.get_result()\n",[54,489,490],{"class":56,"line":148},[54,491,492],{},"    if report.when == \"call\" and report.failed:\n",[54,494,495],{"class":56,"line":157},[54,496,497],{},"        ARTIFACTS.mkdir(parents=True, exist_ok=True)\n",[54,499,500],{"class":56,"line":163},[54,501,502],{},"        target = ARTIFACTS \u002F f\"{item.name}.json\"\n",[54,504,505],{"class":56,"line":176},[54,506,507],{},"        target.write_text(json.dumps({\n",[54,509,510],{"class":56,"line":185},[54,511,512],{},"            \"nodeid\": item.nodeid,\n",[54,514,516],{"class":56,"line":515},15,[54,517,518],{},"            \"worker\": os.environ.get(\"PYTEST_XDIST_WORKER\", \"master\"),\n",[54,520,522],{"class":56,"line":521},16,[54,523,524],{},"            \"cwd\": os.getcwd(),\n",[54,526,528],{"class":56,"line":527},17,[54,529,530],{},"            \"env\": {k: v for k, v in os.environ.items() if k.startswith(\"APP_\")},\n",[54,532,534],{"class":56,"line":533},18,[54,535,536],{},"            \"longrepr\": str(report.longrepr)[:4000],\n",[54,538,540],{"class":56,"line":539},19,[54,541,542],{},"        }, indent=2))\n",[237,544,546,646],{"className":545},[240],[242,547,250,552,250,555,250,558,250,575,250,577,250,579,250,585,250,591,250,596,250,602,250,605,250,609,250,612,250,616,250,619,250,623,250,627,250,631,250,634,250,638,250,641],{"viewBox":548,"role":245,"ariaLabelledBy":549,"xmlns":249},"0 0 760 172",[550,551],"ciloop-t","ciloop-d",[252,553,554],{"id":550},"The CI debugging loop",[256,556,557],{"id":551},"A left-to-right loop: the run fails and uploads artefacts, the environment snapshot is compared with the local one, the container is run locally with matching settings, and the now-reproducible failure is debugged normally.",[559,560,561,562,250],"defs",{},"\n    ",[563,564,571],"marker",{"id":565,"viewBox":566,"refX":567,"refY":568,"markerWidth":569,"markerHeight":569,"orient":570},"ciloop-a","0 0 10 10","9","5","7","auto-start-reverse",[204,572],{"d":573,"fill":574},"M0 0 L10 5 L0 10 z","#81b29a",[260,576],{"x":262,"y":262,"width":263,"height":330,"rx":265,"fill":266},[268,578,554],{"x":270,"y":271,"textAnchor":272,"fontSize":273,"fontWeight":274,"fill":275},[260,580],{"x":581,"y":582,"width":583,"height":288,"rx":289,"fill":302,"stroke":574,"strokeWidth":584},"26","62","142","1.8",[268,586,590],{"x":587,"y":588,"textAnchor":272,"fontSize":589,"fontWeight":274,"fill":275},"97","96","12.5","fail + upload",[268,592,595],{"x":587,"y":593,"textAnchor":272,"fontSize":594,"fill":275},"113","11","artefacts, always",[56,597],{"x1":598,"y1":599,"x2":600,"y2":599,"stroke":574,"strokeWidth":584,"markerEnd":601},"176","100","208","url(#ciloop-a)",[260,603],{"x":604,"y":582,"width":583,"height":288,"rx":289,"fill":266,"stroke":574,"strokeWidth":584},"214",[268,606,608],{"x":607,"y":588,"textAnchor":272,"fontSize":589,"fontWeight":274,"fill":275},"286","compare env",[268,610,611],{"x":607,"y":593,"textAnchor":272,"fontSize":594,"fill":275},"find the difference",[56,613],{"x1":614,"y1":599,"x2":615,"y2":599,"stroke":574,"strokeWidth":584,"markerEnd":601},"364","396",[260,617],{"x":618,"y":582,"width":583,"height":288,"rx":289,"fill":302,"stroke":574,"strokeWidth":584},"403",[268,620,622],{"x":621,"y":588,"textAnchor":272,"fontSize":589,"fontWeight":274,"fill":275},"474","run the image",[268,624,626],{"x":621,"y":593,"textAnchor":272,"fontSize":625,"fill":275},"9.5","same command and seeds",[56,628],{"x1":629,"y1":599,"x2":630,"y2":599,"stroke":574,"strokeWidth":584,"markerEnd":601},"552","584",[260,632],{"x":633,"y":582,"width":583,"height":288,"rx":289,"fill":266,"stroke":574,"strokeWidth":584},"592",[268,635,637],{"x":636,"y":588,"textAnchor":272,"fontSize":589,"fontWeight":274,"fill":275},"663","debug locally",[268,639,640],{"x":636,"y":593,"textAnchor":272,"fontSize":594,"fill":275},"ordinary tools",[268,642,645],{"x":270,"y":643,"textAnchor":272,"fontSize":307,"fontStyle":644,"fill":275},"164","italic","If it still will not reproduce, the difference is in the list you have not checked yet.",[375,647,648],{},"The goal of every step is the third one: a CI-only failure becomes tractable the moment it reproduces on a machine you control.",[14,650,652],{"id":651},"reading-a-ci-log-like-a-diagnostic-instrument","Reading a CI log like a diagnostic instrument",[10,654,655],{},"Most CI logs are read as a wall of text ending in a traceback. Configured deliberately, the same log answers four questions before the traceback is reached, and each answer costs one flag.",[10,657,658,397,661,664,665,668],{},[28,659,660],{},"What ran, and in what order?",[51,662,663],{},"-v"," prints every node id as it executes, and ",[51,666,667],{},"-p randomly"," prints the seed in the header. Together they make the run reproducible.",[10,670,671,397,674,677,678,680],{},[28,672,673],{},"Why was a test skipped or expected to fail?",[51,675,676],{},"-ra"," prints a short summary of every non-passing outcome with its reason. A suite reporting \"142 passed, 38 skipped\" tells you nothing; the same run with ",[51,679,676],{}," tells you that 38 tests skipped because an environment variable was unset, which is usually the actual bug.",[10,682,683,397,686,689],{},[28,684,685],{},"How long did each phase take?",[51,687,688],{},"--durations=15"," reports the slowest tests including setup and teardown, which is how a timeout in CI is attributed to a fixture rather than to the test that happened to be running.",[10,691,692,695],{},[28,693,694],{},"What was the environment?"," The snapshot step from earlier: version, packages, environment, CPU count, locale.",[44,697,701],{"className":698,"code":699,"language":700,"meta":49,"style":49},"language-console shiki shiki-themes github-light github-dark","$ pytest -q -ra -v --durations=15 --junitxml=artifacts\u002Fjunit.xml\n...\n=========================== short test summary info ============================\nSKIPPED [12] tests\u002Fintegration\u002Fconftest.py:18: RUN_INTEGRATION not set\nXFAIL tests\u002Ftest_legacy.py::test_v1_format - known broken, see PROJ-4417\nFAILED tests\u002Ftest_orders.py::test_total - assert 250 == 300\n=============================== slowest durations ==============================\n41.02s setup    tests\u002Fintegration\u002Ftest_checkout.py::test_flow\n0.31s  call     tests\u002Ftest_orders.py::test_total\n","console",[51,702,703,708,713,718,723,728,733,738,743],{"__ignoreMap":49},[54,704,705],{"class":56,"line":57},[54,706,707],{},"$ pytest -q -ra -v --durations=15 --junitxml=artifacts\u002Fjunit.xml\n",[54,709,710],{"class":56,"line":75},[54,711,712],{},"...\n",[54,714,715],{"class":56,"line":87},[54,716,717],{},"=========================== short test summary info ============================\n",[54,719,720],{"class":56,"line":93},[54,721,722],{},"SKIPPED [12] tests\u002Fintegration\u002Fconftest.py:18: RUN_INTEGRATION not set\n",[54,724,725],{"class":56,"line":99},[54,726,727],{},"XFAIL tests\u002Ftest_legacy.py::test_v1_format - known broken, see PROJ-4417\n",[54,729,730],{"class":56,"line":105},[54,731,732],{},"FAILED tests\u002Ftest_orders.py::test_total - assert 250 == 300\n",[54,734,735],{"class":56,"line":114},[54,736,737],{},"=============================== slowest durations ==============================\n",[54,739,740],{"class":56,"line":125},[54,741,742],{},"41.02s setup    tests\u002Fintegration\u002Ftest_checkout.py::test_flow\n",[54,744,745],{"class":56,"line":137},[54,746,747],{},"0.31s  call     tests\u002Ftest_orders.py::test_total\n",[10,749,750],{},"Those two blocks are worth more than the traceback in most investigations: the skip reason explains why coverage is lower than expected, and the durations block explains a job timeout that the traceback cannot.",[237,752,754,827],{"className":753},[240],[242,755,250,760,250,763,250,766,250,769,250,771,250,773,250,775,250,778,250,781,250,783,250,785,250,788,250,791,250,793,250,795,250,798,250,801,250,803,250,806,250,809,250,812,250,814,250,817,250,820,250,823,250,825],{"viewBox":756,"role":245,"ariaLabelledBy":757,"xmlns":249},"0 0 760 270",[758,759],"ciflags-t","ciflags-d",[252,761,762],{"id":758},"Four flags that make a CI log diagnostic",[256,764,765],{"id":759},"A table of four pytest flags used in CI - ra, v, durations and junitxml - describing what each adds to the log and the question it answers.",[260,767],{"x":262,"y":262,"width":263,"height":768,"rx":265,"fill":266},"270",[268,770,762],{"x":270,"y":271,"textAnchor":272,"fontSize":273,"fontWeight":274,"fill":275},[260,772],{"x":278,"y":279,"width":280,"height":281,"rx":282,"fill":283,"stroke":275,"strokeWidth":284},[268,774,290],{"x":287,"y":288,"fontSize":289,"fontWeight":274,"fill":275},[268,776,777],{"x":293,"y":288,"textAnchor":272,"fontSize":289,"fontWeight":274,"fill":275},"Adds",[268,779,780],{"x":297,"y":288,"textAnchor":272,"fontSize":289,"fontWeight":274,"fill":275},"Answers",[260,782],{"x":278,"y":301,"width":280,"height":281,"fill":302,"stroke":275,"strokeWidth":303},[268,784,676],{"x":287,"y":306,"fontSize":307,"fill":275},[268,786,787],{"x":293,"y":306,"textAnchor":272,"fontSize":307,"fill":275},"a reason per non-pass",[268,789,790],{"x":297,"y":306,"textAnchor":272,"fontSize":307,"fill":275},"why was it skipped?",[260,792],{"x":278,"y":317,"width":280,"height":281,"fill":266,"stroke":275,"strokeWidth":303},[268,794,663],{"x":287,"y":320,"fontSize":307,"fill":275},[268,796,797],{"x":293,"y":320,"textAnchor":272,"fontSize":307,"fill":275},"node ids in order",[268,799,800],{"x":297,"y":320,"textAnchor":272,"fontSize":307,"fill":275},"what ran, in what order?",[260,802],{"x":278,"y":330,"width":280,"height":281,"fill":302,"stroke":275,"strokeWidth":303},[268,804,805],{"x":287,"y":333,"fontSize":307,"fill":275},"--durations=N",[268,807,808],{"x":293,"y":333,"textAnchor":272,"fontSize":307,"fill":275},"slowest setup and call",[268,810,811],{"x":297,"y":333,"textAnchor":272,"fontSize":307,"fill":275},"what caused the timeout?",[260,813],{"x":278,"y":343,"width":280,"height":281,"fill":266,"stroke":275,"strokeWidth":303},[268,815,816],{"x":287,"y":346,"fontSize":307,"fill":275},"--junitxml",[268,818,819],{"x":293,"y":346,"textAnchor":272,"fontSize":307,"fill":275},"machine-readable results",[268,821,822],{"x":297,"y":346,"textAnchor":272,"fontSize":307,"fill":275},"what changed since last run?",[56,824],{"x1":369,"y1":279,"x2":369,"y2":356,"stroke":275,"strokeWidth":303},[56,826],{"x1":373,"y1":279,"x2":373,"y2":356,"stroke":275,"strokeWidth":303},[375,828,829],{},"None of these change what the tests do; all four change how much a failing log explains without a re-run.",[14,831,833],{"id":832},"verification","Verification",[10,835,836,837,425],{},"Confirm the capture works before you need it. Add a deliberately failing test on a branch, run the pipeline, and check that the artefact bundle contains the junit file, the temporary directories, the environment snapshot and the per-test dump. A capture pipeline that has never been exercised usually has one missing ",[51,838,217],{},[44,840,842],{"className":698,"code":841,"language":700,"meta":49,"style":49},"$ unzip -l test-artifacts.zip | head\n  artifacts\u002Fenv.txt\n  artifacts\u002Fjunit.xml\n  artifacts\u002Ftmp\u002Ftest_writes_report0\u002Fout\u002Freport.csv\n  artifacts\u002Ftest_writes_report.json\n",[51,843,844,849,854,859,864],{"__ignoreMap":49},[54,845,846],{"class":56,"line":57},[54,847,848],{},"$ unzip -l test-artifacts.zip | head\n",[54,850,851],{"class":56,"line":75},[54,852,853],{},"  artifacts\u002Fenv.txt\n",[54,855,856],{"class":56,"line":87},[54,857,858],{},"  artifacts\u002Fjunit.xml\n",[54,860,861],{"class":56,"line":93},[54,862,863],{},"  artifacts\u002Ftmp\u002Ftest_writes_report0\u002Fout\u002Freport.csv\n",[54,865,866],{"class":56,"line":99},[54,867,868],{},"  artifacts\u002Ftest_writes_report.json\n",[10,870,871],{},"Then verify the reproduction path: run the same image locally with the same entry point.",[44,873,875],{"className":698,"code":874,"language":700,"meta":49,"style":49},"$ docker run --rm -it -v \"$PWD:\u002Fsrc\" -w \u002Fsrc ghcr.io\u002Facme\u002Fci-python:3.12 \\\n    pytest -q -n 4 -p no:randomly\n",[51,876,877,882],{"__ignoreMap":49},[54,878,879],{"class":56,"line":57},[54,880,881],{},"$ docker run --rm -it -v \"$PWD:\u002Fsrc\" -w \u002Fsrc ghcr.io\u002Facme\u002Fci-python:3.12 \\\n",[54,883,884],{"class":56,"line":75},[54,885,886],{},"    pytest -q -n 4 -p no:randomly\n",[14,888,890],{"id":889},"container-constraints-that-change-test-behaviour","Container constraints that change test behaviour",[10,892,893],{},"A container is not a small virtual machine, and four of its differences from a developer laptop change how tests behave rather than merely how fast they run.",[10,895,896,397,899,901,902,904],{},[28,897,898],{},"CPU quota is not CPU count.",[51,900,230],{}," inside a container usually reports the host's core count, while the cgroup quota limits actual usage. ",[51,903,390],{}," therefore starts sixteen workers on a two-CPU quota, and every one of them contends. Pin the worker count explicitly, or read the quota:",[44,906,908],{"className":438,"code":907,"language":440,"meta":49,"style":49},"# conftest.py — worker count that respects the cgroup quota\nimport os, pathlib\n\ndef cpu_quota() -> int:\n    try:\n        quota, period = pathlib.Path(\"\u002Fsys\u002Ffs\u002Fcgroup\u002Fcpu.max\").read_text().split()\n        if quota != \"max\":\n            return max(1, int(int(quota) \u002F int(period)))\n    except (OSError, ValueError):\n        pass\n    return os.cpu_count() or 1\n",[51,909,910,915,920,924,929,934,939,944,949,954,959],{"__ignoreMap":49},[54,911,912],{"class":56,"line":57},[54,913,914],{},"# conftest.py — worker count that respects the cgroup quota\n",[54,916,917],{"class":56,"line":75},[54,918,919],{},"import os, pathlib\n",[54,921,922],{"class":56,"line":87},[54,923,458],{"emptyLinePlaceholder":457},[54,925,926],{"class":56,"line":93},[54,927,928],{},"def cpu_quota() -> int:\n",[54,930,931],{"class":56,"line":99},[54,932,933],{},"    try:\n",[54,935,936],{"class":56,"line":105},[54,937,938],{},"        quota, period = pathlib.Path(\"\u002Fsys\u002Ffs\u002Fcgroup\u002Fcpu.max\").read_text().split()\n",[54,940,941],{"class":56,"line":114},[54,942,943],{},"        if quota != \"max\":\n",[54,945,946],{"class":56,"line":125},[54,947,948],{},"            return max(1, int(int(quota) \u002F int(period)))\n",[54,950,951],{"class":56,"line":137},[54,952,953],{},"    except (OSError, ValueError):\n",[54,955,956],{"class":56,"line":148},[54,957,958],{},"        pass\n",[54,960,961],{"class":56,"line":157},[54,962,963],{},"    return os.cpu_count() or 1\n",[10,965,966,969],{},[28,967,968],{},"Memory limits kill rather than slow."," An out-of-memory kill inside a container terminates the process with no traceback, and the CI log shows a truncated run and exit code 137. A test suite that \"hangs and dies\" in CI but runs locally is usually hitting the limit — check the exit code before looking anywhere else.",[10,971,972,975,976,979,980,983],{},[28,973,974],{},"PID 1 does not reap children."," A container whose entry point is pytest makes it PID 1, and zombie processes from subprocess tests accumulate because PID 1 has no default ",[51,977,978],{},"SIGCHLD"," handling. Run with ",[51,981,982],{},"--init"," or an init shim when tests spawn processes.",[10,985,986,989,990,993],{},[28,987,988],{},"The filesystem may be read-only or overlay-backed."," A read-only root filesystem breaks anything writing outside the mounted volumes, including ",[51,991,992],{},".pytest_cache"," and coverage data files; point both at a writable path explicitly.",[44,995,997],{"className":698,"code":996,"language":700,"meta":49,"style":49},"$ pytest -q -o cache_dir=\u002Ftmp\u002Fpytest_cache --basetemp=\u002Ftmp\u002Fpytest_tmp\n",[51,998,999],{"__ignoreMap":49},[54,1000,1001],{"class":56,"line":57},[54,1002,996],{},[237,1004,1006,1075],{"className":1005},[240],[242,1007,250,1012,250,1015,250,1018,250,1021,250,1023,250,1027,250,1031,250,1035,250,1040,250,1043,250,1045,250,1049,250,1052,250,1055,250,1057,250,1060,250,1063,250,1066,250,1068,250,1072],{"viewBox":1008,"role":245,"ariaLabelledBy":1009,"xmlns":249},"0 0 760 320",[1010,1011],"containerdiffs-t","containerdiffs-d",[252,1013,1014],{"id":1010},"Four container constraints and their symptoms",[256,1016,1017],{"id":1011},"A stack of four container constraints - CPU quota versus core count, memory limits, PID 1 signal handling, and a read-only filesystem - with the symptom each produces in a test run.",[260,1019],{"x":262,"y":262,"width":263,"height":1020,"rx":265,"fill":266},"320",[268,1022,1014],{"x":270,"y":271,"textAnchor":272,"fontSize":273,"fontWeight":274,"fill":275},[260,1024],{"x":271,"y":1025,"width":274,"height":279,"rx":282,"fill":302,"stroke":1026,"strokeWidth":584},"56","#e07a5f",[260,1028],{"x":271,"y":1025,"width":1029,"height":279,"rx":1030,"fill":1026},"8","4",[268,1032,1034],{"x":279,"y":1033,"fontSize":589,"fontWeight":274,"fill":275},"86","CPU quota vs nproc",[268,1036,1039],{"x":1037,"y":1033,"textAnchor":1038,"fontSize":594,"fill":275},"714","end","-n auto over-subscribes and everything contends",[260,1041],{"x":271,"y":1042,"width":274,"height":279,"rx":282,"fill":266,"stroke":283,"strokeWidth":584},"120",[260,1044],{"x":271,"y":1042,"width":1029,"height":279,"rx":1030,"fill":283},[268,1046,1048],{"x":279,"y":1047,"fontSize":589,"fontWeight":274,"fill":275},"150","memory limit",[268,1050,1051],{"x":1037,"y":1047,"textAnchor":1038,"fontSize":594,"fill":275},"exit code 137, truncated log, no traceback",[260,1053],{"x":271,"y":1054,"width":274,"height":279,"rx":282,"fill":302,"stroke":574,"strokeWidth":584},"184",[260,1056],{"x":271,"y":1054,"width":1029,"height":279,"rx":1030,"fill":574},[268,1058,1059],{"x":279,"y":604,"fontSize":589,"fontWeight":274,"fill":275},"PID 1 is pytest",[268,1061,1062],{"x":1037,"y":604,"textAnchor":1038,"fontSize":594,"fill":275},"zombie processes accumulate from subprocess tests",[260,1064],{"x":271,"y":1065,"width":274,"height":279,"rx":282,"fill":266,"stroke":275,"strokeWidth":584},"248",[260,1067],{"x":271,"y":1065,"width":1029,"height":279,"rx":1030,"fill":275},[268,1069,1071],{"x":279,"y":1070,"fontSize":589,"fontWeight":274,"fill":275},"278","read-only root",[268,1073,1074],{"x":1037,"y":1070,"textAnchor":1038,"fontSize":594,"fill":275},"cache and coverage writes fail outside volumes",[375,1076,1077],{},"None of these produce a Python-level error message, which is why they are diagnosed from the exit code and the environment rather than from the traceback.",[14,1079,1081],{"id":1080},"troubleshooting","Troubleshooting",[1083,1084,1085,1101],"table",{},[1086,1087,1088],"thead",{},[1089,1090,1091,1095,1098],"tr",{},[1092,1093,1094],"th",{},"Symptom",[1092,1096,1097],{},"Root cause",[1092,1099,1100],{},"Fix",[1102,1103,1104,1120,1136,1147,1161,1179],"tbody",{},[1089,1105,1106,1110,1113],{},[1107,1108,1109],"td",{},"Passes locally, fails in CI",[1107,1111,1112],{},"different test order or worker count",[1107,1114,1115,1116,1119],{},"pin the seed and ",[51,1117,1118],{},"-n",", then re-run",[1089,1121,1122,1127,1130],{},[1107,1123,1124,1125],{},"Fails only under ",[51,1126,1118],{},[1107,1128,1129],{},"shared port, database or temp path",[1107,1131,1132,1133],{},"partition by ",[51,1134,1135],{},"PYTEST_XDIST_WORKER",[1089,1137,1138,1141,1144],{},[1107,1139,1140],{},"Timeouts only in CI",[1107,1142,1143],{},"slower runner, smaller CPU quota",[1107,1145,1146],{},"raise timeouts or match container resources",[1089,1148,1149,1152,1155],{},[1107,1150,1151],{},"Assertion about a file, no file to inspect",[1107,1153,1154],{},"temporary directories not collected",[1107,1156,1157,1160],{},[51,1158,1159],{},"--basetemp"," into the artefact path",[1089,1162,1163,1166,1169],{},[1107,1164,1165],{},"Date or number formatting differs",[1107,1167,1168],{},"locale or timezone unset in the image",[1107,1170,1171,1172,231,1175,1178],{},"pin ",[51,1173,1174],{},"LC_ALL",[51,1176,1177],{},"TZ"," in the job",[1089,1180,1181,1184,1187],{},[1107,1182,1183],{},"Works in the image locally, not in CI",[1107,1185,1186],{},"cached layer or different image digest",[1107,1188,1189],{},"pin the image by digest, not by tag",[14,1191,1193],{"id":1192},"a-capture-policy-worth-adopting","A capture policy worth adopting",[10,1195,1196],{},"The difference between a pipeline that can be debugged and one that cannot is a short, deliberate list of what every job keeps. Adopting it once removes most re-runs.",[10,1198,1199,1202],{},[28,1200,1201],{},"Always, on every run:"," the junit XML and the full log. Both are small, both are machine-readable, and having them for passing runs is what makes a comparison possible when a later run fails.",[10,1204,1205,1208],{},[28,1206,1207],{},"On failure:"," the pytest temporary directories, the environment snapshot, per-test failure dumps, and any application logs the tests produced. This is the bundle that answers \"what state was the system in\", and it costs nothing on green runs because it is gated on failure.",[10,1210,1211,1214],{},[28,1212,1213],{},"On a schedule:"," a full artefact bundle from a known-good run, kept as a baseline. Comparing a failing environment snapshot against a passing one from the same week is frequently the fastest route to a dependency change nobody noticed.",[10,1216,1217,1220],{},[28,1218,1219],{},"Never:"," secrets. An environment dump includes every variable, and CI environments are full of tokens. Filter the snapshot before writing it:",[44,1222,1224],{"className":698,"code":1223,"language":700,"meta":49,"style":49},"$ env | sort | grep -vE '(TOKEN|SECRET|PASSWORD|KEY|CREDENTIAL)' > artifacts\u002Fenv.txt\n",[51,1225,1226],{"__ignoreMap":49},[54,1227,1228],{"class":56,"line":57},[54,1229,1223],{},[10,1231,1232],{},"The policy is worth writing into the pipeline template rather than into each job, so new services inherit it. And it is worth testing: a deliberately failing branch, run once per quarter, confirms the artefacts still upload and still contain what the policy says they do.",[10,1234,1235],{},"The underlying principle is that a CI failure is a measurement you get to make once. Everything the runner discards has to be recreated by re-running, and re-running a flaky failure is exactly the thing that does not work.",[14,1237,1239],{"id":1238},"making-ci-failures-attributable","Making CI failures attributable",[10,1241,1242],{},"The last piece is organisational rather than technical: a failing pipeline needs an owner within minutes, or the artefacts are never read.",[10,1244,1245,1246,1249,1250,1253,1254,1257,1258,1261],{},"Three signals make attribution automatic. ",[28,1247,1248],{},"Name the failing test in the job summary",", not just in the log — most CI systems render a markdown summary, and one line naming the test and its file routes the failure to the person who touched it. ",[28,1251,1252],{},"Group failures by directory",", because a monorepo's pipeline failing on ",[51,1255,1256],{},"services\u002Fbilling"," is a billing problem and everyone else can ignore it. And ",[28,1259,1260],{},"separate infrastructure failures from test failures"," with a distinct exit path: a job that fails because a container could not pull an image should not look like a test failure, or the on-call engineer starts debugging the wrong thing.",[44,1263,1265],{"className":698,"code":1264,"language":700,"meta":49,"style":49},"$ pytest -q --junitxml=artifacts\u002Fjunit.xml || rc=$?\n$ python - \u003C\u003C'EOF' >> \"$GITHUB_STEP_SUMMARY\"\nimport xml.etree.ElementTree as ET\nroot = ET.parse(\"artifacts\u002Fjunit.xml\").getroot()\nfailures = [c for c in root.iter(\"testcase\") if c.find(\"failure\") is not None]\nprint(f\"### {len(failures)} failing test(s)\")\nfor case in failures[:20]:\n    print(f\"- `{case.get('classname')}::{case.get('name')}`\")\nEOF\n$ exit ${rc:-0}\n",[51,1266,1267,1272,1277,1282,1287,1292,1297,1302,1307,1312],{"__ignoreMap":49},[54,1268,1269],{"class":56,"line":57},[54,1270,1271],{},"$ pytest -q --junitxml=artifacts\u002Fjunit.xml || rc=$?\n",[54,1273,1274],{"class":56,"line":75},[54,1275,1276],{},"$ python - \u003C\u003C'EOF' >> \"$GITHUB_STEP_SUMMARY\"\n",[54,1278,1279],{"class":56,"line":87},[54,1280,1281],{},"import xml.etree.ElementTree as ET\n",[54,1283,1284],{"class":56,"line":93},[54,1285,1286],{},"root = ET.parse(\"artifacts\u002Fjunit.xml\").getroot()\n",[54,1288,1289],{"class":56,"line":99},[54,1290,1291],{},"failures = [c for c in root.iter(\"testcase\") if c.find(\"failure\") is not None]\n",[54,1293,1294],{"class":56,"line":105},[54,1295,1296],{},"print(f\"### {len(failures)} failing test(s)\")\n",[54,1298,1299],{"class":56,"line":114},[54,1300,1301],{},"for case in failures[:20]:\n",[54,1303,1304],{"class":56,"line":125},[54,1305,1306],{},"    print(f\"- `{case.get('classname')}::{case.get('name')}`\")\n",[54,1308,1309],{"class":56,"line":137},[54,1310,1311],{},"EOF\n",[54,1313,1314],{"class":56,"line":148},[54,1315,1316],{},"$ exit ${rc:-0}\n",[10,1318,1319],{},"That snippet costs nothing and changes the experience of a red pipeline from \"open the log and scroll\" to \"read three lines\". Combined with the artefact bundle, it is usually enough to diagnose without a re-run — which is the whole objective, because a re-run of an intermittent failure is a coin toss rather than an investigation.\nOne organisational habit closes the loop: when a CI-only failure is finally diagnosed, write the cause into the testing README as a one-line entry — \"March 2026: locale unset in the runner image made date assertions fail\". The list becomes the first thing to check next time, and after a handful of entries it is a far faster diagnostic than any tool, because the same handful of causes recur across every project.",[14,1321,1323],{"id":1322},"frequently-asked-questions","Frequently Asked Questions",[10,1325,1326,1329],{},[28,1327,1328],{},"Why do tests pass locally and fail in CI?","\nBecause something differs and the difference is usually invisible: test ordering, parallelism, CPU count, timezone, locale, filesystem case sensitivity, available memory, or an environment variable your shell sets and the runner does not.",[10,1331,1332,1335],{},[28,1333,1334],{},"Is an interactive debugger usable in CI?","\nRarely and only deliberately. A prompt with no terminal hangs the job. Capture artefacts by default, and use a remote debugger only for a manually triggered debugging run.",[10,1337,1338,1341],{},[28,1339,1340],{},"What should every CI job upload on failure?","\nThe junit XML, the full log, any coverage data, the pytest temporary directories, and a dump of the environment. Those five cover most investigations without a re-run.",[10,1343,1344,1347],{},[28,1345,1346],{},"How do I reproduce a CI failure locally?","\nRun the same container image with the same command, the same test order seed and the same worker count. Matching those four resolves the majority of CI-only failures.",[14,1349,1351],{"id":1350},"building-a-debuggable-pipeline-from-the-start","Building a debuggable pipeline from the start",[10,1353,1354],{},"Everything above is easier to add before it is needed. Five properties distinguish a pipeline that can be debugged from one that produces only a red cross, and all five are configuration rather than code.",[10,1356,1357,1360],{},[28,1358,1359],{},"Deterministic by default."," The seeds, the timezone, the locale and the worker count are pinned in the project configuration, so the same command produces the same run everywhere. This is the property that makes every other one useful.",[10,1362,1363,1366,1367,1370],{},[28,1364,1365],{},"Artefacts on failure, always."," The upload step is gated on ",[51,1368,1369],{},"always()"," and the bundle contains the log, the junit XML, the temporary directories and the environment. Nothing needs deciding at incident time.",[10,1372,1373,1376,1377,1380],{},[28,1374,1375],{},"One command."," The pipeline invokes the same ",[51,1378,1379],{},"make test"," a developer runs, in the same image. A workflow file containing a bespoke pytest invocation is a second, undocumented test configuration that will drift.",[10,1382,1383,1386],{},[28,1384,1385],{},"Fast feedback split from slow feedback."," Unit tests, integration tests and the scheduled jobs are separate, so a failure names its category before anyone reads a log. A single job running everything means every failure looks the same at a distance.",[10,1388,1389,1392],{},[28,1390,1391],{},"Reproducible images."," Pinned by digest, with the digest recorded in the run. \"It worked yesterday\" is answerable only when yesterday's image can be pulled today.",[44,1394,1398],{"className":1395,"code":1396,"language":1397,"meta":49,"style":49},"language-makefile shiki shiki-themes github-light github-dark","# Makefile — one entry point, used by developers and CI alike\nIMAGE ?= ghcr.io\u002Facme\u002Fci-python@sha256:6b1c...\n\ntest:\n    docker run --rm -v \"$(PWD):\u002Fsrc\" -w \u002Fsrc      -e TZ=UTC -e LC_ALL=C.UTF-8 -e PYTHONHASHSEED=0     --cpus=2 --memory=4g $(IMAGE)       pytest -q -ra -n 2 --basetemp=artifacts\u002Ftmp --junitxml=artifacts\u002Fjunit.xml\n","makefile",[51,1399,1400,1405,1410,1414,1419],{"__ignoreMap":49},[54,1401,1402],{"class":56,"line":57},[54,1403,1404],{},"# Makefile — one entry point, used by developers and CI alike\n",[54,1406,1407],{"class":56,"line":75},[54,1408,1409],{},"IMAGE ?= ghcr.io\u002Facme\u002Fci-python@sha256:6b1c...\n",[54,1411,1412],{"class":56,"line":87},[54,1413,458],{"emptyLinePlaceholder":457},[54,1415,1416],{"class":56,"line":93},[54,1417,1418],{},"test:\n",[54,1420,1421],{"class":56,"line":99},[54,1422,1423],{},"    docker run --rm -v \"$(PWD):\u002Fsrc\" -w \u002Fsrc      -e TZ=UTC -e LC_ALL=C.UTF-8 -e PYTHONHASHSEED=0     --cpus=2 --memory=4g $(IMAGE)       pytest -q -ra -n 2 --basetemp=artifacts\u002Ftmp --junitxml=artifacts\u002Fjunit.xml\n",[10,1425,1426],{},"A team that adopts that Makefile stops having CI-only failures for the ordinary reasons, because there is no longer a meaningful difference between the two environments. What remains — genuine concurrency bugs, resource-limit interactions, upstream flakiness — is the set of problems worth spending debugging effort on.",[237,1428,1430,1499],{"className":1429},[240],[242,1431,250,1436,250,1439,250,1442,250,1445,250,1447,250,1449,250,1451,250,1454,250,1457,250,1459,250,1461,250,1464,250,1467,250,1469,250,1471,250,1474,250,1477,250,1479,250,1481,250,1484,250,1487,250,1490,250,1492,250,1496],{"viewBox":1432,"role":245,"ariaLabelledBy":1433,"xmlns":249},"0 0 760 384",[1434,1435],"pipelineprops-t","pipelineprops-d",[252,1437,1438],{"id":1434},"Five properties of a debuggable pipeline",[256,1440,1441],{"id":1435},"A stack of five pipeline properties: deterministic defaults, artefacts uploaded on every failure, a single shared command, separated fast and slow jobs, and images pinned by digest.",[260,1443],{"x":262,"y":262,"width":263,"height":1444,"rx":265,"fill":266},"384",[268,1446,1438],{"x":270,"y":271,"textAnchor":272,"fontSize":273,"fontWeight":274,"fill":275},[260,1448],{"x":271,"y":1025,"width":274,"height":279,"rx":282,"fill":302,"stroke":574,"strokeWidth":584},[260,1450],{"x":271,"y":1025,"width":1029,"height":279,"rx":1030,"fill":574},[268,1452,1453],{"x":279,"y":1033,"fontSize":589,"fontWeight":274,"fill":275},"deterministic defaults",[268,1455,1456],{"x":1037,"y":1033,"textAnchor":1038,"fontSize":594,"fill":275},"seeds, TZ, locale and workers pinned in config",[260,1458],{"x":271,"y":1042,"width":274,"height":279,"rx":282,"fill":266,"stroke":283,"strokeWidth":584},[260,1460],{"x":271,"y":1042,"width":1029,"height":279,"rx":1030,"fill":283},[268,1462,1463],{"x":279,"y":1047,"fontSize":589,"fontWeight":274,"fill":275},"artefacts on failure",[268,1465,1466],{"x":1037,"y":1047,"textAnchor":1038,"fontSize":594,"fill":275},"gated on always(), never decided at incident time",[260,1468],{"x":271,"y":1054,"width":274,"height":279,"rx":282,"fill":302,"stroke":1026,"strokeWidth":584},[260,1470],{"x":271,"y":1054,"width":1029,"height":279,"rx":1030,"fill":1026},[268,1472,1473],{"x":279,"y":604,"fontSize":589,"fontWeight":274,"fill":275},"one command",[268,1475,1476],{"x":1037,"y":604,"textAnchor":1038,"fontSize":594,"fill":275},"developers and CI run the same make target",[260,1478],{"x":271,"y":1065,"width":274,"height":279,"rx":282,"fill":266,"stroke":275,"strokeWidth":584},[260,1480],{"x":271,"y":1065,"width":1029,"height":279,"rx":1030,"fill":275},[268,1482,1483],{"x":279,"y":1070,"fontSize":589,"fontWeight":274,"fill":275},"split fast and slow",[268,1485,1486],{"x":1037,"y":1070,"textAnchor":1038,"fontSize":594,"fill":275},"the failing job names the category",[260,1488],{"x":271,"y":1489,"width":274,"height":279,"rx":282,"fill":302,"stroke":574,"strokeWidth":584},"312",[260,1491],{"x":271,"y":1489,"width":1029,"height":279,"rx":1030,"fill":574},[268,1493,1495],{"x":279,"y":1494,"fontSize":589,"fontWeight":274,"fill":275},"342","images pinned by digest",[268,1497,1498],{"x":1037,"y":1494,"textAnchor":1038,"fontSize":594,"fill":275},"yesterday’s run can be reproduced today",[375,1500,1501],{},"All five are configuration, all five are cheap, and together they remove most of the reasons a failure is CI-only.",[1503,1504,1506],"h3",{"id":1505},"reading-the-exit-code-first","Reading the exit code first",[10,1508,1509],{},"Before any log is opened, the exit code narrows the problem to a category. pytest's own codes are documented and distinct, and the shell adds a few that matter more.",[10,1511,1512,1513,1515,1516,1519,1520,1523,1524,1526,1527,1529],{},"Exit ",[51,1514,303],{}," is ordinary test failures. Exit ",[51,1517,1518],{},"2"," is an interrupted run — usually a timeout or a cancellation. Exit ",[51,1521,1522],{},"3"," is an internal error, which almost always means a broken plugin or hook rather than a broken test. Exit ",[51,1525,1030],{}," is a usage error, typically a bad flag or a path that does not exist in the container. Exit ",[51,1528,568],{}," means no tests were collected at all, which is the silent failure worth alerting on separately: a green-looking pipeline that ran nothing.",[10,1531,1532,1533,1536,1537,1540,1541,1544,1545,1548],{},"Beyond pytest, ",[51,1534,1535],{},"137"," is a SIGKILL, which in a container means the memory limit was hit; ",[51,1538,1539],{},"139"," is a segmentation fault, pointing at a native extension; and ",[51,1542,1543],{},"124"," is the shell ",[51,1546,1547],{},"timeout"," command firing.",[44,1550,1552],{"className":698,"code":1551,"language":700,"meta":49,"style":49},"$ pytest -q; echo \"exit: $?\"\nexit: 5\n",[51,1553,1554,1559],{"__ignoreMap":49},[54,1555,1556],{"class":56,"line":57},[54,1557,1558],{},"$ pytest -q; echo \"exit: $?\"\n",[54,1560,1561],{"class":56,"line":75},[54,1562,1563],{},"exit: 5\n",[10,1565,1566,1567,1569],{},"Adding that one line to the job, and alerting on ",[51,1568,568],{}," specifically, catches the class of failure where a path change or a marker typo silently empties the run — the one CI failure mode that looks exactly like success.",[14,1571,1573],{"id":1572},"related-guides","Related guides",[19,1575,1576,1583,1590,1596],{},[22,1577,1578,1582],{},[421,1579,1581],{"href":1580},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Freproducing-ci-only-test-failures-locally\u002F","Reproducing CI-only test failures locally"," — the environment checklist in detail.",[22,1584,1585,1589],{},[421,1586,1588],{"href":1587},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fattaching-debugpy-to-a-container\u002F","Attaching debugpy to a container"," — a real debugger session against a containerised run.",[22,1591,1592,1595],{},[421,1593,1594],{"href":423},"Debugging flaky tests with pytest-rerunfailures"," — recording every attempt so the pattern is visible.",[22,1597,1598,1602],{},[421,1599,1601],{"href":1600},"\u002Fsystematic-debugging-performance-profiling\u002Finteractive-debugging-with-pdb-and-ipdb\u002Fpost-mortem-debugging-with-pdb-pm\u002F","Post-mortem debugging with pdb.pm()"," — dumping frames when no terminal exists.",[10,1604,1605,1606],{},"← Back to ",[421,1607,1609],{"href":1608},"\u002Fsystematic-debugging-performance-profiling\u002F","Systematic Debugging & Performance Profiling",[1611,1612,1613],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .s9eBZ, html code.shiki .s9eBZ{--shiki-default:#22863A;--shiki-dark:#85E89D}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}",{"title":49,"searchDepth":75,"depth":75,"links":1615},[1616,1617,1618,1619,1620,1621,1622,1623,1624,1625,1626,1629],{"id":16,"depth":75,"text":17},{"id":38,"depth":75,"text":39},{"id":380,"depth":75,"text":381},{"id":651,"depth":75,"text":652},{"id":832,"depth":75,"text":833},{"id":889,"depth":75,"text":890},{"id":1080,"depth":75,"text":1081},{"id":1192,"depth":75,"text":1193},{"id":1238,"depth":75,"text":1239},{"id":1322,"depth":75,"text":1323},{"id":1350,"depth":75,"text":1351,"children":1627},[1628],{"id":1505,"depth":87,"text":1506},{"id":1572,"depth":75,"text":1573},"Debug failures that only happen in CI: capturing artefacts, matching the environment, remote debuggers, container constraints, and turning a CI-only failure into a local one.","md",{"slug":1633,"type":1634,"breadcrumb":1635,"datePublished":1636,"dateModified":1636,"faq":1637,"howto":1646},"debugging-tests-in-ci-and-containers","topic","CI & Containers","2026-08-01",[1638,1640,1642,1644],{"q":1328,"a":1639},"Because something differs and the difference is usually invisible: test ordering, parallelism, CPU count, timezone, locale, filesystem case sensitivity, available memory, or an environment variable your shell sets and the runner does not.",{"q":1334,"a":1641},"Rarely and only deliberately. A prompt with no terminal hangs the job. Capture artefacts by default, and use a remote debugger only for a manually triggered debugging run.",{"q":1340,"a":1643},"The junit XML, the full log, any coverage data, the pytest temporary directories, and a dump of the environment. Those five cover most investigations without a re-run.",{"q":1346,"a":1645},"Run the same container image with the same command, the same test order seed and the same worker count. Matching those four resolves the majority of CI-only failures.",{"name":1647,"description":1648,"steps":1649},"How to debug a CI-only test failure","Capture enough evidence from CI to diagnose a failure without an interactive session.",[1650,1653,1656,1659],{"name":1651,"text":1652},"Capture artefacts on every failing run","Upload logs, junit XML, temporary directories and the environment dump.",{"name":1654,"text":1655},"Compare the environments","Diff the CI environment against your local one for ordering, parallelism, locale and CPU count.",{"name":1657,"text":1658},"Reproduce in the same image","Run the CI container locally with the same command and the same seeds.",{"name":1660,"text":1661},"Attach a remote debugger only if needed","Use debugpy for a manually triggered run when artefacts are insufficient.","\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers",{"title":5,"description":1630},"systematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Findex","ps5lo1TuCsy46H9hvfIXG-7g47IRhsPrRQ2gOBZxXWM",1785613403106]