[{"data":1,"prerenderedAt":995},["ShallowReactive",2],{"page-\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fcapturing-artifacts-from-a-failed-ci-test-run\u002F":3},{"id":4,"title":5,"body":6,"description":961,"extension":962,"meta":963,"navigation":85,"path":991,"seo":992,"stem":993,"__hash__":994},"content\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fcapturing-artifacts-from-a-failed-ci-test-run\u002Findex.md","Capturing Artifacts from a Failed CI Test Run",{"type":7,"value":8,"toc":950},"minimark",[9,13,16,21,41,45,262,291,382,520,524,538,541,548,552,562,573,659,663,666,688,698,708,714,802,806,816,825,829,876,880,889,902,908,912,941,946],[10,11,12],"p",{},"A CI failure you cannot reproduce is only debuggable through what the job left behind. Too often that is a truncated log: the assertion line, perhaps a captured log section, and nothing about the state that led there. The temporary directory with the generated file that failed validation is gone. The database the integration test wrote to was torn down with its container. The screenshot that would have shown the browser test's error dialog was never taken. Re-running the job may pass, and the only evidence of the bug disappears.",[10,14,15],{},"Keeping evidence is cheap if it is planned. pytest knows exactly when a test fails and has access to its fixtures, so a hook can copy the relevant state into a directory at that moment. The CI system then uploads that directory, and only when the job has failed, so passing runs cost nothing. With a consistent naming scheme, every artefact maps back to the test that produced it.",[17,18,20],"h2",{"id":19},"prerequisites","Prerequisites",[22,23,24,32],"ul",{},[25,26,27,31],"li",{},[28,29,30],"code",{},"pytest >= 8.0",", a CI system with artefact upload (GitHub Actions, GitLab CI or similar).",[25,33,34,35,40],{},"Background from ",[36,37,39],"a",{"href":38},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002F","Debugging tests in CI and containers",".",[17,42,44],{"id":43},"solution","Solution",[46,47,52],"pre",{"className":48,"code":49,"language":50,"meta":51,"style":51},"language-python shiki shiki-themes github-light github-dark","# conftest.py\nimport re\nimport shutil\nfrom pathlib import Path\n\nimport pytest\n\nARTIFACTS = Path(\"test-artifacts\")\n\ndef _slug(nodeid: str) -> str:\n    return re.sub(r\"[^A-Za-z0-9_.-]+\", \"_\", nodeid)[:150]\n\n@pytest.hookimpl(wrapper=True)\ndef pytest_runtest_makereport(item, call):\n    rep = yield\n    if rep.failed and rep.when in (\"setup\", \"call\"):\n        out = ARTIFACTS \u002F _slug(item.nodeid)\n        out.mkdir(parents=True, exist_ok=True)\n        (out \u002F \"report.txt\").write_text(rep.longreprtext)\n        for name, content in rep.sections:\n            (out \u002F f\"{_slug(name)}.txt\").write_text(content)\n        tmp = item.funcargs.get(\"tmp_path\")\n        if tmp and Path(tmp).exists():\n            shutil.copytree(tmp, out \u002F \"tmp_path\", dirs_exist_ok=True)\n        for collect in item.stash.get(ARTIFACT_COLLECTORS, []):\n            collect(out)                    # fixtures register extra collectors\n    return rep\n\nARTIFACT_COLLECTORS = pytest.StashKey[list]()\n\n@pytest.fixture\ndef artifact(request):\n    \"\"\"Let a fixture or test register a callable that saves evidence on failure.\"\"\"\n    collectors = request.node.stash.setdefault(ARTIFACT_COLLECTORS, [])\n    return collectors.append\n","python","",[28,53,54,62,68,74,80,87,93,98,104,109,115,121,126,132,138,144,150,156,162,168,174,180,186,192,198,204,210,216,221,227,232,238,244,250,256],{"__ignoreMap":51},[55,56,59],"span",{"class":57,"line":58},"line",1,[55,60,61],{},"# conftest.py\n",[55,63,65],{"class":57,"line":64},2,[55,66,67],{},"import re\n",[55,69,71],{"class":57,"line":70},3,[55,72,73],{},"import shutil\n",[55,75,77],{"class":57,"line":76},4,[55,78,79],{},"from pathlib import Path\n",[55,81,83],{"class":57,"line":82},5,[55,84,86],{"emptyLinePlaceholder":85},true,"\n",[55,88,90],{"class":57,"line":89},6,[55,91,92],{},"import pytest\n",[55,94,96],{"class":57,"line":95},7,[55,97,86],{"emptyLinePlaceholder":85},[55,99,101],{"class":57,"line":100},8,[55,102,103],{},"ARTIFACTS = Path(\"test-artifacts\")\n",[55,105,107],{"class":57,"line":106},9,[55,108,86],{"emptyLinePlaceholder":85},[55,110,112],{"class":57,"line":111},10,[55,113,114],{},"def _slug(nodeid: str) -> str:\n",[55,116,118],{"class":57,"line":117},11,[55,119,120],{},"    return re.sub(r\"[^A-Za-z0-9_.-]+\", \"_\", nodeid)[:150]\n",[55,122,124],{"class":57,"line":123},12,[55,125,86],{"emptyLinePlaceholder":85},[55,127,129],{"class":57,"line":128},13,[55,130,131],{},"@pytest.hookimpl(wrapper=True)\n",[55,133,135],{"class":57,"line":134},14,[55,136,137],{},"def pytest_runtest_makereport(item, call):\n",[55,139,141],{"class":57,"line":140},15,[55,142,143],{},"    rep = yield\n",[55,145,147],{"class":57,"line":146},16,[55,148,149],{},"    if rep.failed and rep.when in (\"setup\", \"call\"):\n",[55,151,153],{"class":57,"line":152},17,[55,154,155],{},"        out = ARTIFACTS \u002F _slug(item.nodeid)\n",[55,157,159],{"class":57,"line":158},18,[55,160,161],{},"        out.mkdir(parents=True, exist_ok=True)\n",[55,163,165],{"class":57,"line":164},19,[55,166,167],{},"        (out \u002F \"report.txt\").write_text(rep.longreprtext)\n",[55,169,171],{"class":57,"line":170},20,[55,172,173],{},"        for name, content in rep.sections:\n",[55,175,177],{"class":57,"line":176},21,[55,178,179],{},"            (out \u002F f\"{_slug(name)}.txt\").write_text(content)\n",[55,181,183],{"class":57,"line":182},22,[55,184,185],{},"        tmp = item.funcargs.get(\"tmp_path\")\n",[55,187,189],{"class":57,"line":188},23,[55,190,191],{},"        if tmp and Path(tmp).exists():\n",[55,193,195],{"class":57,"line":194},24,[55,196,197],{},"            shutil.copytree(tmp, out \u002F \"tmp_path\", dirs_exist_ok=True)\n",[55,199,201],{"class":57,"line":200},25,[55,202,203],{},"        for collect in item.stash.get(ARTIFACT_COLLECTORS, []):\n",[55,205,207],{"class":57,"line":206},26,[55,208,209],{},"            collect(out)                    # fixtures register extra collectors\n",[55,211,213],{"class":57,"line":212},27,[55,214,215],{},"    return rep\n",[55,217,219],{"class":57,"line":218},28,[55,220,86],{"emptyLinePlaceholder":85},[55,222,224],{"class":57,"line":223},29,[55,225,226],{},"ARTIFACT_COLLECTORS = pytest.StashKey[list]()\n",[55,228,230],{"class":57,"line":229},30,[55,231,86],{"emptyLinePlaceholder":85},[55,233,235],{"class":57,"line":234},31,[55,236,237],{},"@pytest.fixture\n",[55,239,241],{"class":57,"line":240},32,[55,242,243],{},"def artifact(request):\n",[55,245,247],{"class":57,"line":246},33,[55,248,249],{},"    \"\"\"Let a fixture or test register a callable that saves evidence on failure.\"\"\"\n",[55,251,253],{"class":57,"line":252},34,[55,254,255],{},"    collectors = request.node.stash.setdefault(ARTIFACT_COLLECTORS, [])\n",[55,257,259],{"class":57,"line":258},35,[55,260,261],{},"    return collectors.append\n",[46,263,265],{"className":48,"code":264,"language":50,"meta":51,"style":51},"# An integration fixture that dumps its database on failure.\n@pytest.fixture\ndef db(postgres, artifact):\n    artifact(lambda out: postgres.dump_to(out \u002F \"db.sql\"))\n    yield postgres.connect()\n",[28,266,267,272,276,281,286],{"__ignoreMap":51},[55,268,269],{"class":57,"line":58},[55,270,271],{},"# An integration fixture that dumps its database on failure.\n",[55,273,274],{"class":57,"line":64},[55,275,237],{},[55,277,278],{"class":57,"line":70},[55,279,280],{},"def db(postgres, artifact):\n",[55,282,283],{"class":57,"line":76},[55,284,285],{},"    artifact(lambda out: postgres.dump_to(out \u002F \"db.sql\"))\n",[55,287,288],{"class":57,"line":82},[55,289,290],{},"    yield postgres.connect()\n",[46,292,296],{"className":293,"code":294,"language":295,"meta":51,"style":51},"language-yaml shiki shiki-themes github-light github-dark","# .github\u002Fworkflows\u002Ftests.yml (excerpt)\n      - run: pytest --junitxml=test-artifacts\u002Fjunit.xml -ra\n      - uses: actions\u002Fupload-artifact@v4\n        if: failure()\n        with:\n          name: test-artifacts-${{ matrix.python }}-${{ github.run_attempt }}\n          path: test-artifacts\u002F\n          retention-days: 14\n","yaml",[28,297,298,304,321,333,343,351,361,371],{"__ignoreMap":51},[55,299,300],{"class":57,"line":58},[55,301,303],{"class":302},"sJ8bj","# .github\u002Fworkflows\u002Ftests.yml (excerpt)\n",[55,305,306,310,314,317],{"class":57,"line":64},[55,307,309],{"class":308},"sVt8B","      - ",[55,311,313],{"class":312},"s9eBZ","run",[55,315,316],{"class":308},": ",[55,318,320],{"class":319},"sZZnC","pytest --junitxml=test-artifacts\u002Fjunit.xml -ra\n",[55,322,323,325,328,330],{"class":57,"line":70},[55,324,309],{"class":308},[55,326,327],{"class":312},"uses",[55,329,316],{"class":308},[55,331,332],{"class":319},"actions\u002Fupload-artifact@v4\n",[55,334,335,338,340],{"class":57,"line":76},[55,336,337],{"class":312},"        if",[55,339,316],{"class":308},[55,341,342],{"class":319},"failure()\n",[55,344,345,348],{"class":57,"line":82},[55,346,347],{"class":312},"        with",[55,349,350],{"class":308},":\n",[55,352,353,356,358],{"class":57,"line":89},[55,354,355],{"class":312},"          name",[55,357,316],{"class":308},[55,359,360],{"class":319},"test-artifacts-${{ matrix.python }}-${{ github.run_attempt }}\n",[55,362,363,366,368],{"class":57,"line":95},[55,364,365],{"class":312},"          path",[55,367,316],{"class":308},[55,369,370],{"class":319},"test-artifacts\u002F\n",[55,372,373,376,378],{"class":57,"line":100},[55,374,375],{"class":312},"          retention-days",[55,377,316],{"class":308},[55,379,381],{"class":380},"sj4cs","14\n",[383,384,387,516],"figure",{"className":385},[386],"diagram",[388,389,396,397,396,401,396,405,396,423,396,431,396,441,396,451,396,457,396,462,396,468,396,473,396,478,396,482,396,486,396,491,396,497,396,500,396,504,396,507,396,511],"svg",{"viewBox":390,"role":391,"ariaLabelledBy":392,"xmlns":395},"0 0 800 256","img",[393,394],"ca-t","ca-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[398,399,400],"title",{"id":393},"Collecting and uploading evidence only on failure",[402,403,404],"desc",{"id":394},"When a test fails, the makereport hook writes the failure report, captured output and tmp_path into a folder named after the test, and runs any collectors that fixtures registered, such as a database dump. After pytest exits with failures, the CI upload step runs only because the job failed and uploads the whole artefact directory.",[406,407,408,409,396],"defs",{},"\n    ",[410,411,418],"marker",{"id":412,"viewBox":413,"refX":414,"refY":415,"markerWidth":416,"markerHeight":416,"orient":417},"ca-a","0 0 10 10","9","5","7","auto-start-reverse",[419,420],"path",{"d":421,"fill":422},"M0 0 L10 5 L0 10 z","#81b29a",[424,425],"rect",{"x":426,"y":426,"width":427,"height":428,"rx":429,"fill":430},"0","800","256","14","#fffdf8",[432,433,440],"text",{"x":434,"y":435,"textAnchor":436,"fontSize":437,"fontWeight":438,"fill":439},"400","28","middle","15.5","700","#3d405b","Evidence captured at the moment of failure",[424,442],{"x":443,"y":444,"width":445,"height":446,"rx":447,"fill":448,"stroke":449,"strokeWidth":450},"26","96","150","60","10","#fbe9e3","#e07a5f","2",[432,452,456],{"x":453,"y":454,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":439},"101","122","12","test fails",[432,458,461],{"x":453,"y":459,"textAnchor":436,"fontSize":460,"fill":439},"142","10.5","fixtures still alive",[424,463],{"x":464,"y":446,"width":465,"height":466,"rx":455,"fill":467,"stroke":422,"strokeWidth":450},"220","260","132","#e6f0ea",[432,469,472],{"x":470,"y":471,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":439},"350","86","test-artifacts\u002F\u003Cnodeid>\u002F",[432,474,477],{"x":475,"y":476,"fontSize":460,"fill":439},"240","112","report.txt · captured logs",[432,479,481],{"x":475,"y":480,"fontSize":460,"fill":439},"134","tmp_path\u002F",[432,483,485],{"x":475,"y":484,"fontSize":460,"fill":439},"156","db.sql · screenshot.png",[432,487,490],{"x":475,"y":488,"fontSize":460,"fill":489},"178","#2a5f49","junit.xml (whole run)",[57,492],{"x1":488,"y1":493,"x2":494,"y2":493,"stroke":422,"strokeWidth":495,"markerEnd":496},"126","216","1.8","url(#ca-a)",[424,498],{"x":499,"y":444,"width":475,"height":446,"rx":447,"fill":439},"534",[432,501,503],{"x":502,"y":454,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":430},"654","upload-artifact",[432,505,506],{"x":502,"y":459,"textAnchor":436,"fontSize":460,"fill":430},"if: failure() · 14 days",[57,508],{"x1":509,"y1":493,"x2":510,"y2":493,"stroke":422,"strokeWidth":495,"markerEnd":496},"482","530",[432,512,515],{"x":434,"y":513,"textAnchor":436,"fontSize":514,"fill":439},"228","11","Passing runs write nothing and upload nothing.",[517,518,519],"figcaption",{},"Collecting inside pytest means fixtures still exist when evidence is taken; uploading from CI keeps it after the runner is gone.",[17,521,523],{"id":522},"why-this-works","Why this works",[10,525,526,529,530,533,534,537],{},[28,527,528],{},"pytest_runtest_makereport"," runs three times per test — for setup, call and teardown — and receives the report for each phase. The hook sees the report after pytest has built it and before fixtures are torn down for the call phase, so ",[28,531,532],{},"item.funcargs"," still holds live fixture values: the ",[28,535,536],{},"tmp_path"," directory exists, the database connection is open, the browser page is still showing whatever went wrong. That is the only moment when most evidence can be collected.",[10,539,540],{},"The collector registry lets each fixture decide what evidence it can provide, instead of the hook knowing about every kind of resource. A database fixture registers a dump, a browser fixture registers a screenshot and the page HTML, an HTTP-mocking fixture registers the recorded requests. Tests that do not use those fixtures pay nothing.",[10,542,543,544,547],{},"The CI side is deliberately dumb. ",[28,545,546],{},"if: failure()"," runs the upload only when a previous step failed; the step uploads whatever is in the directory. Including the Python version and run attempt in the artefact name keeps matrix jobs and re-runs from overwriting each other's evidence.",[17,549,551],{"id":550},"the-junit-report-as-an-index","The JUnit report as an index",[10,553,554,557,558,561],{},[28,555,556],{},"--junitxml"," produces a structured record of every test in the run: name, outcome, duration, failure message and captured output. CI systems render it as a test summary, but it is also the index into the artefact directory. Each failed ",[28,559,560],{},"testcase"," element's name and class map to a node id, and that node id maps to a folder. When reviewing a failed run days later, start with the JUnit summary to see which tests failed, then open only their folders.",[10,563,564,565,568,569,572],{},"Two settings make the report more useful: ",[28,566,567],{},"junit_family = \"xunit2\""," for the modern schema, and ",[28,570,571],{},"junit_logging = \"all\""," to include captured logs and output in the XML itself, so even without the artefact directory the essentials survive in the test summary view.",[383,574,576,656],{"className":575},[386],[388,577,396,582,396,585,396,588,396,595,396,598,396,601,396,607,396,611,396,615,396,620,396,626,396,631,396,634,396,637,396,641,396,644,396,647,396,652],{"viewBox":578,"role":391,"ariaLabelledBy":579,"xmlns":395},"0 0 800 200",[580,581],"caj-t","caj-d",[398,583,584],{"id":580},"From JUnit summary to per-test folder",[402,586,587],{"id":581},"The JUnit XML report lists every test with its outcome. A failed test case, identified by class and name, maps to its pytest node id, which maps to the artefact folder of the same name containing that test's report, logs, tmp_path and database dump.",[406,589,408,590,396],{},[410,591,593],{"id":592,"viewBox":413,"refX":414,"refY":415,"markerWidth":416,"markerHeight":416,"orient":417},"caj-a",[419,594],{"d":421,"fill":422},[424,596],{"x":426,"y":426,"width":427,"height":597,"rx":429,"fill":430},"200",[432,599,600],{"x":434,"y":435,"textAnchor":436,"fontSize":437,"fontWeight":438,"fill":439},"Summary first, then only the folders you need",[424,602],{"x":443,"y":603,"width":464,"height":604,"rx":455,"fill":605,"stroke":439,"strokeWidth":606},"70","100","#f4f1de","1.5",[432,608,610],{"x":609,"y":444,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":439},"136","junit.xml",[432,612,614],{"x":609,"y":613,"textAnchor":436,"fontSize":460,"fill":439},"120","1,204 passed",[432,616,619],{"x":609,"y":617,"textAnchor":436,"fontSize":460,"fill":618},"140","#8f3d22","2 failed",[424,621],{"x":622,"y":623,"width":464,"height":623,"rx":455,"fill":624,"stroke":625,"strokeWidth":450},"290","80","#f7f0da","#f2cc8f",[432,627,630],{"x":434,"y":628,"textAnchor":436,"fontSize":629,"fontWeight":438,"fill":439},"110","11.5","node id",[432,632,633],{"x":434,"y":466,"textAnchor":436,"fontSize":447,"fill":439},"tests\u002Ftest_export.py::test_csv",[424,635],{"x":636,"y":603,"width":464,"height":604,"rx":455,"fill":467,"stroke":422,"strokeWidth":450},"554",[432,638,640],{"x":639,"y":444,"textAnchor":436,"fontSize":629,"fontWeight":438,"fill":439},"664","artefact folder",[432,642,643],{"x":639,"y":613,"textAnchor":436,"fontSize":460,"fill":439},"report · logs · tmp_path",[432,645,646],{"x":639,"y":617,"textAnchor":436,"fontSize":460,"fill":489},"db.sql",[57,648],{"x1":649,"y1":613,"x2":650,"y2":613,"stroke":422,"strokeWidth":495,"markerEnd":651},"248","286","url(#caj-a)",[57,653],{"x1":654,"y1":613,"x2":655,"y2":613,"stroke":422,"strokeWidth":495,"markerEnd":651},"512","550",[517,657,658],{},"Consistent naming turns a directory of evidence into something you can navigate from the test summary.",[17,660,662],{"id":661},"evidence-for-different-kinds-of-tests","Evidence for different kinds of tests",[10,664,665],{},"What counts as evidence depends on what the test exercises, and it is worth deciding per fixture rather than collecting everything everywhere.",[10,667,668,672,673,676,677,680,681,684,685,40],{},[669,670,671],"strong",{},"Unit tests"," rarely need more than the report and captured output. Their inputs are in the test code; if a failure is not reproducible from that, the problem is usually environmental — a timezone, locale or environment variable — and the most useful extra artefact is a small ",[28,674,675],{},"environment.json"," written once per session with ",[28,678,679],{},"platform",", ",[28,682,683],{},"sys.version",", the relevant environment variables and installed package versions from ",[28,686,687],{},"importlib.metadata",[10,689,690,693,694,697],{},[669,691,692],{},"Integration tests"," need the state of the systems they touched. A SQL dump of the test database (or just the tables the test wrote), the recorded HTTP interactions from a mocking fixture, and the logs of any service containers — collected with ",[28,695,696],{},"docker logs"," into the artefact folder — usually explain a failure that the assertion alone does not.",[10,699,700,703,704,707],{},[669,701,702],{},"Browser tests"," need visual evidence. Playwright and Selenium fixtures should register a screenshot, the page HTML and, for Playwright, a trace file (",[28,705,706],{},"context.tracing.stop(path=...)","), which records every action, network request and DOM snapshot. Opening that trace in the Playwright trace viewer replays the failed test step by step, which is the closest thing to reproducing the failure without re-running it.",[10,709,710,713],{},[669,711,712],{},"Crashes and hangs"," need process-level evidence: faulthandler output written to a file, core dumps, and the pytest-timeout stack dump. These must be configured before the run, because the hook will never be called.",[383,715,717,799],{"className":716},[386],[388,718,396,723,396,726,396,729,396,732,396,735,396,741,396,745,396,749,396,752,396,755,396,759,396,762,396,765,396,769,396,772,396,776,396,779,396,782,396,786,396,790,396,793,396,796],{"viewBox":719,"role":391,"ariaLabelledBy":720,"xmlns":395},"0 0 800 236",[721,722],"cak-t","cak-d",[398,724,725],{"id":721},"Evidence by test kind",[402,727,728],{"id":722},"Four columns list useful evidence per test kind. Unit tests: report, captured output and an environment snapshot. Integration tests: database dump, recorded HTTP traffic and service container logs. Browser tests: screenshot, page HTML and a Playwright trace. Crashes and hangs: faulthandler output, core dumps and timeout stack dumps, configured before the run.",[424,730],{"x":426,"y":426,"width":427,"height":731,"rx":429,"fill":430},"236",[432,733,734],{"x":434,"y":435,"textAnchor":436,"fontSize":437,"fontWeight":438,"fill":439},"Collect what explains this kind of failure",[424,736],{"x":737,"y":738,"width":739,"height":740,"rx":514,"fill":467,"stroke":422,"strokeWidth":495},"20","50","180","164",[432,742,744],{"x":628,"y":743,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":439},"76","unit",[432,746,748],{"x":628,"y":747,"textAnchor":436,"fontSize":460,"fill":439},"108","report · output",[432,750,675],{"x":628,"y":751,"textAnchor":436,"fontSize":460,"fill":439},"130",[424,753],{"x":754,"y":738,"width":739,"height":740,"rx":514,"fill":624,"stroke":625,"strokeWidth":495},"215",[432,756,758],{"x":757,"y":743,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":439},"305","integration",[432,760,761],{"x":757,"y":747,"textAnchor":436,"fontSize":460,"fill":439},"db dump",[432,763,764],{"x":757,"y":751,"textAnchor":436,"fontSize":460,"fill":439},"HTTP recordings",[432,766,768],{"x":757,"y":767,"textAnchor":436,"fontSize":460,"fill":439},"152","container logs",[424,770],{"x":771,"y":738,"width":739,"height":740,"rx":514,"fill":605,"stroke":439,"strokeWidth":606},"410",[432,773,775],{"x":774,"y":743,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":439},"500","browser",[432,777,778],{"x":774,"y":747,"textAnchor":436,"fontSize":460,"fill":439},"screenshot · HTML",[432,780,781],{"x":774,"y":751,"textAnchor":436,"fontSize":460,"fill":439},"Playwright trace",[424,783],{"x":784,"y":738,"width":785,"height":740,"rx":514,"fill":448,"stroke":449,"strokeWidth":495},"605","175",[432,787,789],{"x":788,"y":743,"textAnchor":436,"fontSize":455,"fontWeight":438,"fill":439},"692","crash \u002F hang",[432,791,792],{"x":788,"y":747,"textAnchor":436,"fontSize":460,"fill":439},"faulthandler file",[432,794,795],{"x":788,"y":751,"textAnchor":436,"fontSize":460,"fill":439},"core dump",[432,797,798],{"x":788,"y":767,"textAnchor":436,"fontSize":460,"fill":618},"set up before the run",[517,800,801],{},"Each fixture registers the evidence it can provide; the hook just calls whatever was registered.",[17,803,805],{"id":804},"using-the-artefacts-a-short-debugging-routine","Using the artefacts: a short debugging routine",[10,807,808,809,811,812,815],{},"Artefacts are only valuable if someone actually opens them, and a fixed routine that everyone on the team follows makes that quick. Download the artefact bundle for the failed job and open ",[28,810,610],{}," or the CI summary to list the failures. For each one, open its folder and read ",[28,813,814],{},"report.txt"," first — it is the same failure output as the log, but complete rather than truncated. Then read the captured log section from the same folder, looking for the last warning or error before the failure. Only then, with a hypothesis in mind, open the heavier evidence: the database dump, the recorded HTTP traffic, the trace.",[10,817,818,819,821,822,824],{},"Most CI-only failures fall into a handful of categories, and the artefacts usually settle which one within minutes. A difference in ",[28,820,675],{}," against a local run points at configuration. Rows in the database dump that the test did not create point at leaked state from an earlier test. A recorded HTTP call to a real host points at a missing mock. And an empty ",[28,823,536],{}," where the test expected output points at a step that silently did nothing, often because a path or flag differs on the runner. Each category has a known next step, and the artefacts get you to it without re-running the job and hoping it fails again. Over time, the categories you keep hitting also tell you which fixtures deserve better evidence collectors, so the routine gets faster the more it is used.",[17,826,828],{"id":827},"edge-cases-and-failure-modes","Edge cases and failure modes",[22,830,831,837,846,852,866],{},[25,832,833,836],{},[669,834,835],{},"xdist workers."," Each worker writes into the same directory; node-id-based folder names keep them apart. Avoid per-worker global files unless the worker id is in the name.",[25,838,839,845],{},[669,840,841,842,844],{},"Huge ",[28,843,536],{}," contents."," Copying gigabytes of generated data fills artefact storage. Cap the copy size or copy selectively.",[25,847,848,851],{},[669,849,850],{},"Secrets in artefacts."," Logs and dumps may contain tokens or personal data. Scrub known secret patterns and keep artefacts private to the repository.",[25,853,854,857,858,861,862,865],{},[669,855,856],{},"Crashes that kill pytest."," A segfault never reaches the hook. Enable core dumps with ",[28,859,860],{},"ulimit -c unlimited"," and ",[28,863,864],{},"faulthandler"," output to a file in the artefact directory.",[25,867,868,871,872,875],{},[669,869,870],{},"Teardown failures."," Errors in fixture teardown produce a report with ",[28,873,874],{},"when == \"teardown\"","; decide whether to collect for those too — the fixtures may already be partly gone.",[17,877,879],{"id":878},"frequently-asked-questions","Frequently Asked Questions",[10,881,882,885,886,888],{},[669,883,884],{},"What should I save when a CI test fails?","\nAt minimum the JUnit XML report and full pytest output. Beyond that, whatever the failing test's own evidence is: captured logs, the ",[28,887,536],{}," contents, HTTP recordings, screenshots for browser tests, database dumps for integration tests, and core dumps for crashes.",[10,890,891,894,895,897,898,901],{},[669,892,893],{},"How do I upload artefacts only when tests fail?","\nWrite artefacts to a known directory from a pytest hook that runs only for failed tests, then use an upload step with ",[28,896,546],{}," in GitHub Actions or ",[28,899,900],{},"when: on_failure"," in GitLab CI so successful runs do not pay the storage cost.",[10,903,904,907],{},[669,905,906],{},"How long should CI artefacts be kept?","\nLong enough to debug a failure after someone notices it — usually 7 to 14 days. Flaky-test investigations benefit from longer retention on the default branch.",[17,909,911],{"id":910},"related","Related",[22,913,914,920,927,934],{},[25,915,916,919],{},[36,917,918],{"href":38},"Debugging Tests in CI and Containers"," — CI-only failure strategy.",[25,921,922,926],{},[36,923,925],{"href":924},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Freproducing-ci-only-test-failures-locally\u002F","Reproducing CI-Only Test Failures Locally"," — using the evidence.",[25,928,929,933],{},[36,930,932],{"href":931},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fdebugging-a-test-that-only-fails-under-xdist\u002F","Debugging a Test That Only Fails Under xdist"," — parallel-only failures.",[25,935,936,940],{},[36,937,939],{"href":938},"\u002Fsystematic-debugging-performance-profiling\u002Flogging-and-observability-for-debugging\u002Fstructured-logging-that-survives-pytest-capture\u002F","Structured Logging That Survives pytest Capture"," — logs worth saving.",[10,942,943,944],{},"← Back to ",[36,945,918],{"href":38},[947,948,949],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .s9eBZ, html code.shiki .s9eBZ{--shiki-default:#22863A;--shiki-dark:#85E89D}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}",{"title":51,"searchDepth":64,"depth":64,"links":951},[952,953,954,955,956,957,958,959,960],{"id":19,"depth":64,"text":20},{"id":43,"depth":64,"text":44},{"id":522,"depth":64,"text":523},{"id":550,"depth":64,"text":551},{"id":661,"depth":64,"text":662},{"id":804,"depth":64,"text":805},{"id":827,"depth":64,"text":828},{"id":878,"depth":64,"text":879},{"id":910,"depth":64,"text":911},"Keep the evidence when CI tests fail: JUnit XML, per-test logs, screenshots, database dumps and core files as artefacts, collected by a pytest hook only on failure and uploaded by the workflow.","md",{"slug":964,"type":965,"breadcrumb":966,"datePublished":967,"dateModified":967,"faq":968,"howto":975},"capturing-artifacts-from-a-failed-ci-test-run","article","CI failure artefacts","2026-09-18",[969,971,973],{"q":884,"a":970},"At minimum the JUnit XML report and full pytest output. Beyond that, whatever the failing test's own evidence is: captured logs, the tmp_path contents, HTTP recordings, screenshots for browser tests, database dumps for integration tests, and core dumps for crashes.",{"q":893,"a":972},"Write artefacts to a known directory from a pytest hook that runs only for failed tests, then use an upload step with if: failure() in GitHub Actions or when: on_failure in GitLab CI so successful runs do not pay the storage cost.",{"q":906,"a":974},"Long enough to debug a failure after someone notices it — usually 7 to 14 days. Flaky-test investigations benefit from longer retention on the default branch.",{"name":976,"description":977,"steps":978},"How to capture artefacts from failed CI tests","Collect per-test evidence on failure with a pytest hook, write it to one directory, and upload it from the CI workflow only when the job fails.",[979,982,985,988],{"name":980,"text":981},"Produce a machine-readable report","Run pytest with --junitxml so every run produces a structured result file.",{"name":983,"text":984},"Collect evidence on failure","Use a pytest_runtest_makereport hook to copy logs, tmp_path and other evidence for failed tests.",{"name":986,"text":987},"Upload only on failure","Add an upload-artifact step with if: failure() pointing at the artefact directory.",{"name":989,"text":990},"Keep names traceable","Name artefact folders after the test node id so files map back to the failing test.","\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fcapturing-artifacts-from-a-failed-ci-test-run",{"title":5,"description":961},"systematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fcapturing-artifacts-from-a-failed-ci-test-run\u002Findex","lJuKk1GSgKjMLSou5iMqG8seE4L4xFdpFCuYvkcBDZY",1789718767513]