[{"data":1,"prerenderedAt":887},["ShallowReactive",2],{"page-\u002Ftesting-async-and-concurrent-python\u002Ftesting-threads-and-race-conditions\u002Fdumping-stacks-on-deadlock-with-faulthandler\u002F":3},{"id":4,"title":5,"body":6,"description":850,"extension":851,"meta":852,"navigation":141,"path":883,"seo":884,"stem":885,"__hash__":886},"content\u002Ftesting-async-and-concurrent-python\u002Ftesting-threads-and-race-conditions\u002Fdumping-stacks-on-deadlock-with-faulthandler\u002Findex.md","Dumping Stacks on Deadlock with faulthandler",{"type":7,"value":8,"toc":839},"minimark",[9,18,23,52,56,62,115,169,177,303,307,313,323,327,379,383,386,395,401,411,417,531,535,538,560,570,576,582,593,597,600,664,675,693,755,759,768,786,796,800,830,835],[10,11,12,13,17],"p",{},"A deadlocked test does not fail; it stops. The job runs to its platform limit, gets killed, and the log ends mid-sentence with no indication of which test was running or what it was waiting for. ",[14,15,16],"code",{},"faulthandler"," closes that gap for the cost of one configuration line: after a chosen number of seconds with no progress, every thread's stack is written to stderr, and the resulting dump usually contains the complete diagnosis.",[19,20,22],"h2",{"id":21},"prerequisites","Prerequisites",[24,25,26,33,43],"ul",{},[27,28,29,30,32],"li",{},"Python 3.3+; ",[14,31,16],{}," is in the standard library and pytest enables it by default.",[27,34,35,38,39,42],{},[14,36,37],{},"pytest >= 8.0",", plus ",[14,40,41],{},"pytest-timeout"," for the failure half of the arrangement.",[27,44,45,46,51],{},"A suite with real threads or subprocesses — a single-threaded hang is better served by ",[47,48,50],"a",{"href":49},"\u002Fsystematic-debugging-performance-profiling\u002Finteractive-debugging-with-pdb-and-ipdb\u002F","pdb",".",[19,53,55],{"id":54},"solution","Solution",[10,57,58,59,61],{},"Arm ",[14,60,16],{}," just below the suite's own ceiling so the dump lands before anything kills the process.",[63,64,69],"pre",{"className":65,"code":66,"language":67,"meta":68,"style":68},"language-toml shiki shiki-themes github-light github-dark","# pyproject.toml\n[tool.pytest.ini_options]\n# Dump every thread's stack after 45 s of a single test making no progress.\nfaulthandler_timeout = 45\n# And fail the test at 60 s, giving the dump 15 s of headroom to be written.\ntimeout = 60\ntimeout_method = \"thread\"\n","toml","",[14,70,71,79,85,91,97,103,109],{"__ignoreMap":68},[72,73,76],"span",{"class":74,"line":75},"line",1,[72,77,78],{},"# pyproject.toml\n",[72,80,82],{"class":74,"line":81},2,[72,83,84],{},"[tool.pytest.ini_options]\n",[72,86,88],{"class":74,"line":87},3,[72,89,90],{},"# Dump every thread's stack after 45 s of a single test making no progress.\n",[72,92,94],{"class":74,"line":93},4,[72,95,96],{},"faulthandler_timeout = 45\n",[72,98,100],{"class":74,"line":99},5,[72,101,102],{},"# And fail the test at 60 s, giving the dump 15 s of headroom to be written.\n",[72,104,106],{"class":74,"line":105},6,[72,107,108],{},"timeout = 60\n",[72,110,112],{"class":74,"line":111},7,[72,113,114],{},"timeout_method = \"thread\"\n",[63,116,120],{"className":117,"code":118,"language":119,"meta":68,"style":68},"language-python shiki shiki-themes github-light github-dark","# conftest.py — an on-demand dump for local reproduction\nimport faulthandler\nimport signal\n\n\ndef pytest_configure(config):\n    # kill -USR1 \u003Cpid> now prints every thread's stack without stopping anything.\n    if hasattr(signal, \"SIGUSR1\"):\n        faulthandler.register(signal.SIGUSR1, all_threads=True, chain=True)\n","python",[14,121,122,127,132,137,143,147,152,157,163],{"__ignoreMap":68},[72,123,124],{"class":74,"line":75},[72,125,126],{},"# conftest.py — an on-demand dump for local reproduction\n",[72,128,129],{"class":74,"line":81},[72,130,131],{},"import faulthandler\n",[72,133,134],{"class":74,"line":87},[72,135,136],{},"import signal\n",[72,138,139],{"class":74,"line":93},[72,140,142],{"emptyLinePlaceholder":141},true,"\n",[72,144,145],{"class":74,"line":99},[72,146,142],{"emptyLinePlaceholder":141},[72,148,149],{"class":74,"line":105},[72,150,151],{},"def pytest_configure(config):\n",[72,153,154],{"class":74,"line":111},[72,155,156],{},"    # kill -USR1 \u003Cpid> now prints every thread's stack without stopping anything.\n",[72,158,160],{"class":74,"line":159},8,[72,161,162],{},"    if hasattr(signal, \"SIGUSR1\"):\n",[72,164,166],{"class":74,"line":165},9,[72,167,168],{},"        faulthandler.register(signal.SIGUSR1, all_threads=True, chain=True)\n",[63,170,175],{"className":171,"code":173,"language":174,"meta":68},[172],"language-text","Timeout (0:00:45)!\nThread 0x00007f2ab3fff700 (most recent call first):\n  File \"app\u002Fcache.py\", line 41 in refresh\n  File \"app\u002Fcache.py\", line 88 in get\n  File \"threading.py\", line 975 in _bootstrap_inner\n\nCurrent thread 0x00007f2ac4a1e740 (most recent call first):\n  File \"app\u002Fcache.py\", line 62 in evict\n  File \"app\u002Fcache.py\", line 91 in put\n  File \"tests\u002Ftest_cache.py\", line 24 in test_concurrent_eviction\n","text",[14,176,173],{"__ignoreMap":68},[178,179,182,299],"figure",{"className":180},[181],"diagram",[183,184,191,192,191,196,191,200,191,218,191,226,191,234,191,241,191,246,191,254,191,259,191,266,191,271,191,275,191,279,191,283,191,286,191,293],"svg",{"viewBox":185,"role":186,"ariaLabelledBy":187,"xmlns":190},"0 0 820 260","img",[188,189],"fh-t","fh-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[193,194,195],"title",{"id":188},"Timing of the dump relative to the suite timeout",[197,198,199],"desc",{"id":189},"A timeline of one hung test. At forty-five seconds faulthandler writes every thread's stack while the process is alive. At sixty seconds pytest-timeout fails the test. If the order were reversed, or if faulthandler were unset, the process would be killed with no output.",[201,202,203,204,191],"defs",{},"\n    ",[205,206,213],"marker",{"id":207,"viewBox":208,"refX":209,"refY":210,"markerWidth":211,"markerHeight":211,"orient":212},"fh-a","0 0 10 10","9","5","7","auto-start-reverse",[214,215],"path",{"d":216,"fill":217},"M0 0 L10 5 L0 10 z","#3d405b",[219,220],"rect",{"x":221,"y":221,"width":222,"height":223,"rx":224,"fill":225},"0","820","260","14","#fffdf8",[174,227,233],{"x":228,"y":229,"textAnchor":230,"fontSize":231,"fontWeight":232,"fill":217},"410","28","middle","16","700","The dump has to happen while the process is still alive",[74,235],{"x1":236,"y1":237,"x2":238,"y2":237,"stroke":217,"strokeWidth":239,"markerEnd":240},"60","120","770","1.6","url(#fh-a)",[174,242,245],{"x":236,"y":243,"fontSize":244,"fill":217},"146","11","0 s",[219,247],{"x":237,"y":248,"width":223,"height":249,"rx":250,"fill":251,"stroke":252,"strokeWidth":253},"86","30","8","#e6f0ea","#81b29a","1.7",[174,255,258],{"x":256,"y":257,"textAnchor":230,"fontSize":244,"fill":217},"250","106","test runs, then blocks",[74,260],{"x1":261,"y1":262,"x2":261,"y2":263,"stroke":264,"strokeWidth":265},"470","70","170","#f2cc8f","3",[174,267,270],{"x":261,"y":268,"textAnchor":230,"fontSize":244,"fontWeight":232,"fill":269},"62","#8a5a00","45 s · faulthandler dumps",[174,272,274],{"x":261,"y":273,"textAnchor":230,"fontSize":244,"fill":217},"190","every thread's stack to stderr",[74,276],{"x1":277,"y1":262,"x2":277,"y2":263,"stroke":278,"strokeWidth":265},"650","#e07a5f",[174,280,282],{"x":277,"y":268,"textAnchor":230,"fontSize":244,"fontWeight":232,"fill":281},"#8f3d22","60 s · timeout fails the test",[174,284,285],{"x":277,"y":273,"textAnchor":230,"fontSize":244,"fill":217},"suite continues",[219,287],{"x":236,"y":288,"width":289,"height":290,"rx":209,"fill":291,"stroke":278,"strokeWidth":292},"212","710","34","#fbe9e3","1.8",[174,294,298],{"x":295,"y":296,"textAnchor":230,"fontSize":297,"fill":217},"415","234","11.5","Without the 45 s dump, all you get at 60 s is \"the test timed out\" and no stacks.",[300,301,302],"figcaption",{},"Fifteen seconds of headroom is generous; the dump itself takes milliseconds, but a loaded runner can delay the writing thread.",[19,304,306],{"id":305},"why-this-works","Why this works",[10,308,309,312],{},[14,310,311],{},"faulthandler.dump_traceback_later(timeout, exit=...)"," starts a watchdog thread that sleeps for the timeout and, if nothing cancelled it, writes the Python stack of every thread using only pre-allocated buffers and async-signal-safe calls. That constraint is why it works when nothing else does: it does not allocate, does not take the GIL, and therefore produces output even when every other thread is blocked.",[10,314,315,316,319,320,322],{},"pytest wires this to the ",[14,317,318],{},"faulthandler_timeout"," option, resetting the watchdog before each test so the countdown measures a single test rather than the whole run. ",[14,321,41],{}," is doing something different — failing the test — and the two are complementary rather than redundant.",[19,324,326],{"id":325},"edge-cases-and-failure-modes","Edge cases and failure modes",[24,328,329,338,348,361,367],{},[27,330,331,337],{},[332,333,334,336],"strong",{},[14,335,318],{}," set above the suite timeout."," The process is killed before the dump is written, which is the same as having no dump at all. It must be lower.",[27,339,340,343,344,347],{},[332,341,342],{},"A dump with only one thread."," The hang is not a deadlock between threads; look at what that thread is blocked on — a socket read, a ",[14,345,346],{},"queue.get"," with no timeout, a subprocess that never exits.",[27,349,350,353,354,356,357,360],{},[332,351,352],{},"Output swallowed by capture."," ",[14,355,16],{}," writes to the real file descriptor 2, so it survives pytest's capture, but a CI system that buffers job output may still delay it. ",[14,358,359],{},"-s"," guarantees it is interleaved in order.",[27,362,363,366],{},[332,364,365],{},"C-level frames missing."," The dump shows Python frames only. A thread blocked inside a C extension shows the Python call that entered it, which is usually enough to identify the library.",[27,368,369,353,375,378],{},[332,370,371,374],{},[14,372,373],{},"exit=True"," when called manually.",[14,376,377],{},"dump_traceback_later(30, exit=True)"," aborts the process after dumping, which is right for a standalone script and wrong inside pytest, where it kills the whole session.",[19,380,382],{"id":381},"which-timeout-mechanism-to-use-where","Which timeout mechanism to use where",[10,384,385],{},"Three mechanisms can end a hung test and they produce very different evidence, so the combination matters more than any one of them.",[10,387,388,390,391,394],{},[14,389,41],{}," with ",[14,392,393],{},"timeout_method = \"signal\""," raises inside the running test at the exact line, producing an ordinary pytest failure with a real traceback and letting the suite continue. It works only on the main thread of a Unix process, so a suite with worker threads or a Windows runner cannot rely on it.",[10,396,397,400],{},[14,398,399],{},"timeout_method = \"thread\""," runs a watchdog that dumps every thread's stack and then kills the process. It works everywhere and gives good evidence, but the session ends — remaining tests do not run.",[10,402,403,405,406,410],{},[14,404,318],{}," dumps without ending anything, which is why it belongs ",[407,408,409],"em",{},"below"," whichever of the two above is configured: the stacks are written first, and the failure mechanism then does its job.",[10,412,413,414,416],{},"For most suites the right combination is ",[14,415,318],{}," plus the thread method, because a suite that hangs is usually going to need a human anyway and the complete stacks are worth more than the remaining tests. Where the suite is long and a single hang should not cost the whole run, the signal method keeps the session alive at the cost of covering only the main thread — acceptable when the threads are library-owned and the main thread is where the test's own code blocks. Whichever pair is chosen, write the two numbers down next to each other in the configuration with a comment, because the relationship between them — dump first, fail second — is the part that breaks silently when somebody later raises one of them in isolation.",[178,418,420,528],{"className":419},[181],[183,421,191,426,191,429,191,432,191,436,191,441,191,449,191,454,191,458,191,461,191,464,191,468,191,472,191,476,191,481,191,485,191,488,191,491,191,495,191,497,191,499,191,502,191,505,191,507,191,509,191,511,191,517,191,520,191,524],{"viewBox":422,"role":186,"ariaLabelledBy":423,"xmlns":190},"0 0 800 244",[424,425],"mech-t","mech-d",[193,427,428],{"id":424},"Three timeout mechanisms and what each leaves behind",[197,430,431],{"id":425},"Three rows comparing the signal method, the thread method and faulthandler on three properties: whether the suite continues afterwards, whether all thread stacks are captured, and where each one works. The signal method continues the suite but only covers the main thread on Unix; the thread method captures everything but ends the session; faulthandler captures everything and ends nothing.",[219,433],{"x":221,"y":221,"width":434,"height":435,"rx":224,"fill":225},"800","244",[174,437,440],{"x":438,"y":229,"textAnchor":230,"fontSize":439,"fontWeight":232,"fill":217},"400","15.5","Layer them; do not choose between them",[219,442],{"x":443,"y":444,"width":445,"height":446,"rx":209,"fill":447,"stroke":217,"strokeWidth":448},"24","48","200","38","#f4f1de","1.5",[174,450,453],{"x":451,"y":452,"textAnchor":230,"fontSize":297,"fontWeight":232,"fill":217},"124","72","mechanism",[219,455],{"x":456,"y":444,"width":457,"height":446,"rx":209,"fill":447,"stroke":217,"strokeWidth":448},"232","172",[174,459,285],{"x":460,"y":452,"textAnchor":230,"fontSize":297,"fontWeight":232,"fill":217},"318",[219,462],{"x":463,"y":444,"width":457,"height":446,"rx":209,"fill":447,"stroke":217,"strokeWidth":448},"412",[174,465,467],{"x":466,"y":452,"textAnchor":230,"fontSize":297,"fontWeight":232,"fill":217},"498","all thread stacks",[219,469],{"x":470,"y":444,"width":471,"height":446,"rx":209,"fill":447,"stroke":217,"strokeWidth":448},"592","184",[174,473,475],{"x":474,"y":452,"textAnchor":230,"fontSize":297,"fontWeight":232,"fill":217},"684","where it works",[174,477,480],{"x":478,"y":479,"fontSize":297,"fill":217},"40","118","timeout_method=signal",[174,482,484],{"x":460,"y":479,"textAnchor":230,"fontSize":297,"fill":483},"#2a5f49","yes",[174,486,487],{"x":466,"y":479,"textAnchor":230,"fontSize":297,"fill":281},"no",[174,489,490],{"x":474,"y":479,"textAnchor":230,"fontSize":297,"fill":217},"Unix main thread",[174,492,494],{"x":478,"y":493,"fontSize":297,"fill":217},"156","timeout_method=thread",[174,496,487],{"x":460,"y":493,"textAnchor":230,"fontSize":297,"fill":281},[174,498,484],{"x":466,"y":493,"textAnchor":230,"fontSize":297,"fill":483},[174,500,501],{"x":474,"y":493,"textAnchor":230,"fontSize":297,"fill":217},"everywhere",[174,503,318],{"x":478,"y":504,"fontSize":297,"fill":217},"194",[174,506,484],{"x":460,"y":504,"textAnchor":230,"fontSize":297,"fill":483},[174,508,484],{"x":466,"y":504,"textAnchor":230,"fontSize":297,"fill":483},[174,510,501],{"x":474,"y":504,"textAnchor":230,"fontSize":297,"fill":217},[74,512],{"x1":443,"y1":513,"x2":514,"y2":513,"stroke":515,"strokeWidth":516},"130","776","rgba(61,64,91,0.14)","1.4",[74,518],{"x1":443,"y1":519,"x2":514,"y2":519,"stroke":515,"strokeWidth":516},"168",[219,521],{"x":443,"y":522,"width":523,"height":229,"rx":250,"fill":251,"stroke":252,"strokeWidth":239},"210","752",[174,525,527],{"x":438,"y":526,"textAnchor":230,"fontSize":244,"fill":217},"229","faulthandler first for the evidence, then signal or thread to end the test.",[300,529,530],{},"The bottom row is the only one with no drawback, which is why it should always be set — it produces evidence without deciding anything.",[19,532,534],{"id":533},"reading-the-dump","Reading the dump",[10,536,537],{},"The dump is a list of threads, each with its frames innermost-first. Three questions answer almost every case.",[10,539,540,543,544,547,548,551,552,555,556,559],{},[332,541,542],{},"What is each thread blocked on?"," The innermost frame names it: ",[14,545,546],{},"acquire"," on a lock, ",[14,549,550],{},"wait"," on a condition, ",[14,553,554],{},"recv"," on a socket, ",[14,557,558],{},"join"," on another thread. A thread whose innermost frame is ordinary application code is not blocked — it is spinning, which is a different bug.",[10,561,562,565,566,569],{},[332,563,564],{},"What does each blocked thread already hold?"," Walk outward until you find the ",[14,567,568],{},"with self._lock:"," that thread is inside. The dump does not say so directly, but the frame at that line tells you which lock the thread acquired on its way in.",[10,571,572,575],{},[332,573,574],{},"Is there a cycle?"," Thread A holds lock 1 and waits for lock 2; thread B holds lock 2 and waits for lock 1. That is the deadlock, and the fix is a global acquisition order rather than more locks.",[63,577,580],{"className":578,"code":579,"language":174,"meta":68},[172],"Thread 0x…700 (most recent call first):\n  File \"threading.py\", line 327 in wait          ← blocked here\n  File \"app\u002Fcache.py\", line 41 in refresh        ← inside `with self._write_lock`\n  File \"app\u002Fcache.py\", line 88 in get\n\nCurrent thread 0x…740 (most recent call first):\n  File \"threading.py\", line 327 in wait          ← blocked here\n  File \"app\u002Fcache.py\", line 62 in evict          ← inside `with self._index_lock`\n  File \"app\u002Fcache.py\", line 91 in put\n",[14,581,579],{"__ignoreMap":68},[10,583,584,585,588,589,592],{},"Two threads, two locks, opposite order: a complete diagnosis from a dump that cost one configuration line. The repair is to establish that ",[14,586,587],{},"_index_lock"," is always taken before ",[14,590,591],{},"_write_lock",", everywhere, which is a rule a reviewer can check by reading rather than a race anyone has to reproduce.",[19,594,596],{"id":595},"dumping-a-process-you-did-not-start","Dumping a process you did not start",[10,598,599],{},"The signal registration above pays off when the hang happens somewhere you cannot easily add configuration: a long-running local reproduction, a worker inside a container, a process a colleague is looking at.",[63,601,605],{"className":602,"code":603,"language":604,"meta":68,"style":68},"language-bash shiki shiki-themes github-light github-dark","# Find the stuck process and ask it for a dump. Nothing is killed.\npgrep -af \"pytest tests\u002Ftest_cache.py\"\nkill -USR1 12345                       # stacks appear on the process's stderr\n\n# Inside a container, the same signal via the runtime:\ndocker kill --signal=SIGUSR1 my-test-container\n","bash",[14,606,607,613,627,641,645,650],{"__ignoreMap":68},[72,608,609],{"class":74,"line":75},[72,610,612],{"class":611},"sJ8bj","# Find the stuck process and ask it for a dump. Nothing is killed.\n",[72,614,615,619,623],{"class":74,"line":81},[72,616,618],{"class":617},"sScJk","pgrep",[72,620,622],{"class":621},"sj4cs"," -af",[72,624,626],{"class":625},"sZZnC"," \"pytest tests\u002Ftest_cache.py\"\n",[72,628,629,632,635,638],{"class":74,"line":87},[72,630,631],{"class":621},"kill",[72,633,634],{"class":621}," -USR1",[72,636,637],{"class":621}," 12345",[72,639,640],{"class":611},"                       # stacks appear on the process's stderr\n",[72,642,643],{"class":74,"line":93},[72,644,142],{"emptyLinePlaceholder":141},[72,646,647],{"class":74,"line":99},[72,648,649],{"class":611},"# Inside a container, the same signal via the runtime:\n",[72,651,652,655,658,661],{"class":74,"line":105},[72,653,654],{"class":617},"docker",[72,656,657],{"class":625}," kill",[72,659,660],{"class":621}," --signal=SIGUSR1",[72,662,663],{"class":625}," my-test-container\n",[10,665,666,667,670,671,674],{},"Because ",[14,668,669],{},"faulthandler.register"," is armed at ",[14,672,673],{},"pytest_configure",", every test session in the repository responds to the signal, including ones started by an IDE. That makes \"send USR1 and read the stacks\" the first move against any hang, ahead of attaching a debugger — it is faster, it is non-invasive, and it works on a process whose environment cannot be changed.",[10,676,677,678,681,682,685,686,689,690,51],{},"Two arguments to ",[14,679,680],{},"register"," matter. ",[14,683,684],{},"all_threads=True"," is what makes it useful; the default dumps only the current thread, which for a deadlock is the least interesting one. And ",[14,687,688],{},"chain=True"," preserves any handler already installed for that signal, so registering does not silently break something else that used ",[14,691,692],{},"SIGUSR1",[178,694,696,752],{"className":695},[181],[183,697,191,702,191,705,191,708,191,711,191,714,191,721,191,726,191,730,191,734,191,738,191,742,191,745,191,748],{"viewBox":698,"role":186,"ariaLabelledBy":699,"xmlns":190},"0 0 800 224",[700,701],"sig-t","sig-d",[193,703,704],{"id":700},"Three ways to obtain a stack dump, by situation",[197,706,707],{"id":701},"Three rows. A timeout-driven dump is automatic and suits CI. A signal-driven dump is on demand and suits local or containerised reproduction. A debugger attach gives interactive inspection but requires changing how the process runs and is the slowest to set up.",[219,709],{"x":221,"y":221,"width":434,"height":710,"rx":224,"fill":225},"224",[174,712,713],{"x":438,"y":229,"textAnchor":230,"fontSize":439,"fontWeight":232,"fill":217},"Reach for these in this order",[219,715],{"x":716,"y":717,"width":718,"height":717,"rx":719,"fill":251,"stroke":252,"strokeWidth":720},"26","50","748","10","2",[174,722,725],{"x":723,"y":452,"fontSize":724,"fontWeight":232,"fill":217},"46","12","1 · faulthandler_timeout",[174,727,729],{"x":723,"y":728,"fontSize":244,"fill":483},"90","automatic, no interaction, right for CI · gives stacks at a fixed delay",[219,731],{"x":716,"y":732,"width":718,"height":717,"rx":719,"fill":733,"stroke":264,"strokeWidth":720},"110","#f7f0da",[174,735,737],{"x":723,"y":736,"fontSize":724,"fontWeight":232,"fill":217},"132","2 · kill -USR1",[174,739,741],{"x":723,"y":740,"fontSize":244,"fill":269},"150","on demand, non-invasive, works in containers · nothing is stopped",[219,743],{"x":716,"y":263,"width":718,"height":744,"rx":719,"fill":291,"stroke":278,"strokeWidth":720},"44",[174,746,747],{"x":723,"y":273,"fontSize":724,"fontWeight":232,"fill":217},"3 · attach a debugger",[174,749,751],{"x":723,"y":750,"fontSize":244,"fill":281},"207","interactive but slowest to set up · needs the process started differently",[300,753,754],{},"The first two answer \"what is everything waiting on\", which is the whole question for a deadlock. The third is for when the answer to that raises a further question about values.",[19,756,758],{"id":757},"frequently-asked-questions","Frequently Asked Questions",[10,760,761,764,765,767],{},[332,762,763],{},"Why does my deadlocked test produce no output at all?","\nBecause nothing asked for one. pytest's own timeout kills the process, and a CI platform's job timeout kills it harder. ",[14,766,318],{}," set below the suite timeout writes every thread's stack while the process is still alive, which is the only window in which the dump can be produced.",[10,769,770,773,774,776,777,779,780,782,783,785],{},[332,771,772],{},"Does faulthandler work with pytest-timeout?","\nThey complement each other. Set ",[14,775,318],{}," lower than ",[14,778,41],{},"'s value so the stacks are written first and the timeout then fails the test cleanly. ",[14,781,41],{},"'s thread method also dumps stacks, so on Unix the signal method plus ",[14,784,16],{}," often gives the most readable combination.",[10,787,788,791,792,795],{},[332,789,790],{},"Can I dump stacks without waiting for a timeout?","\nYes. ",[14,793,794],{},"faulthandler.register(signal.SIGUSR1)"," lets you send a signal to a stuck process and get a dump immediately, which is invaluable when reproducing a hang locally. It also works against a process running under a debugger or inside a container.",[19,797,799],{"id":798},"related","Related",[24,801,802,809,816,823],{},[27,803,804,808],{},[47,805,807],{"href":806},"\u002Ftesting-async-and-concurrent-python\u002Ftesting-threads-and-race-conditions\u002F","Testing Threads & Race Conditions"," — the lock-ordering discipline that prevents the cycle in the first place.",[27,810,811,815],{},[47,812,814],{"href":813},"\u002Ftesting-async-and-concurrent-python\u002Ftimeouts-cancellation-and-deadlines\u002Ffailing-fast-with-pytest-timeout\u002F","Failing Fast with pytest-timeout"," — the layer that turns the hang into a reported failure.",[27,817,818,822],{},[47,819,821],{"href":820},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-async-code-and-event-loops\u002Ftracking-down-a-hung-await-with-task-stacks\u002F","Tracking Down a Hung await with Task Stacks"," — the asyncio equivalent, where the stacks live on tasks rather than threads.",[27,824,825,829],{},[47,826,828],{"href":827},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fcapturing-artifacts-from-a-failed-ci-test-run\u002F","Capturing Artifacts from a Failed CI Test Run"," — making sure the dump survives the job.",[10,831,832,833],{},"← Back to ",[47,834,807],{"href":806},[836,837,838],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}",{"title":68,"searchDepth":81,"depth":81,"links":840},[841,842,843,844,845,846,847,848,849],{"id":21,"depth":81,"text":22},{"id":54,"depth":81,"text":55},{"id":305,"depth":81,"text":306},{"id":325,"depth":81,"text":326},{"id":381,"depth":81,"text":382},{"id":533,"depth":81,"text":534},{"id":595,"depth":81,"text":596},{"id":757,"depth":81,"text":758},{"id":798,"depth":81,"text":799},"Turn a hung test into a diagnosis: faulthandler_timeout, dump_traceback_later, pytest-timeout's thread method, and reading which lock each thread is waiting on.","md",{"slug":853,"type":854,"breadcrumb":855,"datePublished":856,"dateModified":856,"faq":857,"howto":864},"dumping-stacks-on-deadlock-with-faulthandler","article","Deadlock Dumps","2026-09-18",[858,860,862],{"q":763,"a":859},"Because nothing asked for one. pytest's own timeout kills the process, and a CI platform's job timeout kills it harder. faulthandler_timeout set below the suite timeout writes every thread's stack while the process is still alive, which is the only window in which the dump can be produced.",{"q":772,"a":861},"They complement each other. Set faulthandler_timeout lower than pytest-timeout's value so the stacks are written first and the timeout then fails the test cleanly. pytest-timeout's thread method also dumps stacks, so on Unix the signal method plus faulthandler often gives the most readable combination.",{"q":790,"a":863},"Yes. faulthandler.register(signal.SIGUSR1) lets you send a signal to a stuck process and get a dump immediately, which is invaluable when reproducing a hang locally. It also works against a process running under a debugger or inside a container.",{"name":865,"description":866,"steps":867},"How to get a stack dump from a deadlocked test","Arm faulthandler below the suite timeout, read which lock each thread waits on, and fix the acquisition order.",[868,871,874,877,880],{"name":869,"text":870},"Set faulthandler_timeout below the suite timeout","Configure faulthandler_timeout so the dump is written while the process is still alive.",{"name":872,"text":873},"Choose pytest-timeout's method to match the platform","Use signal on Unix main threads for a clean failure, thread elsewhere for a dump plus a hard stop.",{"name":875,"text":876},"Read the frames for each thread","Identify what each thread is blocked on and which lock it already holds.",{"name":878,"text":879},"Look for a cycle","Two threads holding one lock each and waiting for the other's is the deadlock; the fix is a global acquisition order.",{"name":881,"text":882},"Add a signal handler for live reproduction","Register SIGUSR1 so a stuck process can be made to dump on demand rather than only on timeout.","\u002Ftesting-async-and-concurrent-python\u002Ftesting-threads-and-race-conditions\u002Fdumping-stacks-on-deadlock-with-faulthandler",{"title":5,"description":850},"testing-async-and-concurrent-python\u002Ftesting-threads-and-race-conditions\u002Fdumping-stacks-on-deadlock-with-faulthandler\u002Findex","E8xahnpBpNHFq0TY9itMepePAF37PukEbNakyv2MVjg",1789718768417]