[{"data":1,"prerenderedAt":1917},["ShallowReactive",2],{"page-\u002Fsystematic-debugging-performance-profiling\u002Flogging-and-observability-for-debugging\u002F":3},{"id":4,"title":5,"body":6,"description":1876,"extension":1877,"meta":1878,"navigation":244,"path":1913,"seo":1914,"stem":1915,"__hash__":1916},"content\u002Fsystematic-debugging-performance-profiling\u002Flogging-and-observability-for-debugging\u002Findex.md","Logging & Observability for Debugging",{"type":7,"value":8,"toc":1855},"minimark",[9,18,23,68,72,82,213,217,222,316,343,347,408,451,464,468,533,544,548,580,617,622,626,688,692,695,726,733,740,744,852,856,859,866,872,878,884,899,928,932,935,954,1081,1114,1117,1121,1129,1132,1254,1271,1275,1278,1408,1415,1527,1531,1534,1544,1552,1611,1625,1629,1632,1643,1696,1706,1726,1732,1735,1741,1745,1768,1774,1791,1797,1803,1807,1844,1851],[10,11,12,13,17],"p",{},"When a test fails in CI and the only evidence is ",[14,15,16],"code",{},"assert 3 == 4",", the next step is always the same: add prints, push, wait, read, repeat. Logs that were already structured, already correlated and already attached to the failure report remove that loop entirely. The work is small and almost all of it is configuration, but it has to be done before the failure rather than after it.",[19,20,22],"h2",{"id":21},"prerequisites","Prerequisites",[24,25,26,34,48,62],"ul",{},[27,28,29,30,33],"li",{},"Python 3.9+; ",[14,31,32],{},"contextvars"," for correlation ids, which is standard library.",[27,35,36,39,40,43,44,47],{},[14,37,38],{},"pytest >= 8.0"," for ",[14,41,42],{},"caplog"," and the ",[14,45,46],{},"log_cli"," options used here.",[27,49,50,53,54,57,58,61],{},[14,51,52],{},"structlog"," or ",[14,55,56],{},"python-json-logger"," if structured output is wanted in production; the ",[14,59,60],{},"extra"," argument covers most of it without a dependency.",[27,63,64,67],{},[14,65,66],{},"opentelemetry-sdk"," only for the tracing section.",[19,69,71],{"id":70},"core-concept-a-record-is-data-not-a-sentence","Core concept: a record is data, not a sentence",[10,73,74,77,78,81],{},[14,75,76],{},"logger.info(\"charged customer %s for %s\", customer_id, amount)"," produces a string. ",[14,79,80],{},"logger.info(\"charge_succeeded\", extra={\"customer_id\": customer_id, \"amount_minor\": amount})"," produces a record with fields, and everything useful follows from the difference: filtering by customer, aggregating by outcome, asserting on a value in a test without matching prose.",[83,84,87,206],"figure",{"className":85},[86],"diagram",[88,89,96,97,96,101,96,105,96,113,96,123,96,133,96,139,96,148,96,154,96,160,96,164,96,168,96,172,96,177,96,181,96,184,96,189,96,193,96,197,96,200,96,203],"svg",{"viewBox":90,"role":91,"ariaLabelledBy":92,"xmlns":95},"0 0 840 276","img",[93,94],"log-t","log-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[98,99,100],"title",{"id":93},"A formatted line versus a structured record",[102,103,104],"desc",{"id":94},"Two paths from the same call. The formatted line produces a single string that must be parsed with a regular expression to extract values, and a rewording breaks both the parser and any test asserting on it. The structured record carries named fields that can be filtered, aggregated and asserted on directly, and rewording the human-readable part changes nothing.",[106,107],"rect",{"x":108,"y":108,"width":109,"height":110,"rx":111,"fill":112},"0","840","276","14","#fffdf8",[114,115,122],"text",{"x":116,"y":117,"textAnchor":118,"fontSize":119,"fontWeight":120,"fill":121},"420","28","middle","16","700","#3d405b","The same event, two levels of usefulness",[106,124],{"x":125,"y":126,"width":127,"height":128,"rx":129,"fill":130,"stroke":131,"strokeWidth":132},"26","52","368","196","12","#fbe9e3","#e07a5f","2",[114,134,138],{"x":135,"y":136,"textAnchor":118,"fontSize":137,"fontWeight":120,"fill":121},"210","78","12.5","formatted string",[106,140],{"x":141,"y":142,"width":143,"height":144,"rx":145,"fill":112,"stroke":146,"strokeWidth":147},"44","92","332","40","8","rgba(61,64,91,0.35)","1.4",[114,149,153],{"x":150,"y":151,"fontSize":152,"fill":121},"58","117","10.5","\"charged customer cus_12 for 1234\"",[114,155,159],{"x":141,"y":156,"fontSize":157,"fill":158},"156","11","#8f3d22","✗ values only recoverable by regex",[114,161,163],{"x":141,"y":162,"fontSize":157,"fill":158},"180","✗ rewording breaks parsers and tests",[114,165,167],{"x":141,"y":166,"fontSize":157,"fill":158},"204","✗ cannot aggregate by outcome",[114,169,171],{"x":141,"y":170,"fontSize":157,"fill":121},"228","grep is the only query language",[106,173],{"x":174,"y":126,"width":127,"height":128,"rx":129,"fill":175,"stroke":176,"strokeWidth":132},"446","#e6f0ea","#81b29a",[114,178,180],{"x":179,"y":136,"textAnchor":118,"fontSize":137,"fontWeight":120,"fill":121},"630","structured record",[106,182],{"x":183,"y":142,"width":143,"height":144,"rx":145,"fill":112,"stroke":146,"strokeWidth":147},"464",[114,185,188],{"x":186,"y":187,"fontSize":152,"fill":121},"478","112","event=charge_succeeded",[114,190,192],{"x":186,"y":191,"fontSize":152,"fill":121},"127","customer_id=cus_12 amount_minor=1234",[114,194,196],{"x":183,"y":156,"fontSize":157,"fill":195},"#2a5f49","✓ filter by any field",[114,198,199],{"x":183,"y":162,"fontSize":157,"fill":195},"✓ assert on values, not wording",[114,201,202],{"x":183,"y":166,"fontSize":157,"fill":195},"✓ aggregate by event name",[114,204,205],{"x":183,"y":170,"fontSize":157,"fill":121},"the message stays human-readable too",[207,208,209,210,212],"figcaption",{},"Structured output costs nothing at the call site — ",[14,211,60],{}," is standard library — and changes what is possible at every later stage.",[19,214,216],{"id":215},"step-by-step-implementation","Step-by-step implementation",[218,219,221],"h3",{"id":220},"_1-emit-fields-alongside-the-message","1. Emit fields alongside the message",[223,224,229],"pre",{"className":225,"code":226,"language":227,"meta":228,"style":228},"language-python shiki shiki-themes github-light github-dark","import logging\n\nlogger = logging.getLogger(__name__)\n\n\ndef charge(customer_id: str, amount_minor: int) -> None:\n    # `extra` keys become attributes on the LogRecord, which a JSON formatter\n    # serialises and caplog exposes for assertions.\n    logger.info(\n        \"charge succeeded\",\n        extra={\"event\": \"charge_succeeded\",\n               \"customer_id\": customer_id,\n               \"amount_minor\": amount_minor},\n    )\n","python","",[14,230,231,239,246,252,257,262,268,274,280,286,292,298,304,310],{"__ignoreMap":228},[232,233,236],"span",{"class":234,"line":235},"line",1,[232,237,238],{},"import logging\n",[232,240,242],{"class":234,"line":241},2,[232,243,245],{"emptyLinePlaceholder":244},true,"\n",[232,247,249],{"class":234,"line":248},3,[232,250,251],{},"logger = logging.getLogger(__name__)\n",[232,253,255],{"class":234,"line":254},4,[232,256,245],{"emptyLinePlaceholder":244},[232,258,260],{"class":234,"line":259},5,[232,261,245],{"emptyLinePlaceholder":244},[232,263,265],{"class":234,"line":264},6,[232,266,267],{},"def charge(customer_id: str, amount_minor: int) -> None:\n",[232,269,271],{"class":234,"line":270},7,[232,272,273],{},"    # `extra` keys become attributes on the LogRecord, which a JSON formatter\n",[232,275,277],{"class":234,"line":276},8,[232,278,279],{},"    # serialises and caplog exposes for assertions.\n",[232,281,283],{"class":234,"line":282},9,[232,284,285],{},"    logger.info(\n",[232,287,289],{"class":234,"line":288},10,[232,290,291],{},"        \"charge succeeded\",\n",[232,293,295],{"class":234,"line":294},11,[232,296,297],{},"        extra={\"event\": \"charge_succeeded\",\n",[232,299,301],{"class":234,"line":300},12,[232,302,303],{},"               \"customer_id\": customer_id,\n",[232,305,307],{"class":234,"line":306},13,[232,308,309],{},"               \"amount_minor\": amount_minor},\n",[232,311,313],{"class":234,"line":312},14,[232,314,315],{},"    )\n",[10,317,318,319,321,322,325,326,329,330,329,333,329,336,339,340,342],{},"One caution: ",[14,320,60],{}," keys that collide with ",[14,323,324],{},"LogRecord","'s own attributes (",[14,327,328],{},"message",", ",[14,331,332],{},"args",[14,334,335],{},"name",[14,337,338],{},"levelname",") raise at call time. Prefixing domain fields, or using ",[14,341,52],{},", avoids the collision entirely.",[218,344,346],{"id":345},"_2-correlate-everything-in-one-operation","2. Correlate everything in one operation",[223,348,350],{"className":225,"code":349,"language":227,"meta":228,"style":228},"import contextvars\nimport logging\nimport uuid\n\nrequest_id: contextvars.ContextVar[str] = contextvars.ContextVar(\"request_id\", default=\"-\")\n\n\nclass CorrelationFilter(logging.Filter):\n    def filter(self, record: logging.LogRecord) -> bool:\n        # Every record gets the current id without any call site passing it.\n        record.request_id = request_id.get()\n        return True\n",[14,351,352,357,361,366,370,375,379,383,388,393,398,403],{"__ignoreMap":228},[232,353,354],{"class":234,"line":235},[232,355,356],{},"import contextvars\n",[232,358,359],{"class":234,"line":241},[232,360,238],{},[232,362,363],{"class":234,"line":248},[232,364,365],{},"import uuid\n",[232,367,368],{"class":234,"line":254},[232,369,245],{"emptyLinePlaceholder":244},[232,371,372],{"class":234,"line":259},[232,373,374],{},"request_id: contextvars.ContextVar[str] = contextvars.ContextVar(\"request_id\", default=\"-\")\n",[232,376,377],{"class":234,"line":264},[232,378,245],{"emptyLinePlaceholder":244},[232,380,381],{"class":234,"line":270},[232,382,245],{"emptyLinePlaceholder":244},[232,384,385],{"class":234,"line":276},[232,386,387],{},"class CorrelationFilter(logging.Filter):\n",[232,389,390],{"class":234,"line":282},[232,391,392],{},"    def filter(self, record: logging.LogRecord) -> bool:\n",[232,394,395],{"class":234,"line":288},[232,396,397],{},"        # Every record gets the current id without any call site passing it.\n",[232,399,400],{"class":234,"line":294},[232,401,402],{},"        record.request_id = request_id.get()\n",[232,404,405],{"class":234,"line":300},[232,406,407],{},"        return True\n",[223,409,411],{"className":225,"code":410,"language":227,"meta":228,"style":228},"import pytest\n\n\n@pytest.fixture(autouse=True)\ndef correlate(request):\n    token = request_id.set(f\"test-{uuid.uuid4().hex[:8]}\")\n    yield\n    request_id.reset(token)\n",[14,412,413,418,422,426,431,436,441,446],{"__ignoreMap":228},[232,414,415],{"class":234,"line":235},[232,416,417],{},"import pytest\n",[232,419,420],{"class":234,"line":241},[232,421,245],{"emptyLinePlaceholder":244},[232,423,424],{"class":234,"line":248},[232,425,245],{"emptyLinePlaceholder":244},[232,427,428],{"class":234,"line":254},[232,429,430],{},"@pytest.fixture(autouse=True)\n",[232,432,433],{"class":234,"line":259},[232,434,435],{},"def correlate(request):\n",[232,437,438],{"class":234,"line":264},[232,439,440],{},"    token = request_id.set(f\"test-{uuid.uuid4().hex[:8]}\")\n",[232,442,443],{"class":234,"line":270},[232,444,445],{},"    yield\n",[232,447,448],{"class":234,"line":276},[232,449,450],{},"    request_id.reset(token)\n",[10,452,453,455,456,459,460,463],{},[14,454,32],{}," rather than a thread local is the right primitive: it follows ",[14,457,458],{},"asyncio"," tasks correctly, so a correlation id set before an ",[14,461,462],{},"await"," is still present after it and inside any task created from that context.",[218,465,467],{"id":466},"_3-capture-precisely-in-tests","3. Capture precisely in tests",[223,469,471],{"className":225,"code":470,"language":227,"meta":228,"style":228},"import logging\n\n\ndef test_charge_logs_the_amount(caplog, gateway):\n    # Set the level on the specific logger, not the root: raising the root\n    # level floods the capture with every library's debug output.\n    caplog.set_level(logging.INFO, logger=\"myapp.billing\")\n\n    gateway.charge(\"cus_12\", 1234)\n\n    record = next(r for r in caplog.records if getattr(r, \"event\", None) == \"charge_succeeded\")\n    assert record.customer_id == \"cus_12\"\n    assert record.amount_minor == 1234\n",[14,472,473,477,481,485,490,495,500,505,509,514,518,523,528],{"__ignoreMap":228},[232,474,475],{"class":234,"line":235},[232,476,238],{},[232,478,479],{"class":234,"line":241},[232,480,245],{"emptyLinePlaceholder":244},[232,482,483],{"class":234,"line":248},[232,484,245],{"emptyLinePlaceholder":244},[232,486,487],{"class":234,"line":254},[232,488,489],{},"def test_charge_logs_the_amount(caplog, gateway):\n",[232,491,492],{"class":234,"line":259},[232,493,494],{},"    # Set the level on the specific logger, not the root: raising the root\n",[232,496,497],{"class":234,"line":264},[232,498,499],{},"    # level floods the capture with every library's debug output.\n",[232,501,502],{"class":234,"line":270},[232,503,504],{},"    caplog.set_level(logging.INFO, logger=\"myapp.billing\")\n",[232,506,507],{"class":234,"line":276},[232,508,245],{"emptyLinePlaceholder":244},[232,510,511],{"class":234,"line":282},[232,512,513],{},"    gateway.charge(\"cus_12\", 1234)\n",[232,515,516],{"class":234,"line":288},[232,517,245],{"emptyLinePlaceholder":244},[232,519,520],{"class":234,"line":294},[232,521,522],{},"    record = next(r for r in caplog.records if getattr(r, \"event\", None) == \"charge_succeeded\")\n",[232,524,525],{"class":234,"line":300},[232,526,527],{},"    assert record.customer_id == \"cus_12\"\n",[232,529,530],{"class":234,"line":306},[232,531,532],{},"    assert record.amount_minor == 1234\n",[10,534,535,536,539,540,543],{},"Asserting on ",[14,537,538],{},"record.amount_minor"," rather than on ",[14,541,542],{},"\"1234\" in caplog.text"," is what makes the test survive a rewording. The message is for humans; the fields are the contract.",[218,545,547],{"id":546},"_4-watch-a-slow-or-hanging-test-live","4. Watch a slow or hanging test live",[223,549,553],{"className":550,"code":551,"language":552,"meta":228,"style":228},"language-toml shiki shiki-themes github-light github-dark","# pyproject.toml\n[tool.pytest.ini_options]\nlog_cli = false                      # off by default; noisy for a whole suite\nlog_cli_level = \"INFO\"\nlog_cli_format = \"%(asctime)s %(levelname)-5s %(name)s [%(request_id)s] %(message)s\"\n","toml",[14,554,555,560,565,570,575],{"__ignoreMap":228},[232,556,557],{"class":234,"line":235},[232,558,559],{},"# pyproject.toml\n",[232,561,562],{"class":234,"line":241},[232,563,564],{},"[tool.pytest.ini_options]\n",[232,566,567],{"class":234,"line":248},[232,568,569],{},"log_cli = false                      # off by default; noisy for a whole suite\n",[232,571,572],{"class":234,"line":254},[232,573,574],{},"log_cli_level = \"INFO\"\n",[232,576,577],{"class":234,"line":259},[232,578,579],{},"log_cli_format = \"%(asctime)s %(levelname)-5s %(name)s [%(request_id)s] %(message)s\"\n",[223,581,585],{"className":582,"code":583,"language":584,"meta":228,"style":228},"language-bash shiki shiki-themes github-light github-dark","pytest tests\u002Ftest_slow.py -o log_cli=true -k stuck    # turn it on for one run\n","bash",[14,586,587],{"__ignoreMap":228},[232,588,589,593,597,601,604,607,610,613],{"class":234,"line":235},[232,590,592],{"class":591},"sScJk","pytest",[232,594,596],{"class":595},"sZZnC"," tests\u002Ftest_slow.py",[232,598,600],{"class":599},"sj4cs"," -o",[232,602,603],{"class":595}," log_cli=",[232,605,606],{"class":599},"true",[232,608,609],{"class":599}," -k",[232,611,612],{"class":595}," stuck",[232,614,616],{"class":615},"sJ8bj","    # turn it on for one run\n",[10,618,619,621],{},[14,620,46],{}," streams records as they happen rather than buffering them for the failure report, which is the only way to see where a hanging test stopped. It is deliberately off by default because a whole suite with it enabled is unreadable.",[218,623,625],{"id":624},"_5-attach-captured-logs-to-the-failure-report","5. Attach captured logs to the failure report",[223,627,629],{"className":225,"code":628,"language":227,"meta":228,"style":228},"# conftest.py\nimport pytest\n\n\n@pytest.hookimpl(hookwrapper=True)\ndef pytest_runtest_makereport(item, call):\n    report = (yield).get_result()\n    if report.when == \"call\" and report.failed:\n        # pytest already captures logs; this makes sure they reach CI's XML too.\n        for name, content in report.sections:\n            if name.startswith(\"Captured log\"):\n                report.user_properties.append((\"captured_log\", content[:4000]))\n",[14,630,631,636,640,644,648,653,658,663,668,673,678,683],{"__ignoreMap":228},[232,632,633],{"class":234,"line":235},[232,634,635],{},"# conftest.py\n",[232,637,638],{"class":234,"line":241},[232,639,417],{},[232,641,642],{"class":234,"line":248},[232,643,245],{"emptyLinePlaceholder":244},[232,645,646],{"class":234,"line":254},[232,647,245],{"emptyLinePlaceholder":244},[232,649,650],{"class":234,"line":259},[232,651,652],{},"@pytest.hookimpl(hookwrapper=True)\n",[232,654,655],{"class":234,"line":264},[232,656,657],{},"def pytest_runtest_makereport(item, call):\n",[232,659,660],{"class":234,"line":270},[232,661,662],{},"    report = (yield).get_result()\n",[232,664,665],{"class":234,"line":276},[232,666,667],{},"    if report.when == \"call\" and report.failed:\n",[232,669,670],{"class":234,"line":282},[232,671,672],{},"        # pytest already captures logs; this makes sure they reach CI's XML too.\n",[232,674,675],{"class":234,"line":288},[232,676,677],{},"        for name, content in report.sections:\n",[232,679,680],{"class":234,"line":294},[232,681,682],{},"            if name.startswith(\"Captured log\"):\n",[232,684,685],{"class":234,"line":300},[232,686,687],{},"                report.user_properties.append((\"captured_log\", content[:4000]))\n",[19,689,691],{"id":690},"verification","Verification",[10,693,694],{},"Confirm that a failing test's output actually contains what you expect, once:",[223,696,698],{"className":582,"code":697,"language":584,"meta":228,"style":228},"pytest tests\u002Ftest_charge.py -q 2>&1 | sed -n '\u002FCaptured log\u002F,\u002F^=\u002Fp'\n",[14,699,700],{"__ignoreMap":228},[232,701,702,704,707,710,714,717,720,723],{"class":234,"line":235},[232,703,592],{"class":591},[232,705,706],{"class":595}," tests\u002Ftest_charge.py",[232,708,709],{"class":599}," -q",[232,711,713],{"class":712},"szBVR"," 2>&1",[232,715,716],{"class":712}," |",[232,718,719],{"class":591}," sed",[232,721,722],{"class":599}," -n",[232,724,725],{"class":595}," '\u002FCaptured log\u002F,\u002F^=\u002Fp'\n",[223,727,731],{"className":728,"code":730,"language":114,"meta":228},[729],"language-text","------------------------------ Captured log call -------------------------------\nINFO  myapp.billing [test-9f31ac2b] charge succeeded\nWARNING myapp.gateway [test-9f31ac2b] retrying after 502\n",[14,732,730],{"__ignoreMap":228},[10,734,735,736,739],{},"Two records, both carrying the same correlation id, both from the loggers you expect. If the list is empty, either the level is too high or something set ",[14,737,738],{},"propagate = False","; if it contains a thousand lines from unrelated libraries, the level was raised on the root logger rather than on a specific one.",[19,741,743],{"id":742},"troubleshooting","Troubleshooting",[745,746,747,763],"table",{},[748,749,750],"thead",{},[751,752,753,757,760],"tr",{},[754,755,756],"th",{},"Symptom",[754,758,759],{},"Root cause",[754,761,762],{},"Fix",[764,765,766,783,796,810,824,838],"tbody",{},[751,767,768,775,780],{},[769,770,771,774],"td",{},[14,772,773],{},"caplog.records"," is empty",[769,776,777,778],{},"Logger has ",[14,779,738],{},[769,781,782],{},"Set it back in a fixture, or capture that logger's handler",[751,784,785,788,791],{},[769,786,787],{},"Log output floods the capture",[769,789,790],{},"Level raised on the root logger",[769,792,793],{},[14,794,795],{},"caplog.set_level(..., logger=\"myapp.x\")",[751,797,798,804,807],{},[769,799,800,803],{},[14,801,802],{},"KeyError"," from the formatter",[769,805,806],{},"A custom field missing on some records",[769,808,809],{},"Give the filter a default for every custom key",[751,811,812,815,821],{},[769,813,814],{},"Records appear twice",[769,816,817,820],{},[14,818,819],{},"basicConfig"," called at import plus pytest's handler",[769,822,823],{},"Configure logging in an entry point, not at import",[751,825,826,829,832],{},[769,827,828],{},"Correlation id always the default",[769,830,831],{},"Set in a different context or thread",[769,833,834,835,837],{},"Use ",[14,836,32],{},"; set it inside the task that logs",[751,839,840,843,846],{},[769,841,842],{},"Nothing shown for a hanging test",[769,844,845],{},"Records buffered, never flushed",[769,847,848,851],{},[14,849,850],{},"-o log_cli=true"," for that run",[19,853,855],{"id":854},"what-to-log-and-at-which-level","What to log, and at which level",[10,857,858],{},"Levels only work if the team agrees what they mean, and a short convention is worth writing down.",[10,860,861,865],{},[862,863,864],"strong",{},"ERROR"," is for something a human must act on. If nobody would be woken for it, it is not an error. Every ERROR should carry enough context to start an investigation without reproducing.",[10,867,868,871],{},[862,869,870],{},"WARNING"," is for a degraded but handled condition: a retry, a fallback, a deprecated path taken. A warning that fires on every request is noise and should be an INFO counter instead.",[10,873,874,877],{},[862,875,876],{},"INFO"," is the operational narrative — the events that describe what the system did. These are the records worth structuring carefully, because they are what an incident is reconstructed from.",[10,879,880,883],{},[862,881,882],{},"DEBUG"," is for the author of the code, not the operator, and may be verbose. It should still be structured, because the moment it matters is the moment somebody is filtering thousands of lines.",[10,885,886,887,890,891,894,895,898],{},"What does not belong in any of them: secrets, tokens, full request bodies with personal data, and entire objects whose ",[14,888,889],{},"__repr__"," is unbounded. The last is a genuine outage risk — a ",[14,892,893],{},"logger.debug(\"state=%s\", huge_object)"," evaluates the repr even when DEBUG is disabled if the formatting is done eagerly with an f-string, which is the reason the ",[14,896,897],{},"%s","-with-args form still matters.",[223,900,902],{"className":225,"code":901,"language":227,"meta":228,"style":228},"# Lazy: the repr is only computed if DEBUG is enabled.\nlogger.debug(\"state=%s\", huge_object)\n\n# Eager: the repr is computed on every call, at every level.\nlogger.debug(f\"state={huge_object}\")\n",[14,903,904,909,914,918,923],{"__ignoreMap":228},[232,905,906],{"class":234,"line":235},[232,907,908],{},"# Lazy: the repr is only computed if DEBUG is enabled.\n",[232,910,911],{"class":234,"line":241},[232,912,913],{},"logger.debug(\"state=%s\", huge_object)\n",[232,915,916],{"class":234,"line":248},[232,917,245],{"emptyLinePlaceholder":244},[232,919,920],{"class":234,"line":254},[232,921,922],{},"# Eager: the repr is computed on every call, at every level.\n",[232,924,925],{"class":234,"line":259},[232,926,927],{},"logger.debug(f\"state={huge_object}\")\n",[19,929,931],{"id":930},"where-records-go-and-why-they-sometimes-go-nowhere","Where records go, and why they sometimes go nowhere",[10,933,934],{},"The logging module's dispatch rules are the source of nearly every \"my logs disappeared\" report, and they are simple enough to hold in mind.",[10,936,937,938,941,942,945,946,949,950,953],{},"A call to ",[14,939,940],{},"logger.info(...)"," on ",[14,943,944],{},"myapp.billing.gateway"," first checks that logger's effective level, walking up the dotted hierarchy until it finds one that has a level set. If the record passes, it is offered to that logger's handlers, then — unless ",[14,947,948],{},"propagate"," is ",[14,951,952],{},"False"," — to its parent's handlers, and so on to the root. Filters may reject it at any point.",[83,955,957,1078],{"className":956},[86],[88,958,96,963,96,966,96,969,96,986,96,990,96,994,96,1001,96,1006,96,1010,96,1016,96,1019,96,1023,96,1027,96,1031,96,1034,96,1038,96,1042,96,1046,96,1051,96,1055,96,1059,96,1063,96,1067,96,1070,96,1074],{"viewBox":959,"role":91,"ariaLabelledBy":960,"xmlns":95},"0 0 820 292",[961,962],"disp-t","disp-d",[98,964,965],{"id":961},"How a log record travels up the logger hierarchy",[102,967,968],{"id":962},"A record emitted on the myapp.billing.gateway logger is checked against the effective level inherited from myapp, then passed to each ancestor's handlers in turn up to the root, where pytest's capturing handler sits. Two interruption points are marked: a level that is too high discards the record immediately, and propagate set to False stops it before it reaches the root.",[970,971,972,973,96],"defs",{},"\n    ",[974,975,982],"marker",{"id":976,"viewBox":977,"refX":978,"refY":979,"markerWidth":980,"markerHeight":980,"orient":981},"disp-a","0 0 10 10","9","5","7","auto-start-reverse",[983,984],"path",{"d":985,"fill":176},"M0 0 L10 5 L0 10 z",[106,987],{"x":108,"y":108,"width":988,"height":989,"rx":111,"fill":112},"820","292",[114,991,993],{"x":992,"y":117,"textAnchor":118,"fontSize":119,"fontWeight":120,"fill":121},"410","Two places a record is lost",[106,995],{"x":996,"y":126,"width":997,"height":126,"rx":998,"fill":999,"stroke":121,"strokeWidth":1000},"34","290","10","#f4f1de","1.6",[114,1002,944],{"x":1003,"y":1004,"textAnchor":118,"fontSize":1005,"fontWeight":120,"fill":121},"179","74","11.5",[114,1007,1009],{"x":1003,"y":1008,"textAnchor":118,"fontSize":157,"fill":121},"93","logger.info(\"charge succeeded\")",[234,1011],{"x1":1003,"y1":1012,"x2":1003,"y2":1013,"stroke":176,"strokeWidth":1014,"markerEnd":1015},"108","128","1.8","url(#disp-a)",[106,1017],{"x":996,"y":1018,"width":997,"height":126,"rx":998,"fill":999,"stroke":121,"strokeWidth":1000},"132",[114,1020,1022],{"x":1003,"y":1021,"textAnchor":118,"fontSize":1005,"fontWeight":120,"fill":121},"154","myapp.billing",[114,1024,1026],{"x":1003,"y":1025,"textAnchor":118,"fontSize":157,"fill":121},"173","its handlers, then upward",[234,1028],{"x1":1003,"y1":1029,"x2":1003,"y2":1030,"stroke":176,"strokeWidth":1014,"markerEnd":1015},"188","208",[106,1032],{"x":996,"y":1033,"width":997,"height":126,"rx":998,"fill":175,"stroke":176,"strokeWidth":132},"212",[114,1035,1037],{"x":1003,"y":1036,"textAnchor":118,"fontSize":1005,"fontWeight":120,"fill":121},"234","root — pytest's handler",[114,1039,1041],{"x":1003,"y":1040,"textAnchor":118,"fontSize":157,"fill":195},"253","this is what caplog reads",[106,1043],{"x":1044,"y":126,"width":116,"height":1045,"rx":157,"fill":130,"stroke":131,"strokeWidth":132},"366","94",[114,1047,1050],{"x":1048,"y":1049,"fontSize":129,"fontWeight":120,"fill":121},"386","76","loss 1 · effective level too high",[114,1052,1054],{"x":1048,"y":1053,"fontSize":157,"fill":121},"100","no level set here, so it inherits from an ancestor",[114,1056,1058],{"x":1048,"y":1057,"fontSize":157,"fill":158},"122","fix: caplog.set_level(INFO, logger=\"myapp.billing\")",[106,1060],{"x":1044,"y":1061,"width":116,"height":1062,"rx":157,"fill":130,"stroke":131,"strokeWidth":132},"160","104",[114,1064,1066],{"x":1048,"y":1065,"fontSize":129,"fontWeight":120,"fill":121},"184","loss 2 · propagate = False",[114,1068,1069],{"x":1048,"y":1030,"fontSize":157,"fill":121},"the record never reaches the root at all",[114,1071,1073],{"x":1048,"y":1072,"fontSize":157,"fill":158},"230","fix: restore propagation for the test's duration",[114,1075,1077],{"x":1048,"y":1076,"fontSize":157,"fill":121},"252","common in libraries avoiding duplicate output",[207,1079,1080],{},"Both losses are silent: no warning, no error, just a record that never arrives. Checking the effective level and the propagate flag resolves almost every case.",[223,1082,1084],{"className":225,"code":1083,"language":227,"meta":228,"style":228},"import logging\n\n# A two-line diagnostic that answers both questions at once.\nlogger = logging.getLogger(\"myapp.billing.gateway\")\nprint(logger.getEffectiveLevel(), logger.propagate,\n      [h for h in logging.getLogger().handlers])\n",[14,1085,1086,1090,1094,1099,1104,1109],{"__ignoreMap":228},[232,1087,1088],{"class":234,"line":235},[232,1089,238],{},[232,1091,1092],{"class":234,"line":241},[232,1093,245],{"emptyLinePlaceholder":244},[232,1095,1096],{"class":234,"line":248},[232,1097,1098],{},"# A two-line diagnostic that answers both questions at once.\n",[232,1100,1101],{"class":234,"line":254},[232,1102,1103],{},"logger = logging.getLogger(\"myapp.billing.gateway\")\n",[232,1105,1106],{"class":234,"line":259},[232,1107,1108],{},"print(logger.getEffectiveLevel(), logger.propagate,\n",[232,1110,1111],{"class":234,"line":264},[232,1112,1113],{},"      [h for h in logging.getLogger().handlers])\n",[10,1115,1116],{},"Running that inside the failing test is faster than any amount of reasoning about configuration files, because it reports the state that actually exists rather than the state the configuration intended.",[19,1118,1120],{"id":1119},"logs-as-the-narrative-of-a-failure","Logs as the narrative of a failure",[10,1122,1123,1124,1128],{},"The most valuable property of a log stream during debugging is not detail but ",[1125,1126,1127],"em",{},"sequence",". Knowing that the cache lookup happened before the database write, and that the retry fired twice between them, usually identifies the bug without any further evidence.",[10,1130,1131],{},"That argues for logging at boundaries rather than inside logic. One INFO record when an operation starts, one when it finishes with its outcome, and one WARNING per unusual branch taken, gives a readable narrative at a cost of three records per request. Logging every intermediate value gives a stream nobody reads and a measurable performance cost.",[223,1133,1135],{"className":225,"code":1134,"language":227,"meta":228,"style":228},"import logging\nimport time\n\nlogger = logging.getLogger(__name__)\n\n\ndef charge(customer_id: str, amount_minor: int) -> Receipt:\n    started = time.monotonic()\n    logger.info(\"charge started\", extra={\"event\": \"charge_started\",\n                                         \"customer_id\": customer_id})\n    try:\n        receipt = _gateway.post(customer_id, amount_minor)\n    except GatewayTimeout:\n        # One record per unusual branch: the retry is part of the narrative.\n        logger.warning(\"gateway timed out, retrying\",\n                       extra={\"event\": \"charge_retry\", \"customer_id\": customer_id})\n        receipt = _gateway.post(customer_id, amount_minor)\n\n    logger.info(\"charge finished\",\n                extra={\"event\": \"charge_finished\",\n                       \"customer_id\": customer_id,\n                       \"duration_ms\": round((time.monotonic() - started) * 1000)})\n    return receipt\n",[14,1136,1137,1141,1146,1150,1154,1158,1162,1167,1172,1177,1182,1187,1192,1197,1202,1208,1214,1219,1224,1230,1236,1242,1248],{"__ignoreMap":228},[232,1138,1139],{"class":234,"line":235},[232,1140,238],{},[232,1142,1143],{"class":234,"line":241},[232,1144,1145],{},"import time\n",[232,1147,1148],{"class":234,"line":248},[232,1149,245],{"emptyLinePlaceholder":244},[232,1151,1152],{"class":234,"line":254},[232,1153,251],{},[232,1155,1156],{"class":234,"line":259},[232,1157,245],{"emptyLinePlaceholder":244},[232,1159,1160],{"class":234,"line":264},[232,1161,245],{"emptyLinePlaceholder":244},[232,1163,1164],{"class":234,"line":270},[232,1165,1166],{},"def charge(customer_id: str, amount_minor: int) -> Receipt:\n",[232,1168,1169],{"class":234,"line":276},[232,1170,1171],{},"    started = time.monotonic()\n",[232,1173,1174],{"class":234,"line":282},[232,1175,1176],{},"    logger.info(\"charge started\", extra={\"event\": \"charge_started\",\n",[232,1178,1179],{"class":234,"line":288},[232,1180,1181],{},"                                         \"customer_id\": customer_id})\n",[232,1183,1184],{"class":234,"line":294},[232,1185,1186],{},"    try:\n",[232,1188,1189],{"class":234,"line":300},[232,1190,1191],{},"        receipt = _gateway.post(customer_id, amount_minor)\n",[232,1193,1194],{"class":234,"line":306},[232,1195,1196],{},"    except GatewayTimeout:\n",[232,1198,1199],{"class":234,"line":312},[232,1200,1201],{},"        # One record per unusual branch: the retry is part of the narrative.\n",[232,1203,1205],{"class":234,"line":1204},15,[232,1206,1207],{},"        logger.warning(\"gateway timed out, retrying\",\n",[232,1209,1211],{"class":234,"line":1210},16,[232,1212,1213],{},"                       extra={\"event\": \"charge_retry\", \"customer_id\": customer_id})\n",[232,1215,1217],{"class":234,"line":1216},17,[232,1218,1191],{},[232,1220,1222],{"class":234,"line":1221},18,[232,1223,245],{"emptyLinePlaceholder":244},[232,1225,1227],{"class":234,"line":1226},19,[232,1228,1229],{},"    logger.info(\"charge finished\",\n",[232,1231,1233],{"class":234,"line":1232},20,[232,1234,1235],{},"                extra={\"event\": \"charge_finished\",\n",[232,1237,1239],{"class":234,"line":1238},21,[232,1240,1241],{},"                       \"customer_id\": customer_id,\n",[232,1243,1245],{"class":234,"line":1244},22,[232,1246,1247],{},"                       \"duration_ms\": round((time.monotonic() - started) * 1000)})\n",[232,1249,1251],{"class":234,"line":1250},23,[232,1252,1253],{},"    return receipt\n",[10,1255,1256,1257,1260,1261,1264,1265,1270],{},"Recording the duration on the finishing record turns the log into a crude but always-available profiler: filtering for ",[14,1258,1259],{},"charge_finished"," with ",[14,1262,1263],{},"duration_ms > 1000"," finds the slow cases without any instrumentation, and it works in production where a profiler does not. Where that is not enough, the real tools are in ",[1266,1267,1269],"a",{"href":1268},"\u002Fsystematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002F","CPU profiling with cProfile and py-spy",", but the log-derived number is what tells you whether to reach for them.",[19,1272,1274],{"id":1273},"tracing-a-request-through-a-test","Tracing a request through a test",[10,1276,1277],{},"For a service that emits spans in production, asserting on the trace is the natural extension of asserting on logs, and the in-memory exporter makes it cheap.",[223,1279,1281],{"className":225,"code":1280,"language":227,"meta":228,"style":228},"import pytest\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import SimpleSpanProcessor\nfrom opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter\n\n\n@pytest.fixture\ndef spans():\n    exporter = InMemorySpanExporter()\n    provider = TracerProvider()\n    provider.add_span_processor(SimpleSpanProcessor(exporter))\n    trace.set_tracer_provider(provider)\n    yield exporter\n    exporter.clear()\n\n\ndef test_charge_emits_a_child_span(spans, gateway):\n    gateway.charge(\"cus_12\", 1234)\n\n    names = [span.name for span in spans.get_finished_spans()]\n    assert names == [\"http.post \u002Fcharges\", \"charge\"]      # child before parent\n\n    child, parent = spans.get_finished_spans()\n    # Context propagation is the thing that breaks silently in production.\n    assert child.parent.span_id == parent.context.span_id\n",[14,1282,1283,1287,1292,1297,1302,1307,1311,1315,1320,1325,1330,1335,1340,1345,1350,1355,1359,1363,1368,1372,1376,1381,1386,1390,1396,1402],{"__ignoreMap":228},[232,1284,1285],{"class":234,"line":235},[232,1286,417],{},[232,1288,1289],{"class":234,"line":241},[232,1290,1291],{},"from opentelemetry import trace\n",[232,1293,1294],{"class":234,"line":248},[232,1295,1296],{},"from opentelemetry.sdk.trace import TracerProvider\n",[232,1298,1299],{"class":234,"line":254},[232,1300,1301],{},"from opentelemetry.sdk.trace.export import SimpleSpanProcessor\n",[232,1303,1304],{"class":234,"line":259},[232,1305,1306],{},"from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter\n",[232,1308,1309],{"class":234,"line":264},[232,1310,245],{"emptyLinePlaceholder":244},[232,1312,1313],{"class":234,"line":270},[232,1314,245],{"emptyLinePlaceholder":244},[232,1316,1317],{"class":234,"line":276},[232,1318,1319],{},"@pytest.fixture\n",[232,1321,1322],{"class":234,"line":282},[232,1323,1324],{},"def spans():\n",[232,1326,1327],{"class":234,"line":288},[232,1328,1329],{},"    exporter = InMemorySpanExporter()\n",[232,1331,1332],{"class":234,"line":294},[232,1333,1334],{},"    provider = TracerProvider()\n",[232,1336,1337],{"class":234,"line":300},[232,1338,1339],{},"    provider.add_span_processor(SimpleSpanProcessor(exporter))\n",[232,1341,1342],{"class":234,"line":306},[232,1343,1344],{},"    trace.set_tracer_provider(provider)\n",[232,1346,1347],{"class":234,"line":312},[232,1348,1349],{},"    yield exporter\n",[232,1351,1352],{"class":234,"line":1204},[232,1353,1354],{},"    exporter.clear()\n",[232,1356,1357],{"class":234,"line":1210},[232,1358,245],{"emptyLinePlaceholder":244},[232,1360,1361],{"class":234,"line":1216},[232,1362,245],{"emptyLinePlaceholder":244},[232,1364,1365],{"class":234,"line":1221},[232,1366,1367],{},"def test_charge_emits_a_child_span(spans, gateway):\n",[232,1369,1370],{"class":234,"line":1226},[232,1371,513],{},[232,1373,1374],{"class":234,"line":1232},[232,1375,245],{"emptyLinePlaceholder":244},[232,1377,1378],{"class":234,"line":1238},[232,1379,1380],{},"    names = [span.name for span in spans.get_finished_spans()]\n",[232,1382,1383],{"class":234,"line":1244},[232,1384,1385],{},"    assert names == [\"http.post \u002Fcharges\", \"charge\"]      # child before parent\n",[232,1387,1388],{"class":234,"line":1250},[232,1389,245],{"emptyLinePlaceholder":244},[232,1391,1393],{"class":234,"line":1392},24,[232,1394,1395],{},"    child, parent = spans.get_finished_spans()\n",[232,1397,1399],{"class":234,"line":1398},25,[232,1400,1401],{},"    # Context propagation is the thing that breaks silently in production.\n",[232,1403,1405],{"class":234,"line":1404},26,[232,1406,1407],{},"    assert child.parent.span_id == parent.context.span_id\n",[10,1409,1410,1411,1414],{},"The assertion worth making is the last one. A missing span is obvious in a trace viewer; a span whose parent is wrong produces a trace that ",[1125,1412,1413],{},"looks"," complete and is silently broken, and it happens whenever context fails to cross a thread, a task or a queue boundary. A test that asserts the parent relationship catches it at the pull request.",[83,1416,1418,1524],{"className":1417},[86],[88,1419,96,1424,96,1427,96,1430,96,1442,96,1445,96,1448,96,1453,96,1458,96,1463,96,1465,96,1469,96,1474,96,1477,96,1480,96,1487,96,1492,96,1496,96,1499,96,1503,96,1507,96,1511,96,1515,96,1520],{"viewBox":1420,"role":91,"ariaLabelledBy":1421,"xmlns":95},"0 0 820 268",[1422,1423],"trc-t","trc-d",[98,1425,1426],{"id":1422},"Where trace context is lost between components",[102,1428,1429],{"id":1423},"A request span propagates correctly into a handler span and a database span. At a queue boundary the context is not carried, so the worker starts a new root span, and the resulting trace appears complete while the worker's work is detached from the request that caused it.",[970,1431,972,1432,972,1437,96],{},[974,1433,1435],{"id":1434,"viewBox":977,"refX":978,"refY":979,"markerWidth":980,"markerHeight":980,"orient":981},"trc-a",[983,1436],{"d":985,"fill":176},[974,1438,1440],{"id":1439,"viewBox":977,"refX":978,"refY":979,"markerWidth":980,"markerHeight":980,"orient":981},"trc-b",[983,1441],{"d":985,"fill":131},[106,1443],{"x":108,"y":108,"width":988,"height":1444,"rx":111,"fill":112},"268",[114,1446,1447],{"x":992,"y":117,"textAnchor":118,"fontSize":119,"fontWeight":120,"fill":121},"A trace that looks complete and is not",[106,1449],{"x":996,"y":1450,"width":1451,"height":1452,"rx":998,"fill":175,"stroke":176,"strokeWidth":132},"56","200","50",[114,1454,1457],{"x":1455,"y":1456,"textAnchor":118,"fontSize":1005,"fontWeight":120,"fill":121},"134","86","http.post \u002Fcharges",[234,1459],{"x1":1455,"y1":1460,"x2":1455,"y2":1461,"stroke":176,"strokeWidth":1014,"markerEnd":1462},"110","130","url(#trc-a)",[106,1464],{"x":996,"y":1455,"width":1451,"height":1452,"rx":998,"fill":175,"stroke":176,"strokeWidth":132},[114,1466,1468],{"x":1455,"y":1467,"textAnchor":118,"fontSize":1005,"fontWeight":120,"fill":121},"164","charge (handler)",[234,1470],{"x1":1471,"y1":1472,"x2":1473,"y2":1472,"stroke":176,"strokeWidth":1014,"markerEnd":1462},"238","159","272",[106,1475],{"x":1476,"y":1455,"width":162,"height":1452,"rx":998,"fill":175,"stroke":176,"strokeWidth":132},"278",[114,1478,1479],{"x":127,"y":1467,"textAnchor":118,"fontSize":1005,"fontWeight":120,"fill":121},"db.insert",[234,1481],{"x1":1036,"y1":1049,"x2":1482,"y2":1049,"stroke":131,"strokeWidth":1014,"strokeDashArray":1483,"markerEnd":1486},"470",[1484,1485],"6","4","url(#trc-b)",[114,1488,1491],{"x":1489,"y":1490,"textAnchor":118,"fontSize":157,"fill":158},"352","66","enqueued — context not carried",[106,1493],{"x":1494,"y":126,"width":1495,"height":1018,"rx":157,"fill":130,"stroke":131,"strokeWidth":132},"476","308",[114,1497,1498],{"x":179,"y":136,"textAnchor":118,"fontSize":129,"fontWeight":120,"fill":121},"worker: new ROOT span",[114,1500,1502],{"x":1501,"y":1062,"fontSize":157,"fill":121},"494","send_receipt — parent is None",[114,1504,1506],{"x":1501,"y":1505,"fontSize":157,"fill":158},"126","appears as an unrelated trace",[114,1508,1510],{"x":1501,"y":1509,"fontSize":157,"fill":121},"148","assert child.parent.span_id",[114,1512,1514],{"x":1501,"y":1513,"fontSize":157,"fill":195},"168","catches it in a test",[106,1516],{"x":996,"y":166,"width":1517,"height":1518,"rx":998,"fill":112,"stroke":146,"strokeWidth":1519},"750","46","1.5",[114,1521,1523],{"x":992,"y":1522,"textAnchor":118,"fontSize":129,"fill":121},"232","Queue, thread and task boundaries are where propagation silently stops; each needs an explicit carrier.",[207,1525,1526],{},"The failure is invisible in a viewer because both traces render fine. Only an assertion on the parent relationship distinguishes them.",[19,1528,1530],{"id":1529},"logging-configuration-that-behaves-in-tests","Logging configuration that behaves in tests",[10,1532,1533],{},"Two configuration mistakes cause most of the confusion, and both come from configuring logging at import time.",[10,1535,1536,1539,1540,1543],{},[14,1537,1538],{},"logging.basicConfig()"," at module level runs whenever the module is imported, including during collection, and installs a handler on the root logger. pytest then adds its own, and every record is emitted twice. Configuration belongs in an entry point — ",[14,1541,1542],{},"main()",", the application factory, a CLI callback — never in a module body.",[10,1545,1546,1548,1549,1551],{},[14,1547,738],{}," on a package logger stops records reaching the root, which is where ",[14,1550,42],{}," listens. Libraries sometimes do this to avoid duplicate output; the effect in a test suite is that the library appears silent. A fixture that restores propagation for the duration of a test is the pragmatic fix:",[223,1553,1555],{"className":225,"code":1554,"language":227,"meta":228,"style":228},"import logging\n\nimport pytest\n\n\n@pytest.fixture\ndef library_logs(caplog):\n    logger = logging.getLogger(\"thirdparty\")\n    original, logger.propagate = logger.propagate, True\n    caplog.set_level(logging.DEBUG, logger=\"thirdparty\")\n    yield caplog\n    logger.propagate = original\n",[14,1556,1557,1561,1565,1569,1573,1577,1581,1586,1591,1596,1601,1606],{"__ignoreMap":228},[232,1558,1559],{"class":234,"line":235},[232,1560,238],{},[232,1562,1563],{"class":234,"line":241},[232,1564,245],{"emptyLinePlaceholder":244},[232,1566,1567],{"class":234,"line":248},[232,1568,417],{},[232,1570,1571],{"class":234,"line":254},[232,1572,245],{"emptyLinePlaceholder":244},[232,1574,1575],{"class":234,"line":259},[232,1576,245],{"emptyLinePlaceholder":244},[232,1578,1579],{"class":234,"line":264},[232,1580,1319],{},[232,1582,1583],{"class":234,"line":270},[232,1584,1585],{},"def library_logs(caplog):\n",[232,1587,1588],{"class":234,"line":276},[232,1589,1590],{},"    logger = logging.getLogger(\"thirdparty\")\n",[232,1592,1593],{"class":234,"line":282},[232,1594,1595],{},"    original, logger.propagate = logger.propagate, True\n",[232,1597,1598],{"class":234,"line":288},[232,1599,1600],{},"    caplog.set_level(logging.DEBUG, logger=\"thirdparty\")\n",[232,1602,1603],{"class":234,"line":294},[232,1604,1605],{},"    yield caplog\n",[232,1607,1608],{"class":234,"line":300},[232,1609,1610],{},"    logger.propagate = original\n",[10,1612,1613,1614,1617,1618,1621,1622,1624],{},"Beyond those, keep the test environment's logging configuration close to production's. A suite that runs with a plain formatter while production emits JSON will not catch a field that breaks serialization, and that failure — a ",[14,1615,1616],{},"datetime"," or a ",[14,1619,1620],{},"Decimal"," in ",[14,1623,60],{}," that the JSON encoder rejects — surfaces as a lost log line at exactly the moment the log was needed.",[19,1626,1628],{"id":1627},"testing-that-the-observability-itself-works","Testing that the observability itself works",[10,1630,1631],{},"Instrumentation is code, and code that is never exercised rots. Three small tests keep it honest without turning the suite into a logging test suite.",[10,1633,1634,1637,1638,1617,1640,1642],{},[862,1635,1636],{},"The formatter survives real payloads."," A JSON formatter that cannot serialise a ",[14,1639,1620],{},[14,1641,1616],{}," drops the record, and the drop is discovered during an incident.",[223,1644,1646],{"className":225,"code":1645,"language":227,"meta":228,"style":228},"import datetime as dt\nimport json\nfrom decimal import Decimal\n\n\ndef test_json_formatter_handles_domain_types(json_handler):\n    record = make_record(extra={\"amount\": Decimal(\"12.34\"),\n                                \"at\": dt.datetime.now(dt.timezone.utc)})\n    output = json_handler.format(record)\n    json.loads(output)          # raises if the formatter produced invalid JSON\n",[14,1647,1648,1653,1658,1663,1667,1671,1676,1681,1686,1691],{"__ignoreMap":228},[232,1649,1650],{"class":234,"line":235},[232,1651,1652],{},"import datetime as dt\n",[232,1654,1655],{"class":234,"line":241},[232,1656,1657],{},"import json\n",[232,1659,1660],{"class":234,"line":248},[232,1661,1662],{},"from decimal import Decimal\n",[232,1664,1665],{"class":234,"line":254},[232,1666,245],{"emptyLinePlaceholder":244},[232,1668,1669],{"class":234,"line":259},[232,1670,245],{"emptyLinePlaceholder":244},[232,1672,1673],{"class":234,"line":264},[232,1674,1675],{},"def test_json_formatter_handles_domain_types(json_handler):\n",[232,1677,1678],{"class":234,"line":270},[232,1679,1680],{},"    record = make_record(extra={\"amount\": Decimal(\"12.34\"),\n",[232,1682,1683],{"class":234,"line":276},[232,1684,1685],{},"                                \"at\": dt.datetime.now(dt.timezone.utc)})\n",[232,1687,1688],{"class":234,"line":282},[232,1689,1690],{},"    output = json_handler.format(record)\n",[232,1692,1693],{"class":234,"line":288},[232,1694,1695],{},"    json.loads(output)          # raises if the formatter produced invalid JSON\n",[10,1697,1698,1701,1702,1705],{},[862,1699,1700],{},"Nothing sensitive is logged."," A test that runs a representative operation and asserts no captured record contains a known secret value costs nothing and catches the accidental ",[14,1703,1704],{},"logger.debug(\"headers=%s\", headers)",".",[223,1707,1709],{"className":225,"code":1708,"language":227,"meta":228,"style":228},"def test_no_secrets_in_logs(caplog, client):\n    client.authenticate(token=\"sk_test_SENSITIVE\")      # a known sentinel\n    assert \"sk_test_SENSITIVE\" not in caplog.text\n",[14,1710,1711,1716,1721],{"__ignoreMap":228},[232,1712,1713],{"class":234,"line":235},[232,1714,1715],{},"def test_no_secrets_in_logs(caplog, client):\n",[232,1717,1718],{"class":234,"line":241},[232,1719,1720],{},"    client.authenticate(token=\"sk_test_SENSITIVE\")      # a known sentinel\n",[232,1722,1723],{"class":234,"line":248},[232,1724,1725],{},"    assert \"sk_test_SENSITIVE\" not in caplog.text\n",[10,1727,1728,1731],{},[862,1729,1730],{},"The audit records that are contractual are emitted."," For the small number of log lines that exist because a regulation or an operational runbook requires them, assert on their presence and their fields exactly as you would any other requirement — they are the one category where asserting on log output is unambiguously correct.",[10,1733,1734],{},"The three together take an afternoon and remove the two failure modes that matter: instrumentation that breaks silently, and instrumentation that leaks. Everything else about logging is best verified by reading one failing test's captured output and asking whether it would have been enough to diagnose the problem without re-running anything. If the answer is no, the fix is usually one more field on one existing record rather than a new logging strategy.",[10,1736,1737,1738,1740],{},"That question is worth asking deliberately after every incident, while the details are fresh: what single field, on a record that already existed, would have shortened the diagnosis? The answers accumulate into instrumentation that is genuinely shaped by how the system fails, rather than by what seemed worth logging when the code was written. It is also the cheapest form of post-incident work available, since adding one key to one ",[14,1739,60],{}," dictionary needs no design discussion and no migration. Over a year it is the difference between a log stream that describes the system and one that describes what somebody once imagined the system would do.",[19,1742,1744],{"id":1743},"frequently-asked-questions","Frequently Asked Questions",[10,1746,1747,1750,1751,1753,1754,1756,1757,1759,1760,1763,1764,1767],{},[862,1748,1749],{},"Why does caplog see no records from my library?","\nBecause the library's logger has ",[14,1752,948],{}," set to ",[14,1755,952],{},", or a handler was configured at import time that swallows the record before it reaches pytest's handler. ",[14,1758,42],{}," attaches at the root, so anything that stops propagation is invisible to it. Use ",[14,1761,1762],{},"caplog.set_level"," with the specific logger name, and check for ",[14,1765,1766],{},"logging.basicConfig"," calls at module import.",[10,1769,1770,1773],{},[862,1771,1772],{},"Should tests assert on log messages?","\nOnly when the log line is part of the contract — an audit trail, a security event, an operator-facing warning. Asserting on ordinary debug output couples the test to wording nobody considers stable, and those assertions break on every rephrasing. Assert on the structured fields rather than the formatted string when you do assert.",[10,1775,1776,1779,1781,1782,1784,1785,1787,1788,1790],{},[862,1777,1778],{},"What is the difference between caplog and log_cli?",[14,1780,42],{}," is a fixture that captures records so a test can inspect them; ",[14,1783,46],{}," is a configuration option that streams log output to the terminal live while tests run. ",[14,1786,42],{}," is for assertions, ",[14,1789,46],{}," is for watching a hanging or slow test in real time.",[10,1792,1793,1796],{},[862,1794,1795],{},"How do correlation ids help in a test suite?","\nThey let every log line from one logical operation be filtered out of an interleaved stream, which matters as soon as tests run in parallel or the code under test uses threads. Generating the id in a fixture and binding it through a context variable gives every record the same key without threading an argument through the code.",[10,1798,1799,1802],{},[862,1800,1801],{},"Is OpenTelemetry worth wiring into tests?","\nFor a service with real tracing in production, yes: an in-memory span exporter lets a test assert on the shape of the trace, which catches missing spans and broken context propagation before they reach production. For a library with no tracing, it is overhead with no payoff.",[19,1804,1806],{"id":1805},"related-guides","Related guides",[24,1808,1809,1816,1823,1830,1837],{},[27,1810,1811,1812,1705],{},"Get capture working reliably in ",[1266,1813,1815],{"href":1814},"\u002Fsystematic-debugging-performance-profiling\u002Flogging-and-observability-for-debugging\u002Fstructured-logging-that-survives-pytest-capture\u002F","structured logging that survives pytest capture",[27,1817,1818,1819,1705],{},"Assert on span relationships with ",[1266,1820,1822],{"href":1821},"\u002Fsystematic-debugging-performance-profiling\u002Flogging-and-observability-for-debugging\u002Ftracing-a-request-through-a-test-with-opentelemetry\u002F","tracing a request through a test with OpenTelemetry",[27,1824,1825,1826,1705],{},"Tune capture levels and streaming via ",[1266,1827,1829],{"href":1828},"\u002Fadvanced-pytest-architecture-configuration\u002Fpytest-configuration-best-practices\u002Fcapturing-logs-with-caplog-and-log-cli\u002F","capturing logs with caplog and log_cli",[27,1831,1832,1833,1705],{},"Put the logs in front of whoever reads the failure using ",[1266,1834,1836],{"href":1835},"\u002Fsystematic-debugging-performance-profiling\u002Fdebugging-tests-in-ci-and-containers\u002Fcapturing-artifacts-from-a-failed-ci-test-run\u002F","capturing artifacts from a failed CI test run",[27,1838,1839,1840,1705],{},"Read the exception those logs surround with ",[1266,1841,1843],{"href":1842},"\u002Fsystematic-debugging-performance-profiling\u002Freading-tracebacks-and-exception-chains\u002F","reading tracebacks and exception chains",[10,1845,1846,1847],{},"← Back to ",[1266,1848,1850],{"href":1849},"\u002Fsystematic-debugging-performance-profiling\u002F","Systematic Debugging & Performance Profiling",[1852,1853,1854],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}",{"title":228,"searchDepth":241,"depth":241,"links":1856},[1857,1858,1859,1866,1867,1868,1869,1870,1871,1872,1873,1874,1875],{"id":21,"depth":241,"text":22},{"id":70,"depth":241,"text":71},{"id":215,"depth":241,"text":216,"children":1860},[1861,1862,1863,1864,1865],{"id":220,"depth":248,"text":221},{"id":345,"depth":248,"text":346},{"id":466,"depth":248,"text":467},{"id":546,"depth":248,"text":547},{"id":624,"depth":248,"text":625},{"id":690,"depth":241,"text":691},{"id":742,"depth":241,"text":743},{"id":854,"depth":241,"text":855},{"id":930,"depth":241,"text":931},{"id":1119,"depth":241,"text":1120},{"id":1273,"depth":241,"text":1274},{"id":1529,"depth":241,"text":1530},{"id":1627,"depth":241,"text":1628},{"id":1743,"depth":241,"text":1744},{"id":1805,"depth":241,"text":1806},"Turn logs into diagnostics: structured records, correlation ids, pytest caplog and log_cli, propagation rules, and tracing a request end to end inside a test.","md",{"slug":1879,"type":1880,"breadcrumb":1881,"datePublished":1882,"dateModified":1882,"faq":1883,"howto":1894},"logging-and-observability-for-debugging","topic","Logging & Observability","2026-09-18",[1884,1886,1888,1890,1892],{"q":1749,"a":1885},"Because the library's logger has propagate set to False, or a handler was configured at import time that swallows the record before it reaches pytest's handler. caplog attaches at the root, so anything that stops propagation is invisible to it. Use caplog.set_level with the specific logger name, and check for logging.basicConfig calls at module import.",{"q":1772,"a":1887},"Only when the log line is part of the contract — an audit trail, a security event, an operator-facing warning. Asserting on ordinary debug output couples the test to wording nobody considers stable, and those assertions break on every rephrasing. Assert on the structured fields rather than the formatted string when you do assert.",{"q":1778,"a":1889},"caplog is a fixture that captures records so a test can inspect them; log_cli is a configuration option that streams log output to the terminal live while tests run. caplog is for assertions, log_cli is for watching a hanging or slow test in real time.",{"q":1795,"a":1891},"They let every log line from one logical operation be filtered out of an interleaved stream, which matters as soon as tests run in parallel or the code under test uses threads. Generating the id in a fixture and binding it through a context variable gives every record the same key without threading an argument through the code.",{"q":1801,"a":1893},"For a service with real tracing in production, yes: an in-memory span exporter lets a test assert on the shape of the trace, which catches missing spans and broken context propagation before they reach production. For a library with no tracing, it is overhead with no payoff.",{"name":1895,"description":1896,"steps":1897},"How to make logs useful for debugging tests","Emit structured records, correlate them per operation, and capture them so a failure carries its own context.",[1898,1901,1904,1907,1910],{"name":1899,"text":1900},"Log structured fields, not formatted prose","Attach values with the extra argument or a structured logging library so fields can be filtered and asserted on.",{"name":1902,"text":1903},"Bind a correlation id per operation","Generate an id in a fixture and put it in a context variable so every record from that test carries it.",{"name":1905,"text":1906},"Capture at the right level in tests","Use caplog.set_level for the specific logger under test rather than raising the root level for everything.",{"name":1908,"text":1909},"Attach captured logs to failures","Use a report hookwrapper so a failing test's log output appears in the terminal and in the CI report.",{"name":1911,"text":1912},"Assert only on contractual log lines","Restrict assertions to audit and operator-facing events, and assert on fields rather than on message wording.","\u002Fsystematic-debugging-performance-profiling\u002Flogging-and-observability-for-debugging",{"title":5,"description":1876},"systematic-debugging-performance-profiling\u002Flogging-and-observability-for-debugging\u002Findex","s9lKV6xmc-WwZ_UMykx_D-hr3F0FpZMwUlAuxAxZrE4",1789718765722]