pytest's most valuable feature is not fixtures or parametrization — it is that a bare assert produces a failure message describing exactly which parts of the expression differed. That behaviour is not magic in the interpreter; it is a bytecode rewrite applied to specific modules under specific conditions. Knowing where those conditions stop applying, and how to extend the output for your own types, is the difference between a failure that explains itself and one that says assert False.
Prerequisites
pytest >= 8.0. The hook signatures here are stable across 7 and 8, butjunit_familydefaults changed in 6.- Write access to
conftest.pyat the root of the test tree, since hook implementations must be collected before the modules they affect. - Familiarity with hookwrappers, covered in writing a hookwrapper for test reports.
- A CI system that consumes JUnit XML, if the reporting half is relevant.
Core concept: rewriting happens at import, to selected modules
When pytest imports a test module it does not execute the source directly. It parses the source to an AST, walks it, and replaces every assert statement with a block that evaluates the expression's sub-parts into temporary variables, checks the result, and — on failure — builds an explanation from those temporaries. The rewritten module is compiled and cached as a .pyc, so the cost is paid once per source change.
The crucial detail is which modules get this treatment: test modules matched by the collection patterns, conftest.py files, and plugins registered through entry points. Everything else — your application code, and any helper module you import from a test — is loaded normally, so an assert inside it produces a bare AssertionError with no explanation at all.
Step-by-step implementation
1. Keep introspection in shared helpers
# conftest.py — must run before anything imports the helper
import pytest
# Rewrites tests/helpers.py too, so asserts inside it explain themselves.
pytest.register_assert_rewrite("tests.helpers")
# tests/helpers.py
def assert_valid_invoice(invoice) -> None:
# With registration: "assert Decimal('12.30') == Decimal('12.34')".
# Without it: "AssertionError" and nothing else.
assert invoice.total == sum(line.amount for line in invoice.lines)
assert invoice.currency in {"GBP", "USD"}
Registration must happen before the first import of the module, which in practice means the top of the root conftest.py. A plugin distributed through entry points gets the same treatment automatically.
2. Teach pytest to compare your own types
# conftest.py
from decimal import Decimal
from myapp.money import Money
def pytest_assertrepr_compare(config, op, left, right):
"""Custom failure output for Money == Money."""
if op != "==" or not isinstance(left, Money) or not isinstance(right, Money):
return None # let other implementations handle it
lines = ["Money instances differ:"]
if left.currency != right.currency:
lines.append(f" currency: {left.currency} != {right.currency}")
if left.amount != right.amount:
delta = Decimal(right.amount - left.amount) / 100
lines.append(f" amount: {left.amount} != {right.amount} (delta {delta:+})")
return lines
E assert Money(1230, 'GBP') == Money(1234, 'GBP')
E Money instances differ:
E amount: 1230 != 1234 (delta +0.04)
Returning None for unhandled cases is not optional courtesy — a hook that returns a list unconditionally suppresses every other implementation, including pytest's own excellent diff for dictionaries and sequences.
3. Attach diagnostics to failures automatically
# conftest.py
import pytest
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
report = (yield).get_result()
# Only decorate real failures in the call phase, not setup or teardown.
if report.when != "call" or not report.failed:
return
# Anything appended here appears in the terminal AND in the JUnit XML.
client = item.funcargs.get("http_client")
if client is not None and client.last_response is not None:
report.sections.append((
"captured HTTP exchange",
f"{client.last_request.method} {client.last_request.url}\n"
f"→ {client.last_response.status_code}\n{client.last_response.text[:2000]}",
))
This is the highest-value hook in the file. A failing API test that prints the request it made and the response it got needs no reproduction step; the same test without it starts a twenty-minute cycle of adding prints and re-running CI.
4. Emit machine-readable results
# pyproject.toml
[tool.pytest.ini_options]
addopts = "--junitxml=reports/junit.xml -o junit_family=xunit2"
junit_logging = "system-out" # captured output lands in the XML
junit_duration_report = "call" # duration excludes fixture setup
def test_charge_is_idempotent(record_property, gateway):
# Properties appear as <property> elements the CI dashboard can display.
record_property("gateway_version", gateway.version)
record_property("idempotency_key", gateway.last_key)
assert gateway.charge_count == 1
xunit2 is the family nearly every modern CI parser expects; the legacy xunit1 output silently loses properties in several of them. junit_duration_report = "call" matters for anyone using the report to find slow tests, since the default includes fixture setup and makes every test that touches a session fixture look slow.
5. Add a summary line the reader will actually see
# conftest.py
def pytest_terminal_summary(terminalreporter, exitstatus, config):
failed = terminalreporter.stats.get("failed", [])
if not failed:
return
terminalreporter.write_sep("=", "artefacts for failed tests", red=True)
for report in failed:
terminalreporter.write_line(f" {report.nodeid} → reports/{report.nodeid}.log")
Verification
Write one deliberately failing test and read its whole output — terminal and XML — before trusting any of this.
pytest tests/test_meta_failure.py -q ; xmllint --format reports/junit.xml | head -30
E assert Money(1230, 'GBP') == Money(1234, 'GBP')
E Money instances differ:
E amount: 1230 != 1234 (delta +0.04)
------------------------- captured HTTP exchange -------------------------
POST https://billing.test/charges
→ 402
{"error": "insufficient_funds"}
<testcase classname="tests.test_meta_failure" name="test_total" time="0.031">
<properties><property name="gateway_version" value="2.4.1"/></properties>
<failure message="assert Money(1230, 'GBP') == Money(1234, 'GBP')">…</failure>
<system-out>POST https://billing.test/charges …</system-out>
</testcase>
If the custom comparison text is missing, the hook returned None — usually an isinstance check that does not match because the objects are subclasses. If the section is missing from the XML, junit_logging is unset.
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| Helper assertions show no detail | Module imported without rewriting | pytest.register_assert_rewrite at the top of conftest.py |
PytestAssertRewriteWarning | Module already imported when registration ran | Move the registration above the import, or out of a plugin's body |
| Custom comparison never appears | Hook returned a list for every case, or wrong operand types | Return None when unhandled; check subclasses |
| Properties missing from XML | junit_family left at the legacy default | Set junit_family=xunit2 |
| Report hook runs but nothing shows | Added to report.sections outside the call phase | Guard on report.when == "call" |
| Rewriting appears disabled entirely | Running with --assert=plain, or PYTHONDONTWRITEBYTECODE plus a read-only tree | Remove the flag; the cache falls back to in-memory |
Reading a rewritten assertion
It is worth seeing what the rewrite produces once, because it explains both the power and the limits.
def test_totals():
assert order.total() == expected_total(order)
becomes, in essence:
def test_totals():
tmp_left = order.total()
tmp_right = expected_total(order)
tmp_result = tmp_left == tmp_right
if not tmp_result:
raise AssertionError(
_format_explanation(tmp_left, "==", tmp_right)
)
Each sub-expression is evaluated exactly once into a temporary, which is what lets the message show both sides without re-running anything. Two consequences follow. Side effects are not duplicated, so assert queue.pop() == 3 is safe. And expressions are evaluated eagerly left to right, so assert x is not None and x.value == 3 still short-circuits correctly — the rewrite preserves boolean semantics rather than flattening them.
The limit is that the explanation is built from the operands' repr(). An object whose repr is <Order object at 0x7f…> produces a useless message no matter how good the rewriting is, which makes a decent __repr__ on domain types one of the highest-leverage testability changes available. Ten minutes adding __repr__ to five model classes improves every future failure in the suite.
Report hooks, in the order they fire
Reporting hooks are easy to use incorrectly because several of them look interchangeable. They are not; each sees a different stage.
pytest_runtest_makereport is called three times per test — once each for setup, call and teardown — and is the only place with access to both the item (fixtures, markers, the node) and the outcome. Anything that needs fixture state at failure time belongs here.
pytest_runtest_logreport receives each finished report and is the right place to forward results: to a dashboard, a message queue, a file. It has the report but not the item, so it cannot reach fixtures.
pytest_terminal_summary runs once at the end with the accumulated statistics, which makes it the place for aggregate output — a list of artefact paths, a count of quarantined tests, a link to the run.
logreport, and outcomes do not exist yet in setup.Capture, and where the output actually goes
Half the confusion about pytest reporting comes from capture. By default pytest replaces sys.stdout, sys.stderr and the file descriptors beneath them for the duration of each test, buffers everything, and prints it only if the test fails. That is the right default — a passing suite should be quiet — but it means the same print behaves differently depending on flags nobody remembers setting.
pytest # captured; shown only on failure
pytest -s # capture disabled entirely; everything streams live
pytest --capture=sys # replaces sys.stdout only; C-level writes escape
pytest --capture=fd # the default: file descriptors too, so C output is caught
pytest -rP # show captured output for PASSED tests as well
--capture=fd versus sys matters more than it looks. A C extension, a subprocess or anything writing to file descriptor 1 directly bypasses sys.stdout; only the file-descriptor mode catches it. A test that shells out and sees no output in its failure report is usually running under sys capture, set by a plugin or a stale flag in addopts.
Logging is captured separately from streams, through a handler pytest installs, which is why caplog can assert on records that never appeared on stdout. Keeping the two straight — streams for print and subprocess output, caplog for logging — removes most of the surprise.
Reporting for humans and for machines
The two audiences want different things and it is worth serving both deliberately rather than hoping one format suffices.
A human reading a terminal wants the failure at the end, in order of severity, with enough context to act. -q for the run, --tb=short for compact tracebacks, and -x when iterating locally give that. The full traceback style is rarely useful in CI, where the volume of --tb=long output from twenty failures buries the first one.
A machine wants stable identifiers and a parseable file. JUnit XML is the common denominator, but two additions make it far more useful. Per-test properties record the run's context — build number, dependency versions, feature flags — so a historical query can ask which versions correlate with a failure. And a stable nodeid matters more than anything else: a test whose id changes because its parametrization ids are generated from object repr cannot be tracked across runs, so flake-detection tooling sees a new test every time.
# conftest.py — stamp every run with its context, once.
def pytest_configure(config):
config.stash["run_context"] = {
"commit": os.environ.get("GIT_COMMIT", "local"),
"python": platform.python_version(),
}
def pytest_runtest_setup(item):
for key, value in item.config.stash["run_context"].items():
item.user_properties.append((key, value))
Three lines of context in every record is what turns a pile of XML into something answerable. The question worth being able to answer is not "did this test fail?" but "has this test failed before, on which commits, and under which interpreter" — and none of that is recoverable after the fact if the reports did not carry it.
What good failure output contains
The test of this whole area is simple: can a colleague diagnose a CI failure from the build log alone, without re-running anything? Four things make that possible, and none of them are expensive.
The values that differed, which assertion rewriting gives for free as long as the assertion is in a rewritten module and the objects have a usable repr.
The inputs, which parametrization supplies in the test id — one reason readable test ids are worth the effort — and which factories supply if the test constructed its data explicitly.
The environment, meaning the versions, flags and configuration this run used. A header printed once per run costs nothing and answers "was this the run with the new dependency?" immediately.
The interaction, for anything that crossed a boundary: the SQL, the HTTP exchange, the message published. This is what the report hookwrapper attaches, and it is the item most often missing.
A team that adds all four typically finds the median time to diagnose a CI failure falls from tens of minutes to a couple, because the second and third re-runs disappear entirely. It is one of the few changes to a test suite whose benefit is immediate and unambiguous, and it applies equally to a suite of fifty tests and one of fifty thousand.
Keeping assertion helpers worth having
Shared assertion helpers are a good idea that goes wrong in a predictable way, and registering them for rewriting only solves half of it.
The half that registration solves is diagnostics: a registered helper's assert reports its operands. The half it does not solve is location. A failure inside assert_valid_invoice reports a line in the helper, and if forty tests call it, the traceback tells you which test only through the frames above. pytest handles this with __tracebackhide__:
def assert_valid_invoice(invoice) -> None:
# Hides this frame from the traceback, so the failure points at the CALLER.
__tracebackhide__ = True
assert invoice.total == sum(line.amount for line in invoice.lines), (
f"invoice {invoice.id} total {invoice.total} != sum of lines"
)
With the flag set, the reported failure line is the test's call site rather than the helper's internals, which is where the reader wants to start. Without it, every failure in the suite points at the same three lines of helper code.
The second discipline is to keep helpers thin. A helper containing branching logic — "if the invoice is in a foreign currency, check differently" — is a second implementation of the behaviour under test, and when it disagrees with the code the failure is genuinely ambiguous. Helpers should assert facts, not compute expectations; anything that needs an if probably belongs in the test, where the reader can see it.
Frequently Asked Questions
Why do assertions in a helper module show no introspection?
pytest rewrites the AST of test modules and of plugins it knows about, not of arbitrary imported modules. A helper that lives outside those is imported unrewritten, so its asserts produce a bare AssertionError. Register the helper with pytest.register_assert_rewrite before it is first imported, usually at the top of conftest.py.
How do I customise the diff pytest prints for my own types?
Implement pytest_assertrepr_compare in conftest.py or a plugin. It receives the operator and both operands and returns a list of lines, which pytest prints in place of the default representation. Return None for comparisons you do not handle so other implementations still get a chance.
Does assertion rewriting slow down collection?
Only the first time. Rewritten modules are cached as .pyc files under __pycache__ keyed on the source's mtime and size, so subsequent runs load the rewritten bytecode directly. A cold CI runner pays the rewrite once; a runner with a warm cache pays nothing.
What is the difference between JUnit XML and pytest's own report hooks? JUnit XML is an output format most CI platforms parse to show a test list and failure messages. Report hooks are the mechanism that lets you observe or modify each test's result as it happens, which is how you attach artefacts, add properties to the XML, or send results elsewhere.
Can I make a failure include application logs automatically?
Yes, with a pytest_runtest_makereport hookwrapper that inspects the report and, when it failed, attaches captured output or files to the report's sections. Everything added there appears in the terminal output and in the XML, so CI shows it without anyone re-running the test.
Related guides
- Extend the diff for domain types in customizing failure output with assertrepr_compare.
- Wire the results into CI with producing JUnit XML reports for CI dashboards.
- Learn the hookwrapper mechanics in writing a hookwrapper for test reports.
- Make captured logs part of the failure using capturing logs with caplog and log_cli.
- Collect the artefacts a failure references with capturing artifacts from a failed CI test run.