Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Pytest & CI

Customizing Failure Output with assertrepr_compare

A failing comparison between two domain objects prints their repr() on either side of ==. For a small dataclass that is fine. For an Invoice with fifteen fields, three nested line items and a Money value object, it is two walls of text in which the one differing field has to be found by eye. pytest_assertrepr_compare lets the suite replace that with a list of exactly what differed.

Prerequisites

  • pytest >= 8.0; the hook has existed for a long time and its signature is stable.
  • A root conftest.py, or a plugin shipped with the package, to host it.
  • Familiarity with how assertion rewriting produces explanations, from assertion introspection and test reporting.

Solution

Python
# conftest.py
from dataclasses import fields, is_dataclass

from myapp.billing import Invoice


def pytest_assertrepr_compare(config, op, left, right):
    # Narrow: only == between two Invoices. Everything else keeps pytest's
    # own explanation, including its dict/list/str diffs.
    if op != "==" or not (isinstance(left, Invoice) and isinstance(right, Invoice)):
        return None

    verbose = config.getoption("verbose") > 0
    lines = [f"Invoice {left.id!r} differs from {right.id!r}:"]
    for field in fields(Invoice):
        a, b = getattr(left, field.name), getattr(right, field.name)
        if a != b:
            lines.append(f"  {field.name}: {a!r} != {b!r}")
        elif verbose:
            lines.append(f"  {field.name}: {a!r} (same)")
    return lines
Plain text
E   AssertionError: assert Invoice(id='inv_1', …) == Invoice(id='inv_1', …)
E     Invoice 'inv_1' differs from 'inv_1':
E       total_minor: 1230 != 1234
E       status: 'open' != 'paid'
Default repr output versus a field-level diff Two failure reports for the same comparison. The default prints both objects' full representations, fifteen fields each, with the two differing fields buried among thirteen identical ones. The custom hook prints only the two differing fields with both values, so the cause is visible immediately. Same failure, very different time to diagnosis default: two reprs 2 differences among 30 printed fields hook: only what differs total_minor: 1230 != 1234 status: 'open' != 'paid' 2 lines, both relevant
The hook changes nothing about whether the test fails. It changes how long it takes a person to see why.

Why this works

When a rewritten assert left == right fails, pytest calls every registered pytest_assertrepr_compare implementation with the operator and both operands, and uses the first non-None result as the explanation. Returning None is therefore not just polite — it is the mechanism by which your hook coexists with pytest's own implementation, which handles dicts, sequences, sets, strings and dataclasses with a built-in diff.

The hook runs only on failure, so it adds no cost to passing tests, and it runs inside pytest's reporting, so its output appears in the terminal, in --tb styles, and in JUnit XML without further wiring.

Edge cases and failure modes

  • Returning a list unconditionally. Every assertion in the suite loses pytest's default explanation. Guard on the operator and the types.
  • An exception inside the hook. pytest reports it as an internal error during failure reporting, which obscures the original failure. Keep the hook simple and defensive.
  • Subclasses. isinstance matches subclasses, which is usually right; if a subclass has extra fields, iterate over fields(type(left)) rather than the base class.
  • Expensive repr calls. Formatting a field that is itself a large object reintroduces the wall of text. Truncate or summarise nested collections.
  • The hook in a subdirectory conftest.py. It applies only beneath that directory, so the same comparison elsewhere falls back to the default. Place it at the root or in a plugin.

Handling nested values without flooding the output

A field-level diff is only readable while each field's value is short. The moment a differing field is itself a list of line items or a nested object, printing both values reintroduces the wall of text one level down. Recursing selectively keeps the output proportionate to the difference.

Python
from dataclasses import fields, is_dataclass


def _diff(path: str, a, b, out: list[str], depth: int = 0) -> None:
    if a == b:
        return
    if is_dataclass(a) and type(a) is type(b) and depth < 3:
        for field in fields(a):
            _diff(f"{path}.{field.name}", getattr(a, field.name),
                  getattr(b, field.name), out, depth + 1)
    elif isinstance(a, (list, tuple)) and isinstance(b, (list, tuple)) and depth < 3:
        if len(a) != len(b):
            out.append(f"  {path}: length {len(a)} != {len(b)}")
        for i, (x, y) in enumerate(zip(a, b)):
            _diff(f"{path}[{i}]", x, y, out, depth + 1)
    else:
        out.append(f"  {path}: {_short(a)} != {_short(b)}")


def _short(value, limit: int = 80) -> str:
    text = repr(value)
    return text if len(text) <= limit else text[: limit - 1] + "…"
Plain text
E     Invoice 'inv_1' differs from 'inv_1':
E       .lines[2].unit_minor: 250 != 275
E       .lines: length 3 != 4

Paths such as .lines[2].unit_minor locate the difference precisely, and the depth limit plus truncation guarantee the output stays short even for pathological objects.

Recursing into nested values to locate the difference An invoice contains a list of three line items. The recursive diff walks from the invoice into the lines list and into the third item, reporting a single path, lines index two unit_minor, with both values. A depth limit and truncation keep the output short for deeply nested or very large values. Report a path, not two trees Invoice .lines [2] .unit_minor 250 != 275 Depth limit 3, values truncated at 80 characters — short output whatever the object size.
The path doubles as a navigation aid: it tells the reader which part of the fixture or factory call to look at.

Deciding which types deserve a hook

Not every class needs custom comparison output, and adding hooks indiscriminately creates a maintenance burden of its own. Three questions identify the types where the effort pays back.

The first is how often the type appears in assertions. A value object compared in two hundred tests — money, an address, a date range — repays a hook many times over, because every one of those failures becomes readable. A class compared in three tests does not justify the code.

The second is how large its default representation is. Dataclasses with a handful of short fields already produce readable failures, and pytest's built-in dataclass diff handles them well since version 5. The case for a hook grows with the number of fields, the depth of nesting, and the presence of fields whose repr is long — timestamps with microseconds, UUIDs, nested collections.

The third is whether the type has a notion of equality that differs from field-by-field comparison. A Money type that considers Money(100, "GBP") equal to Money(1.00, "GBP", unit="major") needs a hook that explains the comparison in its own terms, because a field-level diff would show differences where the type sees none.

When the answer to any of the three is strongly yes, the hook is worth writing. When all three are weak, a good __repr__ on the class delivers most of the benefit with none of the machinery — and improves every log line, debugger session and traceback that mentions the object, not just test failures.

Three signals that a type deserves a comparison hook Three cards. A type compared in many tests repays a hook many times over. A type with a large or deeply nested representation produces unreadable default failures. A type with its own notion of equality needs a hook that explains comparison in its own terms. When none apply, a good repr is the cheaper improvement. Write the hook where one of these is strongly true compared often money, addresses, date ranges hundreds of failures made readable at once large repr many fields, nesting, long timestamps the diff is the only way to find the field custom equality equal despite differing fields, or the reverse explain in the type's own terms
Where none of the three applies, invest in __repr__ instead; it improves logs and tracebacks as well as test failures.

Testing the hook itself

A comparison hook is code that runs only when something else has already failed, which makes it easy to break without noticing. A broken hook is worse than none: pytest reports an internal error during failure reporting, and the original assertion's explanation is lost at exactly the moment someone needed it.

The pytester fixture makes the hook testable in the ordinary way. Write a small test file that performs a failing comparison, run it in a subprocess-like sandbox, and assert on the lines the hook should have produced. Two cases are worth covering: a comparison the hook handles, whose output should contain the field names and values, and a comparison it does not handle — two plain dictionaries, say — whose output should still contain pytest's own diff. The second case is the one that catches a hook accidentally returning a list for everything, and it is the regression most likely to slip through review because nothing about it looks wrong in the hook's own code. Both checks together take a few seconds to run and are covered in detail in testing a pytest plugin with the pytester fixture.

Shipping comparison output with a library

When the domain types live in a package other teams depend on, their tests benefit from the same output — and they should not have to copy the hook into their own conftest.py. A pytest plugin registered through an entry point delivers it automatically to anyone who installs the package with its testing extra.

TOML
# pyproject.toml of the library
[project.optional-dependencies]
testing = ["pytest>=8"]

[project.entry-points.pytest11]
myapp_compare = "myapp.testing.pytest_plugin"
Python
# myapp/testing/pytest_plugin.py
from myapp.testing.compare import pytest_assertrepr_compare  # noqa: F401  (re-export)

Every consumer who installs myapp[testing] now gets field-level diffs for Invoice comparisons with no configuration, and improvements to the diff ship with the library's normal releases. The mechanics of entry-point registration, including how to disable a plugin with -p no:myapp_compare, are covered in packaging a pytest plugin with entry points. It is a small addition with an outsized effect on how pleasant a domain library is to test against, and one of the clearest signals that its authors expect it to be used in tested code.

Frequently Asked Questions

Where does pytest_assertrepr_compare have to live? In a conftest.py or a registered plugin, because it is a hook. A conftest.py in a subdirectory applies only to tests collected beneath it, so domain-wide comparison output belongs in the root conftest or in a plugin the package ships.

What happens if the hook returns a list for every comparison? It replaces pytest's own explanation for every assertion, including the excellent built-in diffs for dicts, lists, sets and strings. Always return None for comparisons you do not specifically handle.

Can I use it for operators other than ==? Yes. The hook receives the operator as a string, so it can explain !=, <, in and others. In practice == covers nearly all the value, and a comparison hook for < on a domain type is usually a sign the type should define its own ordering explanation.

← Back to Assertion Introspection & Test Reporting