Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Pytest & CI

Producing JUnit XML Reports for CI Dashboards

Every CI platform can read JUnit XML, and every CI platform shows a better or worse dashboard depending on what is in it. A report with stable names, context attached to each test and captured output on failure turns the dashboard into a diagnostic tool; a report with unstable parametrised ids and no context turns it into a list of names that changes every run.

The configuration needed is small and entirely declarative: a handful of options in pyproject.toml, one autouse fixture for context, and a discipline about parametrised ids that the suite should have anyway. The payoff compounds over time, because the report becomes a history — of which tests fail, under which conditions, and how long each takes — that nobody has to collect deliberately. This guide covers the settings, what a dashboard does with each part of the file, merging reports from sharded jobs, and the questions a consistent set of reports can answer that a single run never can.

Prerequisites

  • pytest >= 8.0. JUnit XML support is built in; no plugin is needed.
  • A CI system that ingests JUnit XML — nearly all of them do.
  • Stable parametrised ids, covered in generating readable test IDs.

Solution

TOML
# pyproject.toml
[tool.pytest.ini_options]
addopts = "--junitxml=reports/junit.xml"
junit_family = "xunit2"            # what current parsers expect; pin it
junit_suite_name = "billing"       # a meaningful suite name in the dashboard
junit_logging = "all"              # stdout, stderr and log records on failure
junit_log_passing_tests = false    # keep passing tests' output out of the file
junit_duration_report = "call"     # durations exclude fixture setup
Python
# conftest.py — context stamped on every test case
import os
import platform

import pytest


@pytest.fixture(autouse=True)
def _report_context(record_property):
    record_property("commit", os.environ.get("GIT_COMMIT", "local"))
    record_property("python", platform.python_version())
    record_property("worker", os.environ.get("PYTEST_XDIST_WORKER", "master"))
XML
<testcase classname="tests.test_invoices" name="test_total[gbp-standard-rate]" time="0.012">
  <properties>
    <property name="commit" value="9f31ac2"/>
    <property name="python" value="3.12.4"/>
    <property name="worker" value="gw3"/>
  </properties>
  <failure message="assert 1230 == 1234">…</failure>
  <system-out>INFO billing charge started customer=cus_1 …</system-out>
</testcase>
Anatomy of a useful JUnit test case A single testcase element broken into four parts. A stable name built from explicit parameter ids lets the dashboard track history. Properties carry the commit, interpreter and worker. The failure element carries the assertion message. System-out carries captured logs for the failing test only. Four parts, four questions answered name="test_total[gbp-standard-rate]" — which test, stable across runs properties: commit, python, worker — under what conditions failure message="assert 1230 == 1234" — what went wrong system-out: captured logs — what the code was doing at the time
Default output provides only the third part reliably. The configuration above supplies the other three for the cost of a few lines.

Why this works

pytest's JUnit writer turns each test report into a <testcase> element, using the node id for the name and class, the call-phase duration for time, and the failure or error representation for the child element. record_property and user_properties attach arbitrary key-value pairs as <property> children, which most dashboards display alongside the result and many can filter or group by.

The node id is also why stable names come for free for ordinary tests and need attention only for parametrised ones: the module path, class and function name are fixed by the source, while the bracketed parameter id is whatever pytest or the author generated.

junit_logging controls whether captured stdout, stderr and log records are copied into <system-out> and <system-err>. Setting it to all while leaving junit_log_passing_tests off keeps the file small — passing tests contribute no output — while guaranteeing that every failure carries the logs needed to diagnose it.

Edge cases and failure modes

  • Unstable parametrised names. Ids generated from repr of objects with addresses or unordered sets change every run, so the dashboard's history and flake detection break. Use explicit ids=.
  • xunit1 left as the family. Several parsers ignore <properties> in the legacy format. Pin xunit2.
  • Huge report files. junit_log_passing_tests = true on a large suite produces hundreds of megabytes. Keep it off.
  • Duration including setup. The default duration covers setup, so every test sharing a slow session fixture looks slow. junit_duration_report = "call" isolates the body.
  • Properties with non-string values. They are stringified, which is fine for numbers and awkward for dicts. Flatten them into separate properties.

What a dashboard does with the file

It helps to know what the consuming side does with each element, because that decides which parts of the report deserve care and which are decoration.

The name and classname attributes are the identity of a test. Every history view, flake detector, "tests that started failing in this build" list and trend chart keys on them. A dashboard cannot tell that test_total[obj0] today and test_total[obj1] tomorrow are the same test, so anything that makes names vary between runs silently disables all of those features at once. That makes stable naming the single most important property of the report, well ahead of anything else in it.

The time attribute drives the "slowest tests" views and any duration trend. It is only meaningful if it measures the same thing every run, which is why reporting the call phase alone is preferable: setup time depends on which test happened to trigger a session fixture, and that changes with ordering.

The <failure> and <error> elements are what most people read. The distinction between them is worth preserving rather than collapsing — a failure is an assertion that did not hold, an error is an exception in setup, teardown or the test body — because the two lead to different investigations. pytest keeps them apart; a custom report hook that converts errors into failures throws that information away.

The <properties> and <system-out> children are the context. Few dashboards chart them directly, but all of them display them next to a failing test, and that is where they earn their place: the reader of a failure sees the commit, the interpreter and the logs without leaving the page.

Which dashboard features depend on which report fields Four report fields mapped to the features that consume them. Stable names feed history, flake detection and new-failure lists. Call-phase durations feed slowest-test views and trends. Failure versus error elements feed triage. Properties and captured output feed the detail pane shown beside a failing test. Every feature keys on one field stable name call-phase time failure vs error properties + output history · flake detection · "newly failing" lists slowest tests · duration trends triage: broken assertion or broken setup? the detail pane beside a failing test
Unstable names disable the whole top row at once, which is why they matter more than any other detail in the file.

Merging reports from sharded jobs

When a suite is split across several CI jobs, each job writes its own XML file, and the dashboard needs to see them as one run. Most platforms handle this natively if every shard uploads its report under a distinct name — the platform then aggregates by test name. Where that is not available, a final job can merge them.

The requirement that matters in either case is that test names are identical regardless of which shard ran the test. A name that includes the shard number, the worker id or anything derived from execution order breaks aggregation, because the same test appears under a different name depending on where it landed. This is another reason to put execution context in properties rather than in names: properties vary per run and are meant to, names must not.

Merging by hand is a small script over the standard library's XML parser, concatenating <testcase> elements under one <testsuite> and summing the counts. It is worth writing only when the platform cannot do it, and worth deleting as soon as the platform can.

Aggregating per-shard reports by stable test name Three shards each upload their own JUnit file. The dashboard aggregates them by test name into one run. Because names are stable and execution context lives in properties, the same test is recognised as the same test whichever shard ran it. Names identify tests; properties describe runs shard-1/junit.xml shard-2/junit.xml shard-3/junit.xml one run in the dashboard aggregated by test name shard id lives in a property, not the name
If a test's name depended on its shard, history and flake detection would treat it as a different test every time the shard assignment changed.

Using the report beyond pass and fail

A JUnit file with stable names and consistent properties is a small dataset, and a few simple questions asked of it across runs are worth more than any dashboard's default view.

Which tests fail intermittently? Group by name over the last fifty runs and count distinct outcomes; any test with both passes and failures on the same commit is flaky by definition, and the properties show whether the failures cluster on one worker, one interpreter or one shard. Which tests are getting slower? Plot the call duration per name over time; a steady climb is a regression in the code or its fixtures long before it becomes a timeout. Which failures correlate with a dependency? Filter by the version property and compare failure rates before and after an upgrade.

None of these need a specialised product. A scheduled job that parses the last few dozen reports into a table and flags the obvious anomalies costs an afternoon to write and changes how a team responds to flakiness — from re-running the build to reading a list of the specific tests that are intermittent, with the conditions under which they fail. That shift is the real return on getting the report's names and properties right, and it is only available if the data was recorded consistently from the start, which is why the configuration above is worth putting in place before it seems necessary rather than after the first flaky-test crisis.

Frequently Asked Questions

Which junit_family should I use?xunit2. It is the format current CI parsers expect, and the legacy xunit1 output drops per-test properties in several of them. Set it explicitly so a pytest upgrade cannot change it underneath you.

Does pytest-xdist produce one XML file or many? One. The controller collects results from every worker and writes a single report, so --junitxml works unchanged under -n. Sharding across separate CI jobs is different: each job writes its own file, and the dashboard or a merge step combines them.

Why do my test names change between runs in the dashboard? Usually because parametrised ids are generated from object reprs that include memory addresses or unordered data. The dashboard then sees a new test every run and cannot track history. Give parameters explicit ids.

← Back to Assertion Introspection & Test Reporting