Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Pytest & CI

Combining Coverage Across a Python Version Matrix

A library tested on Python 3.10 through 3.13, on Linux and Windows, runs its suite eight times and produces eight coverage reports that disagree. The 3.10 job never executes the branch guarded by sys.version_info >= (3, 11); the Linux jobs never run the Windows path handling. Each report looks worse than the truth, and a threshold enforced on any single one of them either fails constantly or has been lowered until it means nothing.

The usual workarounds make things worse. Lowering the threshold until the weakest job passes weakens the gate for everyone; marking version-specific code with # pragma: no cover hides it from the one job that does exercise it; picking a single "canonical" job to enforce on quietly stops measuring every other platform. The fix is to treat coverage as data to merge rather than a number each job reports. Every job writes its raw data file, a final job combines them, and the threshold is enforced once on the combined result — which is the only number that describes what the suite as a whole actually exercises.

Prerequisites

Solution

TOML
# pyproject.toml
[tool.coverage.run]
branch = true
parallel = true                 # .coverage.<host>.<pid>.<random> per process
source = ["myapp"]

[tool.coverage.paths]
# Every spelling of the same source tree, canonical form first.
source = [
    "src/myapp",
    "*/site-packages/myapp",    # when testing the installed wheel
    "*\\myapp",                 # Windows runners
    "/home/runner/work/*/src/myapp",
]

[tool.coverage.report]
fail_under = 90                 # enforced on the COMBINED report only
show_missing = true
Bash
# In every matrix job: run tests, keep the raw data, no threshold here.
pytest --cov --cov-report= -q
# upload .coverage.* as an artefact

# In the final job, after all matrix jobs:
coverage combine artefacts/            # merges and remaps paths
coverage report                        # applies fail_under once
coverage html -d htmlcov               # one browsable report
Matrix jobs feeding one combined report Four matrix jobs for Python 3.10 and 3.13 on Linux and Windows each write a raw coverage data file without enforcing a threshold. A final job downloads all four, remaps their differing source paths to one canonical tree, combines them, and enforces a single threshold on the merged result. Each job writes data; one job judges it 3.10 · linux — 84% 3.13 · linux — 86% 3.10 · windows — 81% 3.13 · windows — 83% coverage combine paths remapped combined: 93% fail_under enforced here
Every individual job is below the 90% threshold and the combined report is above it. Enforcing per job would fail all four for code that is, in fact, tested.

Why this works

parallel = true makes every process write its own data file with a unique suffix instead of overwriting a shared .coverage. Those files contain raw arcs keyed by file path, not percentages, so they can be merged later without loss. coverage combine reads them all, uses [tool.coverage.paths] to decide which differently-spelled paths refer to the same source file, and unions the recorded arcs into one data set.

The percentage computed from that union is the one that reflects reality: a line is covered if any job executed it. Version-specific code is covered by the job running that version, platform-specific code by the job on that platform, and the report no longer penalises any single job for code it was never meant to run.

Edge cases and failure modes

  • Paths that do not match. Combine silently keeps unmatched paths as separate files, so the report lists the same module twice with low coverage each. Add every path variant to [tool.coverage.paths].
  • Threshold in every job. --cov-fail-under in the matrix jobs fails them for version-specific code. Remove it there; enforce only after combining.
  • A failed matrix job. Its data file is missing, so combined coverage drops. Make the combine job depend on all matrix jobs succeeding, or it reports misleading numbers.
  • Subprocess coverage. Code run in subprocesses writes its own data only if coverage is started in them. Set COVERAGE_PROCESS_START or use pytest-cov, which handles it.
  • Stale data files in the workspace. A leftover .coverage.* from a previous run gets combined too. Clean the directory before downloading artefacts.

Verifying the merge actually merged

A combine step that silently fails to merge is worse than none, because it produces a report that looks authoritative and understates coverage — or, if paths are mismatched the other way, double-counts files and hides gaps. Two quick checks after coverage combine catch both.

First, list the files in the combined data and look for duplicates. coverage report --format=total gives the headline, but coverage debug data lists every measured file path; any module appearing under two spellings means a path variant is missing from [tool.coverage.paths]. The fix is one more entry in that list, and the check takes seconds.

Second, confirm the combined number is at least as high as the highest single job. Combining can only add covered arcs, never remove them, so a combined total below any individual job's figure means some data was dropped — a missing artefact, a job that failed before uploading, or a stale file from a previous run overwriting a fresh one.

Scripting both as assertions in the combine job turns them from things someone might check into things the pipeline enforces. A combine job that fails when a path variant is unmatched is annoying exactly once, on the day a new runner type is added, and saves the confusion of a coverage drop nobody can explain.

Two checks on the combined data After combining, two assertions run. The first checks that no module appears under two different path spellings, which would indicate a missing path mapping. The second checks that the combined percentage is at least the highest individual job's percentage, since merging can only add coverage. Trust the merged number only after checking it no duplicate modules src/myapp/pricing.py and D:\a\app\src\myapp\pricing.py means a path variant is missing add it to [tool.coverage.paths] combined ≥ best single job merging only adds arcs a lower total means lost data missing artefact, failed upload, or a stale file overwriting
Both checks are a few lines of shell in the combine job and fail loudly on exactly the misconfigurations that otherwise produce a silently wrong report.

Exclusions for code that only runs somewhere

Some code genuinely runs on only one platform or version and is tested there. Other code is defensive and runs nowhere under test. The two need different treatment, and conflating them either hides real gaps or penalises correct code.

Platform- and version-specific branches should be left in the measurement. The combined report covers them, because some job runs each one. Excluding them with a pragma removes a real check: if the Windows job stops executing the Windows path — because a skip was added, say — the combined report should notice.

Code that is unreachable under test — a fallback for a Python version the project no longer tests, a branch for an interpreter it does not support — is a legitimate exclusion, and coverage's exclude_also setting expresses it once rather than as a pragma on every line.

TOML
[tool.coverage.report]
exclude_also = [
    "if sys.version_info < \\(3, 10\\):",     # unsupported versions: never under test
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
]

The distinction to keep is whether some job in the matrix should execute the code. If yes, leave it measured and let the combined report hold the suite to it. If no job ever will, exclude it explicitly and document why in the pattern.

Measure or exclude, by whether any job runs the code Two kinds of conditional code. Platform- and version-specific branches that some matrix job executes stay measured, so the combined report notices if that job stops running them. Code for unsupported versions or type-checking only, which no job executes, is excluded once in configuration. Does any job in the matrix run it? yes — keep it measured if sys.platform == "win32": if sys.version_info >= (3, 12): combined report covers it and notices if the job stops no — exclude once unsupported-version fallbacks if TYPE_CHECKING: exclude_also in config documented by the pattern
Excluding the left column would make the combined report blind to a job quietly skipping its platform-specific tests.

Combining is also how pull-request gating works

The same machinery answers a question that looks unrelated: what coverage should a pull request be held to when its matrix jobs run in parallel? The answer is the combined figure, computed by a final job that waits for every matrix entry, and it is also the right input for diff coverage — the check that asks whether the lines a pull request changed are covered.

Diff coverage computed from a single job has the same flaw as a per-job threshold. A change that adds Windows-specific handling looks uncovered in the Linux job, and the check fails a correctly tested change. Computed from the combined data, it sees the Windows job's arcs and passes. Feeding the combined .coverage file into the diff-coverage step, rather than any one job's, is therefore not an optimisation but a correctness requirement for any project whose matrix exercises different code on different jobs.

That single final job ends up owning three responsibilities: combining, enforcing the global threshold, and enforcing diff coverage. Keeping all three there means there is exactly one place in the pipeline where coverage is judged, which makes failures easy to find and the configuration easy to reason about. The matrix jobs stay simple — run the tests, keep the data — and every decision about whether coverage is adequate is made once, on complete information, by the job that is able to see all of it.

Frequently Asked Questions

Why does each matrix job report lower coverage than the combined total? Because version- and platform-specific branches only run on their own job. Code guarded by sys.version_info or sys.platform is uncovered everywhere except where the condition holds, so each job alone looks incomplete. The combined report is the one that reflects what the suite actually exercises.

Why does coverage combine fail to merge files from different runners? Usually because the source paths differ — /home/runner/work/app/src on Linux, D:\a\app\src on Windows, a site-packages path when testing the installed package. The [tool.coverage.paths] setting tells coverage which paths are equivalent.

Where should the threshold be enforced? Once, on the combined report, in a final job that runs after all matrix jobs finish. Enforcing it per job fails the jobs whose version-specific code runs elsewhere.

← Back to Coverage Measurement and Enforcement