Coverage is the most reported and least trusted number in a Python test suite. It is trusted too much when it is high — a suite can execute every line and assert nothing — and dismissed too easily when it drops for a reason nobody investigates. The value is not the percentage; it is the list of lines your tests never reached, and getting that list to be accurate takes more configuration than --cov alone.
Prerequisites
- pytest 7.0+, pytest-cov 4.1+, coverage 7.3+.
- A parallel run configured as described in pytest-xdist vs pytest-parallel if the suite uses workers.
Core concept: what coverage actually records
coverage.py installs a trace function that records which lines executed and, in branch mode, which line-to-line transitions occurred. That distinction is the whole difference between a number that means something and one that does not.
Line coverage marks a line as covered when it executed at all. Branch coverage records the arcs between lines, so an if without an else has two possible arcs — into the body and past it — and a test that only takes one of them leaves the other uncovered. On a typical codebase, switching from line to branch coverage drops the reported figure by five to fifteen points, and every point of that drop is a decision path no test exercises.
# pyproject.toml
[tool.coverage.run]
branch = true # measure arcs, not just lines
source = ["src/myapp"] # only our code; not site-packages, not tests
parallel = true # per-process data files, combined later
concurrency = ["multiprocessing"] # required when tests spawn subprocesses
omit = ["*/migrations/*", "*/_generated/*"]
[tool.coverage.report]
show_missing = true
skip_covered = true # keep the report to the lines that matter
exclude_also = [
"if TYPE_CHECKING:", # never executed at runtime
"raise NotImplementedError",
"if __name__ == .__main__.:",
]
source matters as much as branch. Without it, coverage measures whatever was imported, which pulls in dependencies and dilutes the number until it stops moving in response to your tests.
Step-by-step implementation
1. Let pytest-cov own the run. The single most common cause of a wrong number is starting coverage outside pytest. Use the plugin, not coverage run -m pytest, so the plugin can hook worker startup and combine the results:
$ pytest --cov --cov-report=term-missing --cov-report=xml -n 8
2. Add contexts so the report can answer "which test covered this". Contextual coverage records the test id alongside each line, which turns the HTML report into a navigable map of what exercises what:
[tool.coverage.run]
dynamic_context = "test_function"
3. Report what is missing, not what is present. --cov-report=term-missing with skip_covered = true prints only files with uncovered lines and the exact line numbers. That output is short enough to read in a pull request, which the full table never is.
4. Keep the data file out of the repository. Set data_file to a path under a build directory, and add it to .gitignore; a committed .coverage file from someone's laptop produces confusing combined reports.
Making the report answer real questions
A percentage answers one question badly. The same data answers four questions well, and each needs a different report.
Which lines are untested? term-missing with skip_covered prints file, percentage and the exact missing line numbers, and nothing about files that are already complete:
$ pytest --cov --cov-report=term-missing:skip-covered -q
Name Stmts Miss Branch BrPart Cover Missing
------------------------------------------------------------------------
src/myapp/billing/refund.py 84 11 26 4 83% 47-52, 91->95, 118
src/myapp/wire/decode.py 56 3 18 2 92% 77, 102->exit
The arrow notation is branch coverage speaking: 91->95 means the arc from line 91 to line 95 never ran — the false path of a conditional — and 102->exit means a loop never terminated normally in any test. Neither is visible in line coverage, and both are usually real gaps.
Which test covers this line? With dynamic_context = "test_function", the HTML report annotates every line with the tests that executed it. That turns two questions into lookups: "is this line covered by a test that actually asserts on it, or incidentally by an integration test three layers up", and "what will break if I change this function".
What changed? coverage json emits a machine-readable report that a CI job can diff against the previous run, which is how a ratchet is implemented without a third-party service.
What is not measured at all? Files with zero coverage are absent from some report formats entirely, which hides whole modules that no test imports. --cov-report=term includes them only when the files are inside source, which is another reason to set it explicitly.
[tool.coverage.report]
# Fail the report step rather than silently skipping unmeasured files.
fail_under = 0 # the ratchet lives in CI, not here
precision = 1
sort = "Cover"
Setting precision = 1 matters more than it looks: with integer precision, a change from 84.4% to 84.6% reads as no change, and a ratchet built on integers accumulates a full percentage point of silent regression before it triggers.
Verification
Three checks confirm the measurement is real before anyone starts enforcing it.
$ coverage debug data # how many data files were combined?
path: /build/.coverage
has_arcs: True
2 files:
gw0: 412 lines, 118 arcs
gw1: 388 lines, 104 arcs
$ coverage report --fail-under=0 -m # does the file list look right?
$ coverage html && open htmlcov/index.html
has_arcs: True proves branch mode is active. A single data file when eight workers ran proves the combine did not happen. And a file list that includes site-packages proves source is unset or too broad. All three are silent failures in the reported percentage, which is why they are worth checking once, at setup, and then never again.
Enforcement that resists gaming
An absolute threshold — "the build fails below 80%" — is easy to satisfy and easy to game. Two better rules, used together:
A ratchet on the total. Store the current percentage and fail when a pull request lowers it. The rule is directional rather than absolute, so a legacy codebase can improve gradually while new work cannot make things worse.
Full coverage on changed lines. diff-cover compares the coverage XML against the diff and reports only lines the pull request touched, which is the number a reviewer can act on. This is the subject of enforcing diff coverage on pull requests.
$ pytest --cov --cov-report=xml -q
$ diff-cover coverage.xml --compare-branch=origin/main --fail-under=100
Both rules share a property the absolute threshold lacks: they are about the change under review, so the feedback lands on the person who can act on it, while it is still cheap to act.
Coverage in the presence of subprocesses and threads
Two execution models break naive coverage measurement, and both are common in real suites.
Subprocesses. Code launched with subprocess.run or multiprocessing runs in a fresh interpreter with no trace function installed, so everything it executes is invisible. coverage.py solves this with a process-startup hook, enabled by an environment variable and a .pth file that the coverage package installs:
# conftest.py
import os
import pytest
@pytest.fixture(autouse=True, scope="session")
def _cover_subprocesses():
# COVERAGE_PROCESS_START points child interpreters at the same config file,
# and coverage's .pth hook starts measurement before the child's main module runs.
os.environ["COVERAGE_PROCESS_START"] = str(pathlib.Path("pyproject.toml").resolve())
yield
With parallel = true each child writes its own data file, and the combine step at the end picks them all up. Without the environment variable the children are simply not measured, and the missing lines look like untested code.
Threads. coverage.py traces threads automatically because the trace function is installed per thread by threading.settrace, but only for threads started after coverage begins. A thread started at import time — a background poller in a module-level singleton, for example — escapes measurement, and the fix is to start it lazily rather than to configure coverage differently.
Native extensions and generated code. Neither is traceable by coverage.py, which is a Python-level tool. Cython modules compiled with tracing enabled are the exception; everything else should be excluded from source so it does not appear as permanently uncovered and depress the number in a way nobody can act on.
Once these are configured, re-run the three verification checks above. The count of combined data files should equal the number of workers plus the number of subprocess groups, and any file you expected to see measured should appear in the report with a non-zero statement count.
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
0% reported under -n 8 | coverage started outside pytest; data files never combined | run through pytest --cov, set parallel = true |
| Percentage includes dependencies | source unset, so everything imported is measured | set source to your package |
| Coverage of subprocess code missing | subprocesses not traced | add concurrency = ["multiprocessing"] and sigterm = true |
| Number moves when tests are reordered | dynamic contexts writing to a shared file | give each worker its own data_file suffix |
if TYPE_CHECKING blocks reported uncovered | they never execute at runtime | list them in exclude_also |
What coverage cannot tell you
Every coverage number carries an implicit claim — "this code is tested" — that the measurement does not support. Being explicit about the gap prevents the two failure modes that follow from over-trusting it.
The first is the assertion-free test. Coverage records execution, so a test that calls a function and asserts nothing produces exactly the same coverage as one that verifies every output. A suite can reach ninety-five percent while proving almost nothing, and the number gives no hint. Mutation testing closes this gap by changing the code and checking that some test notices: mutmut and cosmic-ray are the usual tools, and both are slow enough to belong in a nightly job rather than in a pull-request gate. Running mutation testing on the ten files with the highest change frequency is a cheap approximation that finds most of the value.
The second is the coverage-driven test. When a threshold is the target, the cheapest way to reach it is a test that imports a module and executes its lines without asserting anything meaningful — and such tests are worse than no tests, because they cost maintenance and provide false confidence. This is precisely why the enforcement rules above are directional (do not regress) and scoped to the diff, rather than absolute.
There is also a category of code that should not be covered. Defensive branches for conditions that cannot occur in the test environment, platform-specific paths, and if TYPE_CHECKING blocks are all legitimately unexercised. Excluding them with exclude_also keeps the report honest, and each exclusion should be narrow enough that a reviewer can tell what was excluded and why. A broad # pragma: no cover on a whole class is where honest exclusion becomes hiding.
Used with those limits in mind, coverage is still the highest-value cheap signal a suite produces: it is the only automated way to discover code that no test touches at all, and that list — not the percentage — is what should reach a reviewer. Pair it with the property-based testing track for the inputs your example tests never generate, and with mutation-style scrutiny of the doubles that might be absorbing assertions before they reach real code.
Rolling coverage out on an existing codebase
Introducing coverage to a repository that has never measured it produces one predictable reaction — the number is lower than anyone expected — and one predictable mistake: setting a target and writing tests to reach it. A three-stage rollout avoids both.
Stage one: measure and publish, enforce nothing. Run the suite with coverage in CI, upload the HTML report as an artefact, and say nothing about the percentage for a sprint. The value in this stage is the list of files with zero coverage, which is almost always a surprise and almost always includes something important — an error-handling module, a serialization path, a migration helper.
Stage two: ratchet the total. Record the current figure and fail when it falls. This is a directional rule, so it can be enabled with no work and no debate, and it stops the situation getting worse while the team decides what to do about the existing gaps.
# A minimal ratchet, no third-party service required
$ coverage json -o coverage.json
$ python -c "
import json, sys
cur = json.load(open('coverage.json'))['totals']['percent_covered']
prev = float(open('.coverage-baseline').read())
print(f'{cur:.1f}% (baseline {prev:.1f}%)')
sys.exit(0 if cur >= prev - 0.05 else 1)
"
Stage three: require diff coverage. Once the ratchet has held for a few weeks, add the per-change gate, which is where the behaviour actually changes: new code arrives tested, and the total drifts upward on its own without anyone writing coverage-driven tests.
The order matters because each stage produces the information the next one needs. Skipping to stage three on a codebase with unmeasured subprocesses or a broken parallel combine means the gate fails on correct code, and the first thing the team learns about coverage is that it cannot be trusted.
Frequently Asked Questions
Is line coverage or branch coverage the right target?
Branch coverage. Line coverage counts a line as covered when any path through it runs, so an if statement with no else is fully covered by a test that never takes the false path. Branch coverage counts both outcomes and is the only measure that reflects decision coverage.
What coverage percentage should a project require? A ratchet rather than a number: require that coverage does not fall, and require full coverage on changed lines. An absolute target invites tests written to touch code rather than to assert behaviour.
Why does coverage drop when tests run in parallel?
Because each xdist worker writes its own data file and they must be combined. pytest-cov does that automatically when it owns the run; starting coverage outside pytest leaves the files uncombined and the report incomplete.
Does coverage measure test quality?
No. It measures which lines executed, not whether anything was asserted. A suite with no assertions can reach full coverage, which is why mutation testing exists as the complementary measure.
A closing note on tooling choice: everything above is standard coverage.py and pytest-cov, with no hosted service required. Services add pull-request annotations and historical charts, which are genuinely useful at scale, but none of them fix a broken measurement — a service fed by an uncombined parallel run displays the same wrong number more attractively. Get the local pipeline correct first, then decide whether the presentation layer is worth its cost.
Related guides
- Why pytest-cov reports 0% under xdist — the combine step, in detail, with the three configurations that break it.
- Enforcing diff coverage on pull requests — turning the report into a review-time gate.
- pytest-xdist vs pytest-parallel performance comparison — the parallel execution model coverage has to follow.
- Pytest configuration best practices — where the coverage settings belong alongside the rest.