Enforcing Diff Coverage on Pull Requests

A total coverage percentage answers a question nobody asked. On a mature codebase it moves by tenths, so it cannot say whether the change under review is tested — and because it is a single global number, the only way to satisfy it is to write tests somewhere, not necessarily where the change is. Diff coverage replaces it with the question a reviewer actually needs answered: of the lines this pull request adds or modifies, how many does the suite execute?

Prerequisites

pytest-cov 4.1+ producing coverage.xml, and diff-cover 8.0+.
A CI checkout with enough git history to compute a merge base (fetch-depth: 0 on GitHub Actions).
A correct combined report, especially under parallel runs — see why pytest-cov reports 0% under xdist.

Solution

The gate is two commands: produce the report, compare it to the diff.

$ pytest -n auto --cov --cov-report=xml -q
$ diff-cover coverage.xml \
    --compare-branch=origin/main \
    --fail-under=100 \
    --html-report diff-coverage.html

diff-cover computes the diff between the working tree and the merge base with --compare-branch, intersects the changed lines with the coverage data, and reports the percentage of changed lines that were executed. A threshold of 100 is realistic here in a way it never is for total coverage, because the denominator is only the lines in this change.

In GitHub Actions the whole gate is one job, with the checkout depth being the detail that most often breaks it:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0                      # merge base must exist locally
- run: pytest -n auto --cov --cov-report=xml -q
- run: |
    diff-cover coverage.xml \
      --compare-branch=origin/${{ github.base_ref }} \
      --fail-under=100 \
      --markdown-report diff-coverage.md
- if: always()
  run: cat diff-coverage.md >> $GITHUB_STEP_SUMMARY

The last step matters more than it looks. A gate that fails with a log line nobody reads produces frustration; the same gate that prints the uncovered lines into the pull request summary produces a fix. --markdown-report writes exactly that: a per-file list of line numbers the change touched and the tests did not.

Only the intersection matters: untouched legacy code cannot fail the gate, and new code cannot pass it untested.

Why this works

Coverage XML records, for every file, which lines executed. Git records, for every file, which lines the change added or modified. Diff coverage is the intersection of those two sets, expressed as a percentage of the second. Because the denominator is scoped to the change, the metric is stable regardless of the size or age of the codebase, and it attributes the result to the person who can act on it. It also degrades gracefully: a change that touches no Python lines yields no denominator and passes trivially, which is the correct behaviour for a documentation-only commit.

Choosing the comparison point

The compare branch is the setting that decides whether the report is trustworthy.

Comparing against the merge base — the commit where the branch diverged — reports exactly the author's changes. Comparing against the tip of main reports the author's changes plus everything merged since the branch was cut, so an unrelated colleague's uncovered line fails your build, and rebasing changes the result. diff-cover uses the merge base when given a branch name and the checkout has the history to compute it, which is why fetch-depth: 0 is not optional.

For a merge queue or a rebase-heavy workflow, compare against the target branch rather than a fixed name:

$ diff-cover coverage.xml --compare-branch="origin/${TARGET_BRANCH:-main}" --fail-under=100

Two further options tune what counts. --diff-range-notation '...' selects the three-dot range explicitly, which is the merge-base semantics spelled out. And --ignore-staged / --ignore-unstaged keep local runs meaningful when the working tree is dirty, so an engineer can run the same gate before pushing.

Only the first row answers the question the gate is supposed to ask, and the last row is the reason so many gates are silently green.

Living with the gate

A gate that cannot be satisfied honestly will be disabled, so the policy needs two escape hatches and one review rule.

The first hatch is exclusion of code that genuinely cannot be tested in CI: platform-specific branches, defensive assertions, if TYPE_CHECKING blocks. These belong in exclude_also in the coverage configuration, where they apply consistently, rather than in per-line pragmas added under deadline pressure.

The second is an explicit override — a label on the pull request that skips the gate, paired with a required comment explaining why. Making the override visible and attributable is what keeps it rare; making it impossible is what gets the gate deleted.

- name: Diff coverage
  if: "!contains(github.event.pull_request.labels.*.name, 'skip-diff-coverage')"
  run: diff-cover coverage.xml --compare-branch=origin/main --fail-under=100

The review rule is that uncovered lines are a review topic, not just a build failure. When the report lands in the pull request summary, a reviewer can see that the uncovered lines are an error branch nobody exercised and ask for a test — which is the outcome the gate exists to produce. A gate whose output only exists in a CI log produces re-runs instead of tests.

Finally, expect the first week to be noisy. Enabling diff coverage on an existing repository surfaces every untested error path in the code people are currently touching, and that is the point; run it as a warning for a sprint, publish the report, then turn on the failure once the team has seen what it asks for.

Edge cases and failure modes

Shallow checkouts pass vacuously. With no merge base the diff is empty, the gate reports 100% and proves nothing. Assert that the changed-line count is non-zero for code changes.
Generated files inflate the denominator. Exclude them in the coverage configuration and, if necessary, with --exclude on diff-cover; a regenerated protobuf module should not be a review topic.
Moved code counts as changed. A large refactor that relocates covered code still shows those lines as changed, and they remain covered — but a move that also renames a branch condition can legitimately fail. Read the report rather than assuming it is noise.
The report reflects the run that produced the XML. If the coverage job ran a subset of the suite, the gate measures that subset; keep the coverage run and the gate in the same job.
Merge commits confuse the range. Prefer squash or rebase merges, or pass an explicit three-dot range so the merge base is unambiguous.
A parallel run with a broken combine reports every changed line as uncovered. The gate then fails on correct code, which is the fastest way to discover the combine problem described in the sibling guide.

Reporting the gate so it changes behaviour

The difference between a gate that improves a codebase and one that annoys people is entirely in how the result is surfaced. Three presentation choices do most of that work.

Show the lines, not the percentage. "Diff coverage 82%" tells an author nothing actionable; a list of four file-and-line references tells them exactly what to write. Both --markdown-report and --html-report produce that list, and the markdown form can be posted directly into a pull request comment or a job summary.

Link the report to the code. diff-cover renders file paths and line numbers; a short post-processing step that turns them into repository links removes the last friction between reading the report and fixing it.

Report on success too, quietly. A gate that only speaks when it fails trains people to treat it as an obstacle. A single line — "diff coverage: 100% of 47 changed lines" — in the job summary makes the gate visible as a routine part of the review, and makes an unexpected zero-line result (the shallow-checkout failure mode) obvious rather than silent.

A report without the first row can be vacuously green; one without the second row cannot be acted on.

Frequently Asked Questions

Why gate on diff coverage instead of total coverage? Because the total is dominated by code nobody is changing, so it moves too slowly to give feedback and invites tests written to raise a number. Diff coverage asks one answerable question of the author: are the lines you just wrote exercised?

What compare branch should diff-cover use? The merge base with the target branch, not the tip. Comparing against the tip counts changes other people merged after the branch was cut, which makes the report unstable and blames the wrong author.

How should a pure refactor be handled? It usually passes unchanged, because moved lines that were covered before are still covered. When a refactor genuinely adds uncovered branches, the gate is correct to fail — the new branches are new behaviour.

Can diff coverage be required at 100%? Yes for most projects, provided the exclusion rules are honest. Reserve an explicit override label or a pragma comment for the rare legitimate case, and require a reason in the pull request.

Coverage measurement and enforcement — branch coverage and the ratchet this gate complements.
Why pytest-cov reports 0% under xdist — the combine problem that makes a diff gate fail on correct code.
Pytest markers for conditional test execution — keeping the coverage job's test selection honest.
Optimizing test discovery — keeping the gate fast enough to run on every pull request.

← Back to Coverage Measurement and Enforcement