A suite of four thousand tests that takes six minutes is fine in CI and painful in an edit-run loop, where the developer changed one function and wants to know within seconds whether anything broke. pytest-testmon narrows the run to the tests that could have been affected, using coverage data from previous runs to know which tests executed which lines. Change a function, and only the tests that ran that function are selected.
It is a genuinely effective tool for local feedback and a risky one as a sole CI gate. The speed-up is not marginal: on a typical service, most edits touch code exercised by a few percent of the suite, so the selected run is one or two orders of magnitude faster than the full one. That changes behaviour — developers who can get an answer in five seconds run the tests after every change rather than once before pushing, and that alone catches a class of mistakes much earlier. Knowing precisely what it can and cannot see — Python lines, yes; configuration, data files and dependency versions, no — is what lets a team take the speed without the blind spots.
Prerequisites
pytest-testmon >= 2.1andpytest >= 8.0.coverageinstalled; testmon uses it to record which lines each test executes.- A suite without heavy cross-test state, since selection assumes a test's outcome depends only on its own recorded code.
Solution
Seed the database once with a full run, then let each subsequent run select.
# 1. Full run: records every test's executed lines into .testmondata.
pytest --testmon -q
4127 passed in 358.2s
# 2. Edit one function in src/myapp/pricing.py, then:
pytest --testmon -q
testmon: changed files: src/myapp/pricing.py, skipping collection of 211 files
38 passed, 4089 deselected in 6.4s
Thirty-eight tests executed that function; those are the ones that ran. Six seconds instead of six minutes is the difference between running tests after every edit and running them once before pushing.
Why this works
During a --testmon run, the plugin uses coverage to trace which code blocks each test executes, and stores a fingerprint of those blocks per test in a SQLite database. On a later run it fingerprints the current source, compares, and marks as affected every test whose recorded blocks have changed. Unchanged tests are deselected, and whole files whose tests are all unaffected are skipped at collection, which is where most of the time saving comes from.
The fingerprinting is block-level rather than file-level, so editing one function in a large module selects only the tests that executed that function — not every test that imported the module. That granularity is what makes the selections small enough to be useful.
Edge cases and failure modes
- Non-Python inputs. A change to a YAML fixture, a SQL migration or a template affects tests without changing any traced Python line. testmon does not see it; run the relevant tests explicitly or the full suite.
- Dependency upgrades. A new version of a library changes behaviour with no change to your source. Invalidate the database, or run everything, whenever the lockfile changes.
- Tests with cross-test state. If test B depends on state left by test A, running B alone because only its code changed can produce a failure — or hide one. testmon exposes order dependence; fix it rather than working around it.
- Coverage overhead on the recording run. Tracing slows the full run noticeably. Record on a schedule or on merge rather than on every CI job.
- A stale database in CI. Without the cached
.testmondata, every CI run selects everything. Restore it from the main branch's last run.
Using it in CI without losing safety
In CI, testmon is most valuable on pull requests, where the change is small and feedback speed matters, and least safe as the final word on whether a change can merge. The arrangement that captures the benefit and bounds the risk has three parts.
On merges to the main branch, run the full suite with --testmon, which both verifies everything and refreshes the database. Cache the resulting .testmondata keyed on the main branch's commit.
On pull requests, restore that cache and run with --testmon. The selection is computed against main, so the run executes exactly the tests the pull request's changes could affect. Most pull requests finish in a fraction of the full suite's time.
Before merge — in the merge queue, or as a required nightly — run the full suite without selection. That catches everything testmon cannot see: configuration changes, dependency bumps, data files, and interactions that coverage tracing misses.
Covering the inputs testmon cannot see
The blind spots are predictable, which means they can be covered by rule rather than left to chance. Three categories account for nearly every case where testmon's selection is wrong.
Files the tests read at runtime — JSON fixtures, SQL files, templates, YAML configuration. A change to one affects every test that loads it, but no Python line changed. The simplest cover is a CI rule: if any file under tests/data/ or config/ changed, skip selection and run everything. testmon's own configuration can also be told about additional dependency files, which is more precise where it is supported.
The dependency set — a lockfile change can alter behaviour anywhere. Treat any change to poetry.lock, uv.lock or requirements*.txt as a reason to run the full suite and rebuild the database.
The environment — interpreter version, environment variables, feature flags set outside the code. These rarely change within a pull request, but when a CI image is updated the database recorded under the old image is no longer trustworthy. Keying the cache on the image version as well as the branch removes that risk.
The rules are cheap to implement because CI systems already know which files a change touched. A few lines in the pipeline definition that check the changed paths and choose between --testmon and a full run give the speed of selection for ordinary code changes and the safety of a full run for exactly the changes selection cannot reason about.
What selection reveals about a suite
An unexpected side effect of adopting testmon is that it exposes problems a full run hides. The most common is order dependence. A test that passes in the full run because an earlier test left the database, a cache or a module-level variable in a helpful state will fail when testmon selects it alone, and the failure looks like a regression in whatever code the developer just changed.
That is uncomfortable the first few times and valuable thereafter. Every such failure is a genuine isolation bug — the test was never self-contained, it was only ever passing by accident of ordering — and fixing it makes the suite more reliable under parallel execution and random ordering as well. Teams that adopt testmon typically find and fix a handful of these in the first week and then rarely see another.
The second thing selection reveals is how much of the suite exercises any given piece of code. A function whose change selects four hundred tests is either central to the system or tested far more redundantly than necessary, and the testmon output makes that visible at every edit. It is a rough but useful signal for where a suite has accumulated overlapping tests that could be consolidated, which is the kind of maintenance that otherwise never gets prioritised because nobody can see where the redundancy is.
Frequently Asked Questions
How does testmon know which tests a change affects? It records, for every test, which lines of which files executed while it ran, and stores the fingerprint in a local database. On the next run it compares the current source against those fingerprints and selects only the tests whose recorded code changed.
Is it safe to use testmon in CI as the only test run? Not as the only one. Coverage-based selection misses changes that affect behaviour without changing executed Python lines — configuration files, data files, dependency upgrades, environment variables. Use it for fast feedback and keep a full run on merge or nightly.
Where is the dependency data stored?
In a .testmondata SQLite file in the project root. Locally it persists between runs; in CI it must be cached and restored between jobs or every run starts from nothing and selects everything.
Related
- Optimizing Test Discovery — the collection costs testmon avoids for unaffected files.
- Sharding a Test Suite Across CI Runners — the complementary way to make the full run fast.
- Bisecting Test-Order Dependencies — the cross-test state that selection tends to expose.
- Enforcing Diff Coverage on Pull Requests — another use of per-change coverage data.
← Back to Optimizing Test Discovery