Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Pytest & CI

Testing a pytest Plugin with the pytester Fixture

A pytest plugin — or the hooks and fixtures in a large conftest.py — is code that changes how other tests are collected, run and reported. Testing it with ordinary unit tests is awkward, because the behaviour only exists inside a pytest run. The pytester fixture solves that by letting a test create a small throwaway project in a temporary directory, run pytest on it, and assert on what happened: how many tests passed, what was printed, which items were collected.

It is the tool pytest itself is tested with, and it makes plugin behaviour as testable as any other function. That matters more than it first appears, because plugin and conftest.py code has an unusual failure profile: when it breaks, it rarely breaks loudly. A collection hook that stops skipping slow tests makes the suite slower rather than red; a report hook that stops attaching logs makes failures harder to diagnose rather than causing new ones; a fixture that stops cleaning up leaks resources that only become a problem weeks later. None of those produce a failing test on their own, which is exactly why they need tests of their own. The investment is small — a handful of tests per hook — and it pays back the first time a refactor changes a plugin's behaviour in a way that nothing else would have noticed.

Prerequisites

  • pytest >= 8.0; pytester ships with it and needs only enabling.
  • A plugin or conftest.py whose behaviour is worth pinning down — see building custom pytest plugins.

Solution

Enable the fixture once, then write each scenario as a generated project and a run.

Python
# tests/conftest.py
pytest_plugins = ["pytester"]
Python
# tests/test_run_slow_option.py
def test_slow_tests_are_skipped_by_default(pytester):
    pytester.makeconftest(open("conftest.py").read())      # the real hook under test
    pytester.makepyfile(
        """
        import pytest

        @pytest.mark.slow
        def test_heavy():
            pass

        def test_light():
            pass
        """
    )
    result = pytester.runpytest("-rs")
    result.assert_outcomes(passed=1, skipped=1)
    result.stdout.fnmatch_lines(["*needs --run-slow*"])


def test_run_slow_includes_them(pytester):
    pytester.makeconftest(open("conftest.py").read())
    pytester.makepyfile(
        """
        import pytest

        @pytest.mark.slow
        def test_heavy():
            pass
        """
    )
    result = pytester.runpytest("--run-slow")
    result.assert_outcomes(passed=1)
A pytester test runs pytest inside pytest The outer test creates a temporary project containing a conftest and a test file, runs an inner pytest session on it with chosen arguments, and receives a result object. It then asserts on the counts of outcomes and on lines of output, which is how the plugin's behaviour is verified without touching the real suite. A small project, a real run, assertions on the result outer test — test_slow_tests_are_skipped_by_default temp project conftest.py (hook) test_x.py (2 tests) inner pytest run runpytest("-rs") plugin active RunResult assert_outcomes(…) stdout.fnmatch_lines
The inner run is a genuine pytest session with the plugin active, so the assertions test exactly what users will experience.

Why this works

pytester creates an isolated temporary directory for each test and changes into it, so files written with makepyfile, makeconftest and makeini form a complete, self-contained project. runpytest then runs a full pytest session against that project — collection, hooks, fixtures, reporting — and returns a RunResult holding the exit code, the parsed outcome counts, and the captured stdout and stderr.

By default the inner run happens in-process, which is fast because it reuses the already-imported interpreter. The isolation is good enough for most plugins because pytester snapshots and restores sys.path, sys.modules and the working directory around each run. runpytest_subprocess trades that speed for complete isolation, launching a fresh interpreter, and is the right choice when a plugin mutates global state that the snapshot does not cover.

Edge cases and failure modes

  • Asserting on exact output. Terminal output changes between pytest versions and terminal widths. Use fnmatch_lines with wildcards, and assert on outcomes wherever possible.
  • Plugin not loaded in the inner run. A plugin registered through an entry point is active automatically; one living in a module is not. Pass -p myplugin to runpytest.
  • State leaking between in-process runs. A plugin that caches at module level or registers atexit handlers can leak. Switch that test to runpytest_subprocess.
  • Slow suites of plugin tests. Each inner run is a full session. Keep generated projects minimal — two or three tests each — and prefer many small scenarios to one large one.
  • Forgetting the negative case. Test that the plugin does not act when it should not — without its flag, on unmarked tests — as well as that it does.

What to test in a plugin

A plugin's surface is small but each part fails differently, and a short checklist covers it. Collection behaviour — which tests are selected, skipped, deselected or multiplied — is best tested with assert_outcomes and, where ids matter, with pytester.inline_run plus getreports to inspect item names. Fixtures the plugin provides are tested by generated tests that request them and assert on their values. Command-line options need the precedence checks described in adding command-line options with pytest_addoption. Report output — summary lines, headers, section titles — is tested with fnmatch_lines.

The negative cases deserve the same attention as the positive ones. A plugin that adds a skip marker to slow tests must be tested both with and without its flag, and with a test that is not marked slow at all, because the bug that ships is rarely "the feature does nothing" and usually "the feature does something to tests it should not touch". Three generated test files with three runs each cover that matrix in well under a second.

What to verify for each part of a plugin Four rows. Collection behaviour is verified with assert_outcomes and report inspection. Provided fixtures are verified by generated tests that request them. Command-line options are verified with precedence scenarios. Report output is verified with wildcard line matching. Each row includes a negative case where the plugin should not act. One assertion style per surface, plus the negative case collection → assert_outcomes · inline_run().getreports() for ids fixtures → a generated test that requests the fixture and asserts on it options → default, ini value, and flag, checked for precedence output → stdout.fnmatch_lines with wildcards, never exact matches
Every row should include a scenario where the plugin must leave the tests alone; that is where regressions in plugins usually hide.

Inspecting collection and reports directly

assert_outcomes answers "how many passed, failed and skipped", which covers most scenarios. Some plugin behaviour is about which items exist or what their reports contain, and pytester exposes that too.

pytester.inline_run(...) returns a HookRecorder rather than a RunResult. It records every hook call during the inner session, so a test can retrieve the collected items, the reports for each phase, and their attributes. That makes it possible to assert that a hook generated items with particular ids, that a report carries a user property the plugin was supposed to attach, or that a deselection hook removed exactly the items it should have — none of which appear in the pass/fail counts.

The trade-off is fidelity to the user's view. runpytest checks what a person running the suite would see; inline_run checks internal state that may not be visible at all. Both are useful, and a good plugin test file uses runpytest for the behaviour users rely on and inline_run for the internal contracts other plugins or tools depend on, such as report properties consumed by a CI dashboard.

A practical rule for choosing: if the assertion would make sense in the plugin's documentation — "tests marked slow are skipped with the reason 'needs --run-slow'" — use runpytest and assert on output. If it only makes sense to someone reading the plugin's code — "the report for each item carries a shard property" — use inline_run and assert on the recorded reports.

runpytest versus inline_run Two ways to run the inner session. runpytest returns outcome counts and captured output, suited to asserting on what a user sees. inline_run returns a hook recorder holding collected items and phase reports, suited to asserting on internal contracts such as report properties or generated ids. Assert on what users see, or on what tools consume runpytest → RunResult outcome counts, exit code captured stdout and stderr documented behaviour "slow tests are skipped with …" inline_run → HookRecorder collected items and their ids reports per phase, properties internal contracts "each report carries shard=…"
Most plugin tests belong on the left. The right-hand side is for guarantees other tools depend on but users never see directly.

Keeping plugin tests fast and readable

A plugin test file can easily become the slowest part of a suite, because every test runs a complete pytest session. Three habits keep it proportionate.

Keep each generated project tiny, and resist the urge to build one elaborate fixture project shared by many tests. Two or three generated tests are almost always enough to demonstrate a behaviour and its negative case; a scenario that needs twenty is usually two scenarios. The inner session's cost scales with what it collects, so small projects mean fast tests.

Share the conftest under test, not the generated projects. Reading the real hook from the repository once — as the examples above do with open("conftest.py").read() — ensures the tests exercise the code that actually ships, while each test still writes its own minimal test file so its scenario is readable on its own.

Name tests after the behaviour, not the mechanism. test_slow_tests_are_skipped_by_default and test_run_slow_includes_them read as a specification of the option; test_plugin_1 and test_plugin_2 tell a reader nothing when one of them fails. Plugin tests are often the only documentation of a suite's custom behaviour, and naming them as statements about that behaviour makes them serve that role without any extra effort. When a new engineer asks what --run-slow does, the answer is a list of test names, and it is guaranteed to be current because the build fails when it is not.

Frequently Asked Questions

How do I enable the pytester fixture? Add pytest_plugins = ["pytester"] to the root conftest.py, or pass -p pytester on the command line. It ships with pytest but is disabled by default because most suites do not test pytest itself.

Should I use runpytest or runpytest_subprocess?runpytest runs in-process and is much faster, which is right for most plugin tests. runpytest_subprocess runs in a separate interpreter and is needed when the plugin changes global state that would leak between tests, or when testing command-line entry-point behaviour exactly.

Can pytester test a conftest.py rather than a packaged plugin? Yes. pytester.makeconftest writes a conftest.py into the temporary directory, so hooks and fixtures that live in a project's conftest can be copied or imported there and tested in isolation from the real suite.

← Back to Building Custom pytest Plugins