Pytest & CI

Building Custom Pytest Plugins with pluggy

The recurring failure mode for home-grown pytest extensions is silent breakage. A conftest.py hook copied between repositories drifts out of sync; an unordered hookimpl clobbers another plugin's collection changes; an assertion helper imports before register_assert_rewrite runs and ships unreadable failure messages like assert False. The symptom is always the same — behaviour that works in one repository and quietly fails in the next, with no error to grep for. Turning pytest from a generic runner into a domain-specific testing framework means replacing those copied snippets with a versioned package built on pluggy hooks, entry-point discovery, and AST-based assertion rewriting. This guide walks the full lifecycle — scaffold, register, extend, rewrite, package, distribute — building on the runner internals covered in the advanced pytest architecture and configuration guide.

Prerequisites

  • Python 3.9+ and pytest 7.0+ (the pytester fixture replaced the older testdir).
  • pluggy 1.0+ (bundled with pytest) and a build + twine toolchain for distribution.
  • tox for multi-version validation.
  • Comfort with entry points and pyproject.toml packaging metadata (PEP 621).

Core concept

pluggy hook ordering A hookwrapper wraps a chain of tryfirst, default, and trylast implementations that pluggy dispatches in LIFO order. pluggy hook dispatch order hookwrapper: code before yield, then after wraps the entire chain below tryfirst runs early default impls LIFO by registration trylast runs late Discovery: pytest11 entry point installed plugins load before local conftest.py
pluggy calls implementations LIFO by registration, with tryfirst and trylast nudging position. A hookwrapper runs code on both sides of the yield, wrapping the whole chain.

pytest relies on pluggy, a lightweight plugin framework that decouples hook specifications (hookspec) from their implementations (hookimpl). When pytest initializes it constructs a plugin manager that registers every discovered plugin, resolves the hook chains, and executes them in a deterministic order. pluggy resolves hooks by name and calls implementations in reverse registration order (LIFO) by default, unless tryfirst=True or trylast=True nudges an implementation's position, or hookwrapper=True wraps the entire chain. This deterministic resolution is what lets a plugin safely intercept core pytest behaviour without monkeypatching internal modules — get the order wrong and you silently clobber another plugin's collection edits.

The lifecycle follows a strict sequence, and each phase exposes hooks a plugin can attach to:

  1. Initialization (pytest_configure): parse configuration, register custom markers, register modules for assertion rewriting, set up global state.
  2. Collection (pytest_collect_file, pytest_pycollect_makeitem): test modules are discovered, parsed, and converted into Item and Collector objects — the same phase covered in depth by optimizing test discovery.
  3. Setup and execution (pytest_runtest_setup, pytest_runtest_call): fixtures are resolved, tests execute, teardown runs.
  4. Reporting (pytest_terminal_summary, pytest_runtest_logreport): results are aggregated, formatted, and emitted to stdout and files.
pytest plugin lifecycle and hook attachment points Four sequential phases — configure, collection, setup and call, reporting — each exposing named hooks that a plugin implementation can attach to. Where a plugin attaches across the run 1 · Configure pytest_configure markers, global state, register_assert_rewrite 2 · Collection pytest_collect_file pytest_generate_tests build Items + Collectors 3 · Setup + call pytest_runtest_setup pytest_runtest_call resolve fixtures, run 4 · Reporting pytest_runtest_makereport pytest_terminal_summary aggregate + emit output A plugin implements hookimpls at any phase — pytest calls them in sequence. Within each hook, pluggy dispatches implementations by the tryfirst / trylast / hookwrapper order above.
The same four phases from the list above, shown as a pipeline. A plugin need not touch every phase — it attaches only the hooks it cares about, and pluggy orders competing implementations within each hook.

Step 1 — Scaffold the plugin package

A conftest.py is fine for rapid iteration inside one directory tree, but it lacks version control and cross-project portability. Once a hook needs to travel — across a monorepo or out to PyPI — promote it to a package. If your extension has simply outgrown a single file, managing conftest hierarchies covers the migration path in detail. A minimal, production-ready plugin separates hook logic from packaging metadata using a src layout:

Plain text
pytest-myplugin/
├── src/
│ └── pytest_myplugin/
│ ├── __init__.py
│ ├── plugin.py # Hook implementations
│ └── fixtures.py # Reusable fixture definitions
├── tests/
│ └── test_integration.py
└── pyproject.toml

plugin.py houses the hookimpl decorators, fixtures.py holds shared fixtures, and __init__.py stays empty so the import path (pytest_myplugin.plugin) matches the entry point you declare next. Keeping fixtures in their own module avoids import cycles when hooks and fixtures reference the same helpers.

Step 2 — Register discovery via entry points

Plugin discovery hinges on Python packaging entry points. When pytest boots it iterates through installed distributions and scans for the pytest11 namespace. Any package exposing an entry point under this namespace is automatically imported and registered with the plugin manager — no import line in a conftest.py required.

Configure discovery in pyproject.toml using PEP 621 standards:

TOML
[project.entry-points.pytest11]
myplugin = "pytest_myplugin.plugin"

This declarative registration ensures the plugin activates across any project where the package is installed, eliminating the need for manual conftest.py imports. However, precedence rules dictate behavior: installed plugins load before local conftest.py files, but conftest.py can override plugin fixtures and hooks within its directory tree. Use pytest --trace-config to audit the exact load order and verify registration:

Plain text
$ pytest --trace-config -q
PLUGIN registered: pytest_myplugin.plugin (from: /path/to/site-packages)
PLUGIN registered: _pytest.main (from: /path/to/site-packages/_pytest/main.py)
...

The precedence is worth internalizing: a locally installed package's hooks and fixtures can still be shadowed by a conftest.py deeper in the tree, which is exactly how a consuming project overrides a shared default without forking the plugin. If a fixture you expected to win is being silently replaced, --trace-config and the load order above are where you look first.

Step 3 — Expose fixtures and dynamic parametrization

Plugins frequently expose domain-specific fixtures that encapsulate complex setup logic. When defining fixtures inside a plugin, scope management and autouse behaviour need care to avoid unintended side effects or resource contention — the same scoping rules covered in mastering pytest fixtures, applied one level up so every consuming project inherits them. For plugin-driven test injection, lean on the generation patterns in advanced parametrization techniques.

A plugin-defined session-scoped fixture for database connection pooling:

Python
import pytest

@pytest.fixture(scope="session", autouse=True)
def db_pool(request):
    config = request.config.getoption("--db-url")
    pool = create_connection_pool(config)
    yield pool
    pool.close_all()

For dynamic test generation, the pytest_generate_tests hook intercepts collection and injects parameter sets before execution. This is particularly valuable for data-driven testing or matrix validation without modifying test signatures:

Python
def pytest_generate_tests(metafunc):
    if "env_config" in metafunc.fixturenames:
        # Load from plugin config or external matrix
        envs = metafunc.config.getoption("--env-matrix", default=["staging", "prod"])
        metafunc.parametrize("env_config", envs, scope="function")

Parallel Execution Considerations: When using pytest-xdist, session-scoped fixtures execute once per worker process. Plugins must implement thread-safe state management or use pytest-xdist's worker_id fixture to isolate resources. Global mutable state in plugins will cause race conditions and flaky failures in distributed runs.

Step 4 — Implement custom hooks and reporting

Hook execution order dictates how plugins interact with pytest's internal state. The following table outlines critical reporting and execution hooks:

HookPhasePurposeExecution Guarantee
pytest_runtest_protocolExecutionWraps setup/call/teardownCalled once per test item
pytest_runtest_makereportReportingModifies TestReport objectsInvoked after each phase
pytest_terminal_summaryPost-runAppends to CLI outputExecuted after all tests

pytest_terminal_summary runs after every test completes, giving a plugin the aggregated stats mapping to summarize. A practical implementation for emitting custom metrics:

Python
def pytest_terminal_summary(terminalreporter, exitstatus, config):
    terminalreporter.section("Custom Plugin Metrics", sep="=")
    passed = len(terminalreporter.stats.get("passed", []))
    failed = len(terminalreporter.stats.get("failed", []))
    terminalreporter.write_line(f"Total Passed: {passed}")
    terminalreporter.write_line(f"Total Failed: {failed}")

For granular execution control, pytest_runtest_protocol allows plugins to intercept the entire test lifecycle. However, overriding this hook requires explicit delegation to item.runtest() and proper exception handling to avoid breaking pytest's internal teardown pipeline:

Python
import pytest

@pytest.hookimpl(hookwrapper=True, tryfirst=True)
def pytest_runtest_protocol(item, nextitem):
    item.config.pluginmanager.get_plugin("capture").suspendcapture()
    try:
        # Custom pre-execution logic
        yield
    finally:
        item.config.pluginmanager.get_plugin("capture").resumecapture()

Pitfall: Failing to call item.runtest() or swallowing exceptions in pytest_runtest_protocol silently breaks test isolation and corrupts the runner's internal state machine. Always wrap custom logic in try/finally and delegate execution explicitly.

Step 5 — Register assertion rewriting for custom validation

Pytest's assertion rewriting is a compile-time AST transformation that enhances standard assert statements with rich introspection. When a plugin requires custom validation logic, it must register its modules for rewriting before they are imported, otherwise Python caches unrewritten bytecode and your helpers emit opaque assert failures.

Registration occurs during pytest_configure:

Python
def pytest_configure(config):
    import pytest
    pytest.register_assert_rewrite("pytest_myplugin.assertions")

The pytest_myplugin/assertions.py module can then define helpers that leverage pytest's internal assertion rewriting:

Python
def assert_json_schema_match(response, schema):
    """Custom assertion with detailed diff output."""
    errors = validate_schema(response, schema)
    assert not errors, f"Schema validation failed:\n{format_errors(errors)}"

Under the hood, pytest replaces assert expr with assert expr, "expr" and injects pytest_assertion_pass/pytest_assertion_fail hooks. When rewriting plugin modules, ensure the import hook is registered before any test imports the module. Otherwise, Python's standard import machinery will cache unrewritten bytecode in __pycache__, resulting in opaque assertion failures.

Debugging Tip: Run pytest --assert=plain to disable rewriting temporarily and verify baseline behavior. Use PYTHONVERBOSE=1 to trace import hooks and confirm that pytest._rewrite intercepts the target module.

Step 6 — Package, test with pytester, and distribute

Production plugins require rigorous integration testing before publication. The pytester fixture, provided by pytest-dev, creates isolated temporary environments, writes test files programmatically, and executes pytest subprocesses to validate output and exit codes.

A tox.ini configuration for multi-environment validation:

INI
[tox]
envlist = py39, py310, py311, py312, lint

[testenv]
deps = pytest>=7.0
commands = pytest tests/ -v

[testenv:lint]
deps = ruff, mypy
commands = ruff check src/ tests/
 mypy src/

Integration test using pytester:

Python
def test_plugin_registers_hook(pytester):
    pytester.makeconftest("""
import pytest

def pytest_configure(config):
    config.pluginmanager.register(MyPlugin())
""")
    result = pytester.runpytest("--help")
    result.stdout.fnmatch_lines(["*--myplugin-option*"])
    assert result.ret == 0

Publishing follows standard PyPI workflows: python -m build generates source and wheel distributions, while twine upload dist/* publishes them. Enforce semantic versioning and pin pytest compatibility in project.dependencies. Always test against the latest pytest minor release in CI to catch breaking changes in pluggy or internal APIs before users encounter them.

A worked example: network-call tracing plugin

The six steps combine cleanly into a VCR-style plugin that intercepts HTTP clients at the transport layer, caches responses, and replays them deterministically. This "cassette" pattern eliminates real network calls from CI while preserving realistic response payloads — the same recording idea explored from the client side in mocking network and HTTP calls, packaged here so every project inherits it for free.

The plugin registers itself in pytest_configure and installs the interception at session start:

Python
import pytest
import requests

def pytest_configure(config):
    config.pluginmanager.register(NetworkTracerPlugin(config))

class NetworkTracerPlugin:
    def __init__(self, config):
        self.config = config
        self.record_mode = config.getoption("--record-mode", default="replay")

    @pytest.hookimpl(tryfirst=True)
    def pytest_sessionstart(self, session):
        self._monkeypatch_requests()

    def _monkeypatch_requests(self):
        original_send = requests.Session.send

        def patched_send(request_self, request, *args, **kwargs):
            key = f"{request.method}:{request.url}"
            if self.record_mode == "record":
                resp = original_send(request_self, request, *args, **kwargs)
                save_cassette(key, resp)
                return resp
            return load_cassette(key)  # raises if the cassette is missing

        requests.Session.send = patched_send

This avoids modifying test code while providing deterministic network behaviour. In CI the plugin defaults to replay mode and fails fast when a cassette is missing, so a new unmocked call surfaces as a hard error rather than a live request. Thread safety is preserved by keying cassette I/O per worker process and using file locking for concurrent writes.

Trade-off: patching requests.Session.send intercepts everything layered on requests, but it does not touch clients that talk to the socket directly (httpx, aiohttp). Choosing the right interception layer is the same decision covered by patching builtins and sys.modules safely; for strict isolation, patch at the urllib3 connection-pool level instead.

Verification

Confirm the plugin is discovered, ordered correctly, and behaves as specified before you ship it:

Bash
# Confirm the plugin registered and inspect load order
pytest --trace-config -q

# Confirm your custom options/markers appear
pytest --help | grep myplugin

# Run the pytester-based integration suite
pytest tests/ -v

--trace-config prints a PLUGIN registered: line for your distribution; if it is missing, the pytest11 entry point is misspelled or the package is not installed in the active environment. To verify hook ordering, add a temporary print in each hookimpl and check the emission sequence matches your tryfirst/trylast intent. For assertion-rewriting helpers, run once with pytest --assert=plain and once without — the rich diff should appear only in the default mode, proving register_assert_rewrite fired before import.

Troubleshooting

SymptomRoot causeFix
Plugin never loadspytest11 entry point typo or package not installedReinstall with pip install -e . and verify the name in pytest --trace-config.
PluginValidationError at startuphookimpl signature does not match the hookspecMatch argument names exactly; drop unused parameters rather than renaming them.
Another plugin overrides your changesUndefined hook orderMark your impl trylast=True (to run after collection edits) or hookwrapper=True to wrap the chain.
Plain assertion messages from a helperHelper imported before register_assert_rewriteCall pytest.register_assert_rewrite("pkg.module") in pytest_configure, before any import.
pytester tests pass locally, fail in CIWorker isolation differences under pytest-xdistAvoid global mutable state; key resources by the worker_id fixture.

Testing the plugin itself

A plugin is code, and the code that shapes every other test deserves tests of its own. pytest ships the tooling for this in the pytester fixture: it runs a throwaway pytest session inside your test, against files you write on the fly, and hands back the result to assert on.

Enable it by declaring the plugin in your test suite's conftest.py, then drive a complete session in a few lines:

Python
# conftest.py in the plugin's own test suite
pytest_plugins = ["pytester"]
Python
def test_marker_skips_when_flag_absent(pytester):
    pytester.makeconftest('pytest_plugins = ["myplugin.hooks"]')
    pytester.makepyfile(
        test_slow="""
        import pytest

        @pytest.mark.slow
        def test_one():
            assert True
        """
    )
    result = pytester.runpytest()                  # a real session, in a temp dir
    result.assert_outcomes(skipped=1)              # the plugin's rule fired
    result.stdout.fnmatch_lines(["*slow tests disabled*"])   # and reported why

assert_outcomes is the assertion that matters: it reads the terminal summary rather than internal state, so it proves the behaviour a user would see. fnmatch_lines pins the reporting, which is worth doing for any plugin that explains itself in output — a silent skip is indistinguishable from a bug.

Three habits keep plugin tests useful as the plugin grows. Run each scenario in its own pytester session rather than accumulating flags in one, because hook registration order differs between a fresh session and a mutated one. Use pytester.runpytest_subprocess() when the plugin touches global state such as sys.modules, logging handlers, or the working directory — an in-process run shares those with your own session and can pass for the wrong reason. And assert on exit codes for the paths that must fail: result.ret == pytest.ExitCode.TESTS_FAILED proves a hook that is supposed to fail the run actually does.

The hook you are testing determines what you can observe. Collection hooks show up in result.parseoutcomes() counts, reporting hooks in stdout, and pytest_runtest_makereport wrappers in the terminal summary sections. Where a hook only mutates in-memory objects, expose the effect through a marker or a report attribute so a test can see it — a hook whose only evidence is internal is a hook nobody can safely change later.

What a pytester session actually runs A sequence diagram with three lanes: the plugin test, the pytester fixture, and the inner pytest session. The test writes a conftest and a test file, pytester launches a full inner session in a temporary directory, the inner session loads the plugin and runs, and the outcome counts and stdout come back for assertion. What a pytester session actually runs plugin test pytester inner session makepyfile / makeconftest runpytest() in a temp dir plugin hooks fire outcomes + stdout assert_outcomes() Use runpytest_subprocess() when the plugin mutates process-global state.
The inner session is a real pytest run, so the assertion is about observable behaviour rather than internal plugin state.

One packaging note that saves an afternoon: while iterating, install the plugin with pip install -e . so the entry point is registered and every session picks up your edits without a reinstall. Running the same code through -p myplugin.hooks instead exercises a different registration path, and a plugin that works under -p but not when installed has almost always declared its pytest11 entry point incorrectly.

Keep the plugin's own test suite out of the package distribution but inside the repository: the tests need pytester, which pulls in pytest's internals, and shipping them makes the wheel heavier for no user benefit.

Frequently Asked Questions

How do I test a custom pytest plugin without installing it globally? Use the pytester fixture provided by pytest-dev. It creates an isolated temporary environment, writes test files, and runs pytest programmatically to assert expected output and exit codes.

What is the difference between conftest.py and a distributed plugin?conftest.py is local to a directory tree and auto-discovered, while distributed plugins are registered via entry points and activated across projects. Plugins are preferred for reusable, versioned extensions.

Can a custom plugin modify test parametrization dynamically? Yes, via the pytest_generate_tests hook. It intercepts collection and injects parameter sets before execution, enabling data-driven testing without modifying test functions directly.

How does pytest handle assertion rewriting in plugins? Plugins register modules for rewriting using pytest.register_assert_rewrite(). pytest intercepts import hooks, compiles AST with enhanced failure introspection, and caches bytecode for subsequent runs.

← Back to Advanced Pytest Architecture & Configuration