Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Pytest & CI

Generating Cases with pytest_generate_tests

@pytest.mark.parametrize covers the common case: a fixed list of inputs, written next to the test that uses them. It cannot express cases that are not known when the test is written — one per JSON file in a fixtures directory, a set selected by a command-line option, a matrix that depends on which optional dependencies are installed. pytest_generate_tests is the hook for that. It runs during collection, sees each test function's requested arguments, and can parametrize them with anything computable at that moment.

The hook is powerful enough to become a maintenance hazard, because it runs for every test function in scope and silently shapes what gets collected. Used with a guard on the argument name, cheap loading and stable ids, it is the cleanest way to turn a directory of data files into a readable set of test cases.

Prerequisites

  • pytest >= 8.0.
  • The static parametrization techniques in advanced parametrization techniques.
  • Case data in a form cheap to enumerate — a directory of files, a small manifest, a configuration value.

Solution

Generate one test case per file in a fixtures directory, guarded so only tests that ask for it are affected.

Python
# tests/parsers/conftest.py
import json
from functools import lru_cache
from pathlib import Path

CASES_DIR = Path(__file__).parent / "cases"


@lru_cache(maxsize=None)
def _load_cases():
    # Loaded once per session, not once per test function.
    return sorted(CASES_DIR.glob("*.json"))


def pytest_generate_tests(metafunc):
    # Guard: only tests that request `case_file` are parametrized.
    if "case_file" not in metafunc.fixturenames:
        return
    files = _load_cases()
    metafunc.parametrize(
        "case_file",
        files,
        ids=[f.stem for f in files],     # stable, readable: the file name
    )
Python
# tests/parsers/test_parser.py
import json


def test_parser_matches_expected_output(case_file, parser):
    case = json.loads(case_file.read_text())
    assert parser.parse(case["input"]) == case["expected"]
Plain text
tests/parsers/test_parser.py::test_parser_matches_expected_output[empty-object] PASSED
tests/parsers/test_parser.py::test_parser_matches_expected_output[nested-arrays] PASSED
tests/parsers/test_parser.py::test_parser_matches_expected_output[unicode-keys] FAILED

Adding a case is now adding a file. No test code changes, and the new case appears in the report under its file name.

From a directory of files to named test items During collection, pytest calls the hook for each test function. The hook checks whether the function requests case_file, loads the cached list of JSON files, and parametrizes the argument with one value per file and the file stem as the id. Collection then produces one test item per file, each named after its file. Cases are data; the hook turns data into items cases/ empty-object.json nested-arrays.json unicode-keys.json pytest_generate_tests requests case_file? → yes load cached file list parametrize with ids=stem collected items test_parser[empty-object] test_parser[nested-arrays] test_parser[unicode-keys] Adding a case means adding a file; the test body never changes.
The ids come from the file names, so the report says which case failed in the same words the fixtures directory uses.

Why this works

During collection pytest builds a Metafunc object for every test function — describing its name, module, requested arguments and markers — and calls every pytest_generate_tests implementation in scope with it. A call to metafunc.parametrize inside the hook has exactly the same effect as the decorator would have had, generating one item per value and appending the id to the name.

Because the hook sees metafunc.fixturenames, it can decide per test whether to act. That is what makes the guard work: tests that do not mention case_file pass through untouched, so the hook can live in a conftest.py without affecting unrelated tests beneath it.

Edge cases and failure modes

  • No guard. A hook that parametrizes unconditionally breaks every test in scope that does not accept the argument. Always check metafunc.fixturenames first.
  • Expensive loading. The hook runs once per test function, including during pytest -k one_unrelated_test. Cache the loaded data at module level.
  • Unstable ids. Ids built from dictionary ordering or object reprs change between runs. Derive them from file names or explicit keys.
  • An empty case list. Parametrizing with an empty list produces a single skipped item with a reason, which is easy to miss. Assert the directory is non-empty in the hook, or accept the skip deliberately.
  • Combining with the decorator. The hook and @pytest.mark.parametrize can target different arguments of the same test and produce a product. Targeting the same argument twice raises.

Letting the command line choose the cases

The second common use of the hook is selecting cases from configuration rather than from files. A suite that tests against several external services, several data sets or several optional backends often wants to run all of them nightly and only one on a developer's machine. A command-line option read inside the hook expresses that without duplicating any test.

Python
# conftest.py
def pytest_addoption(parser):
    parser.addoption("--backend", action="append", default=[],
                     help="backend to test against (repeatable); default: memory")


def pytest_generate_tests(metafunc):
    if "backend" not in metafunc.fixturenames:
        return
    chosen = metafunc.config.getoption("backend") or ["memory"]
    metafunc.parametrize("backend", chosen, ids=chosen)
Bash
pytest -q                                          # memory only, fast
pytest -q --backend memory --backend sql --backend redis   # the full matrix

This is the one situation where dynamic parametrization is clearly better than fixture params. A parametrized fixture always produces every variant; the hook can produce whichever variants the invocation asked for, so the same test file serves the fast local loop and the thorough nightly run without a single conditional in the tests.

One test file, different matrices per invocation The same test file collected under two invocations. With no options, the hook parametrizes with the memory backend only, producing a fast local run. With three backend options, the hook produces three items per test for the nightly job. No test code differs between the two. The invocation decides the breadth pytest test_save[memory] test_load[memory] 2 items · seconds the developer's loop pytest --backend × 3 test_save[memory|sql|redis] test_load[memory|sql|redis] 6 items · minutes the nightly job
A parametrized fixture cannot do this: it always produces every variant. The hook produces exactly what was asked for.

Collection cost, and how to keep it flat

Because the hook runs during collection, its cost is paid on every invocation — including the ones that select a single unrelated test with -k. A hook that parses a hundred JSON files on each call, in a conftest.py covering three hundred test functions, parses thirty thousand files before any test runs. The symptom is a suite that takes twenty seconds to start and a developer who stops running tests individually because "collection is slow".

Three habits keep it flat. The guard on metafunc.fixturenames means the hook does nothing for the functions that do not ask for the argument, which is usually most of them. Caching the loaded data at module level — functools.lru_cache on the loader, as above — means the files are read once per session rather than once per function. And enumerating cheaply before loading expensively means the ids can come from file names while the file contents are read only inside the test, where they are needed; sorted(CASES_DIR.glob("*.json")) touches only directory entries.

With all three in place, the hook's collection cost is a directory listing, and the test's own cost includes reading exactly one file. That split is worth preserving as the case count grows: a thousand cases should cost a thousand small reads spread across the run, not a thousand reads concentrated at the start of every invocation.

Where the hook's work happens Two arrangements. In the naive hook, every call parses every case file during collection, so selecting one unrelated test still pays for all of them. In the disciplined hook, collection only lists file names from a cached directory scan, and each test reads its own file when it runs. List during collection; read during the test naive hook every call parses every file no guard, no cache -k one_test still pays for thousands of parses disciplined hook guard on the argument name cached directory listing collection: one listing each test reads its own file
The ids come from names, the data comes from contents, and only the second needs to be read by the test that uses it.

Keeping the hook discoverable

The hook's weakness is that it is invisible from the test: a reader of test_parser_matches_expected_output(case_file, parser) sees an argument that is neither a fixture defined anywhere obvious nor a decorator on the function. Two conventions remove the mystery.

Put the hook in the conftest.py of the directory that uses it, never at the root, so the search space for "where does case_file come from" is one file. And give the argument a name that is obviously data rather than infrastructure — case_file, sample, fixture_document — so readers do not go looking for a fixture of that name. A one-line comment on the test pointing at the hook costs nothing and saves the next reader the search entirely, which is the right trade for a mechanism that is powerful precisely because it is indirect.

Golden files and updating expectations

Directory-driven cases pair naturally with golden files: each case stores an input and the expected output, and the test compares the two. The awkward moment comes when the expected output legitimately changes — a formatter improves, a serializer adds a field — and dozens of golden files need updating.

A small option makes that deliberate rather than tedious. An --update-golden flag, read in the test or a fixture, rewrites the expected output from the actual output instead of asserting, and the resulting diff is reviewed like any other change. The discipline that keeps this safe is that updating is never the default: a normal run always asserts, and regenerating expectations is an explicit act whose result a human reads in the pull request.

Without that flag teams tend to regenerate golden files with ad-hoc scripts, and the regenerated files are reviewed less carefully than they should be, because the script feels like tooling rather than a change in behaviour. Putting the update path inside the test suite, next to the comparison it replaces, keeps the two in step and makes the review of an expectation change look exactly like the review of the code change that caused it.

Frequently Asked Questions

When should I use pytest_generate_tests instead of @pytest.mark.parametrize? When the cases are not known when the test is written — they come from files on disk, a command-line option, an environment, or a computation that depends on configuration. For a fixed list written in the source, the decorator is clearer and should be preferred.

Where can pytest_generate_tests be defined? In a test module, where it applies to tests in that module, or in a conftest.py, where it applies to every test beneath it. In either place it is called once per test function during collection.

Does loading cases in the hook slow down collection? Yes, because it runs during collection for every test function in scope, including runs that select only one unrelated test. Keep the loading cheap, cache it at module level, and guard it so it only does work for tests that request the relevant argument.

← Back to Advanced Parametrization Techniques