Using tmp_path Instead of tempfile in Tests

tempfile.mkdtemp() in a test works until something goes wrong. Then the cleanup line never runs, the directory accumulates in /tmp, and — because the path was reused or guessable — two parallel workers write to it at once and produce a failure that only appears with -n. tmp_path solves all three problems and costs one argument.

Prerequisites

• pytest 7.0+ (tmp_path and tmp_path_factory return pathlib.Path).
• The wider isolation picture in faking the filesystem and environment.

Solution

Take tmp_path as an argument and build whatever the code needs inside it:

import json

def test_writes_report_to_the_target_directory(tmp_path):
    source = tmp_path / "input.json"
    source.write_text(json.dumps({"rows": [1, 2, 3]}), encoding="utf-8")
    out_dir = tmp_path / "out"
    out_dir.mkdir()

    write_report(source, out_dir)

    report = out_dir / "report.csv"
    assert report.exists()
    assert report.read_text(encoding="utf-8").startswith("row,value")

There is no cleanup code, no try/finally, and no shared path. Each test gets a directory named after itself under a per-run base directory, so two tests cannot collide and neither can two xdist workers.

For a fixture wider than function scope — a database seeded from a file, a checkout of a sample repository — tmp_path cannot be used, because it is function-scoped. Its factory sibling is the correct answer, and the reason ScopeMismatch appears when you reach for the wrong one:

import pytest

@pytest.fixture(scope="session")
def sample_repo(tmp_path_factory):
    root = tmp_path_factory.mktemp("repo")        # session-scoped directory
    (root / "README.md").write_text("# sample\n")
    clone_fixtures_into(root)                     # expensive, done once
    return root

mktemp returns a fresh directory each call, with a numeric suffix, so a session fixture can hand out several without collision.

Why this works

pytest creates one base temporary directory per run — /tmp/pytest-of-<user>/pytest-<n>/ — and derives every tmp_path from it using the test's name, truncated and de-duplicated. Under pytest-xdist the worker id is part of that base, so paths are unique across processes by construction. At the end of a run pytest removes base directories older than the last three, which is what makes the files inspectable after a failure without letting them accumulate forever. Because the fixture returns a pathlib.Path, tests compose paths with / and never build platform-specific separators by hand.

Inspecting what a failing test wrote

The retained directories turn a class of opaque failure into a two-command investigation. When an assertion about file contents fails, the files are still there:

$ pytest tests/test_reports.py -q
FAILED tests/test_reports.py::test_writes_report - assert 'row,value' in ''
...
$ ls /tmp/pytest-of-$USER/pytest-current/test_writes_report0/
input.json  out/
$ cat /tmp/pytest-of-$USER/pytest-current/test_writes_report0/out/report.csv

pytest-current is a symlink to the most recent run, which makes the path stable enough to use in a shell alias. In CI, point the base elsewhere and collect it as an artefact so the same inspection is possible after the fact:

$ pytest --basetemp=build/pytest-tmp -q

Note that --basetemp is wiped at the start of each run, so it must be a directory you do not otherwise use — passing a project directory there deletes its contents.

Retention is configurable when the defaults do not suit. tmp_path_retention_count sets how many run directories are kept, and tmp_path_retention_policy chooses between keeping all of them, only failed ones, or none:

[tool.pytest.ini_options]
tmp_path_retention_count = 3
tmp_path_retention_policy = "failed"     # keep only what you might want to read

Migrating an existing suite

The conversion is mechanical and worth doing in one pass, because a suite with both patterns keeps the failure modes of the worse one.

Search for the three shapes: tempfile.mkdtemp, tempfile.NamedTemporaryFile and any string path containing /tmp/. Each converts differently. mkdtemp becomes tmp_path directly. NamedTemporaryFile usually becomes a named file inside tmp_path, which also fixes the Windows behaviour where an open NamedTemporaryFile cannot be reopened by another handle. Hardcoded paths become tmp_path / "name" and their cleanup code disappears.

# Before: manual, leaks on failure, collides under -n
import tempfile, shutil

def test_old():
    d = tempfile.mkdtemp()
    try:
        ...
    finally:
        shutil.rmtree(d)

# After
def test_new(tmp_path):
    ...

Two things to watch during the migration. A fixture that was module- or session-scoped and used mkdtemp must move to tmp_path_factory, not tmp_path, or collection fails with ScopeMismatch. And code that passed a string path into the system under test may now receive a Path; that is usually an improvement, but a function doing path + "/sub" will raise, and the fix belongs in the production code rather than in str() calls at the call site.

Working with paths the code did not expect

tmp_path hands the test a directory, but the code under test may want something else: a file object, a string path, a path that does not exist yet, or a directory with specific permissions. Each has a short idiom.

import os
import stat

def test_missing_file_is_reported(tmp_path):
    missing = tmp_path / "nope.toml"          # a path that deliberately does not exist
    assert load_config(missing) is None

def test_unreadable_file_raises(tmp_path):
    path = tmp_path / "secret.toml"
    path.write_text("x = 1\n")
    path.chmod(stat.S_IWUSR)                  # write-only: reading must fail
    with pytest.raises(PermissionError):
        load_config(path)

def test_accepts_a_string_path(tmp_path):
    path = tmp_path / "app.toml"
    path.write_text("x = 1\n")
    assert load_config(str(path)) is not None  # the API accepts str as well as Path

def test_accepts_an_open_file(tmp_path):
    path = tmp_path / "app.toml"
    path.write_text("x = 1\n")
    with path.open(encoding="utf-8") as fh:
        assert parse_config(fh) is not None    # separate the reading from the parsing

The last example is worth generalising. A function that takes an open file object rather than a path is testable with io.StringIO and needs no filesystem at all — which is the cheapest possible test and the strongest argument for splitting "find and open the file" from "parse its contents".

Permissions deserve one caution: a test running as root ignores the mode bits entirely, so the PermissionError case passes locally and silently does nothing in a container that runs as root. Skip such tests when os.geteuid() == 0 rather than letting them pass vacuously.

Edge cases and failure modes

• Long test names are truncated. Directory names are shortened to fit filesystem limits, so two similarly named parametrized tests get numeric suffixes rather than descriptive paths.
• tmp_path is not empty of parents. It is nested inside a per-run directory, so a test that walks upwards from it will find pytest's own structure.
• Symlinks and case sensitivity follow the real filesystem. A test that passes on Linux may fail on a case-insensitive macOS volume; pyfakefs is one way to normalise that, at the cost of realism.
• --basetemp is deleted at the start of the run. Pointing it at an existing directory destroys the contents; always use a dedicated path.
• Files left open at teardown block removal on Windows. Close handles explicitly, or the cleanup of the retained directory fails with a permission error several runs later.
• A session-scoped directory is shared, including its dirt. Tests that write into a tmp_path_factory tree must not assume it is empty; give each test a subdirectory if isolation matters. One consequence of retention worth planning for in CI: --basetemp accumulates the files from every test in the run, and a suite that writes large fixtures can produce an artefact bundle of hundreds of megabytes. Either point --basetemp at a path that is only collected on failure, or prune it before upload — keeping the directories for the ten failing tests rather than for all four thousand. The retention policy setting handles this natively when the pytest version supports it, and a find-based prune is the portable fallback.

Frequently Asked Questions

Does pytest delete tmp_path immediately after the test? No. pytest keeps the last three test-run directories and removes older ones, so the files from a failed run are still on disk for inspection. The path is printed in the failure output when the fixture is used.

Is tmp_path safe under pytest-xdist? Yes. The base temporary directory includes the worker id, so two workers never share a path. Hand-rolled paths under /tmp do collide, which is a common cause of parallel-only flakiness.

What is the difference between tmp_path and tmp_path_factory?tmp_path is function-scoped and gives one directory per test. tmp_path_factory is session-scoped and hands out directories on demand, which is what a session-scoped fixture must use to avoid a ScopeMismatch.

Can I control where the temporary directories are created? Yes, with --basetemp on the command line or the tmp_path_retention settings in the ini file. Use --basetemp in CI to place them on a fast volume and to collect them as artefacts. Does tmp_path work inside a container with a read-only root filesystem? Only if the base temporary directory is writable. Point --basetemp at a mounted volume or at /tmp when the root filesystem is read-only, and set the pytest cache directory alongside it — both default to locations that a hardened container image forbids writing to.

Related

• Faking the filesystem and environment — the full set of isolation tools and when each applies.
• Patching environment variables with monkeypatch.setenv — the other half of process-state isolation.
• Fixing ScopeMismatch errors in pytest — the error that sends you to tmp_path_factory.
• pytest-xdist vs pytest-parallel performance comparison — why per-worker paths matter.

← Back to Faking the Filesystem and Environment