Faking the Filesystem and Environment in Tests

Tests that touch the filesystem or read the environment fail in the two ways that are hardest to debug: they pass alone and fail in a suite, or they pass on one machine and fail on another. Both symptoms have the same cause — process-global state that one test changed and another observed. The tools to fix it are all in pytest and the standard library, and choosing between them is mostly a question of how much of the filesystem the code under test insists on owning.

Prerequisites

pytest 7.0+ for tmp_path, tmp_path_factory and monkeypatch.
pyfakefs 5.3+ only for the whole-filesystem case in the last section.
Familiarity with fixture scope from mastering pytest fixtures.

Core concept: real files beat mocked ones

The instinct to patch("builtins.open") is almost always wrong. A mocked open returns whatever you told it to, which means the test never exercises encoding, newline translation, permissions, partial reads, or the difference between a missing file and an empty one — and those are where file-handling bugs live.

A real temporary directory costs microseconds and removes all of that guesswork:

def test_config_loader_handles_utf8(tmp_path):
    config = tmp_path / "app.toml"
    config.write_text('name = "café"\n', encoding="utf-8")   # a real file, real bytes

    loaded = load_config(config)

    assert loaded["name"] == "café"          # encoding actually verified

tmp_path is a pathlib.Path to a fresh directory, unique per test, removed automatically after a few runs are kept for inspection. tmp_path_factory is its session-scoped sibling for expensive shared trees, and it is the correct fix for the ScopeMismatch described in fixing ScopeMismatch errors in pytest.

Only the last row skips real file behaviour, which is exactly the behaviour most file-handling bugs depend on.

Step-by-step implementation

1. Give the test its own directory. Build whatever tree the code expects inside tmp_path, using pathlib rather than string paths so the test reads the same on every platform:

def test_walks_only_yaml(tmp_path):
    (tmp_path / "conf").mkdir()
    (tmp_path / "conf" / "a.yaml").write_text("x: 1\n")
    (tmp_path / "conf" / "b.json").write_text("{}")
    (tmp_path / "conf" / "nested").mkdir()
    (tmp_path / "conf" / "nested" / "c.yaml").write_text("y: 2\n")

    found = discover_configs(tmp_path / "conf")

    assert sorted(p.name for p in found) == ["a.yaml", "c.yaml"]

2. Set environment variables through monkeypatch. os.environ["X"] = "1" is never undone; monkeypatch.setenv records the previous value — including its absence — and restores it even when the test raises:

def test_reads_region_from_environment(monkeypatch):
    monkeypatch.setenv("APP_REGION", "eu-west-1")
    monkeypatch.delenv("APP_DEBUG", raising=False)     # absent is a valid state

    assert build_client().region == "eu-west-1"

3. Redirect the working directory rather than assuming it. Code that resolves relative paths depends on the process's current directory, which pytest does not control by default:

def test_writes_report_relative_to_cwd(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)               # restored at teardown
    write_report({"ok": True})
    assert (tmp_path / "report.json").exists()

4. Make paths injectable where you can. Every one of the techniques above is a workaround for a hardcoded dependency. A function that takes base: Path needs no monkeypatching at all, and the test reads as an ordinary call — the same argument made for collaborators in dependency injection for testability.

Each rung down the ladder removes a category of test scaffolding; the bottom rung needs none.

Replacing the whole filesystem with pyfakefs

Everything above redirects code that is willing to be redirected. Some code is not: a config loader that reads /etc/app/settings.toml, a library that writes to ~/.cache, a routine that walks / looking for a certificate bundle. Those paths cannot be pointed at tmp_path without changing the code, and sometimes the code belongs to a dependency.

pyfakefs replaces the filesystem layer itself. It patches the os, io, shutil and pathlib modules so every path operation resolves against an in-memory tree, and the real disk is untouched:

def test_reads_system_config(fs):           # `fs` is the pyfakefs fixture
    fs.create_file("/etc/app/settings.toml", contents='region = "eu-west-1"\n')

    settings = load_system_settings()        # reads the hardcoded /etc path

    assert settings["region"] == "eu-west-1"

Three properties make this different from the other tools. It is total: absolute paths, os.walk, shutil.copytree and pathlib all see the fake tree, so no path escapes it. It is fast: nothing touches a disk, so a test that creates ten thousand files runs in milliseconds. And it is hermetic: a test cannot accidentally read a developer's real ~/.aws/credentials, which is a genuine safety property when the code under test looks for credentials.

The trade-offs are equally real. Native extensions that open files through C bypass the patched modules entirely and see an empty directory, which produces confusing failures in code using lxml, numpy.load or a database driver. Anything that shells out to a subprocess is unaffected, because the child process has its own real filesystem. And a fake filesystem does not reproduce permission semantics, filesystem-specific case sensitivity, or the behaviour of a full disk unless you configure it to.

def test_handles_permission_error(fs):
    fs.create_file("/etc/app/settings.toml", contents="")
    fs.chmod("/etc/app/settings.toml", 0o000)      # simulate an unreadable file

    with pytest.raises(PermissionError):
        load_system_settings()

Reach for pyfakefs when the paths are unredirectable, and prefer tmp_path everywhere else. The rule of thumb: if you can pass a path in, do; if the code insists on an absolute path you cannot change, fake the filesystem; if the code shells out, neither will help and the test belongs at the integration level.

Prefer the leftmost branch that works: each step right replaces more of the real system with a simulation.

Environment isolation beyond variables

os.environ is the obvious process-global state, but it is not the only one a test can dirty. Four others leak just as easily and are worth handling in the same autouse fixture.

The working directory, as above. sys.path, which a test that inserts a directory must remove — monkeypatch.syspath_prepend does both. Locale and timezone, which change number and date formatting process-wide; TZ in particular needs time.tzset() after the change on POSIX. And umask, which affects the permissions of every file created afterwards and has no restore helper at all.

import os
import pytest

@pytest.fixture(autouse=True)
def _clean_process_state(monkeypatch, tmp_path):
    monkeypatch.setenv("TZ", "UTC")
    if hasattr(os, "tzset"):
        os.tzset()                              # C-level clocks read TZ once
    monkeypatch.setenv("LC_ALL", "C.UTF-8")     # stable formatting and sorting
    monkeypatch.chdir(tmp_path)                 # never the repository root
    yield

Making the working directory a fresh temporary path for every test is the single highest-value line in that fixture. It turns "the test wrote a file into the repository" from an invisible side effect into an impossibility, and it exposes code that silently depended on being run from the project root.

Testing code that shells out

Subprocesses sit outside every isolation mechanism on this page. A child process gets a copy of the environment at spawn time, sees the real filesystem regardless of pyfakefs, and inherits the working directory — so a test that patches all three and then shells out is testing something quite different from what it configured.

Three techniques cover the realistic cases.

Pass the environment explicitly. subprocess.run(..., env={...}) replaces inheritance with an explicit mapping, which is both more testable and safer: the child receives exactly what you listed and nothing from the developer's shell.

import subprocess

def run_migration(dsn: str, *, env: dict[str, str] | None = None) -> None:
    subprocess.run(
        ["alembic", "upgrade", "head"],
        env={"DATABASE_URL": dsn, "PATH": "/usr/bin:/bin", **(env or {})},
        check=True,
    )

Give the child a real directory. Where the subprocess reads or writes files, tmp_path works normally — the child sees the same real filesystem, and the temporary directory is a real path. This is one place where pyfakefs actively hurts: the parent sees the fake tree and the child does not, so the two disagree about whether a file exists.

Fake the boundary, not the process. For a test about how your code reacts to a subprocess result, patch the runner rather than launching anything:

from unittest.mock import patch
import subprocess

def test_reports_migration_failure():
    failure = subprocess.CalledProcessError(returncode=1, cmd=["alembic"])
    with patch("myapp.migrate.subprocess.run", side_effect=failure):
        assert run_and_report() == "migration failed"

Reserve real subprocess execution for tests that are genuinely about the integration, mark them, and keep them out of the fast suite — a real process launch is milliseconds at best and seconds when it starts an interpreter.

The last two rows are why a test that patches Python objects and then shells out is testing two different worlds at once.

Verification

Two checks confirm the isolation is real rather than assumed.

$ pytest -p no:randomly tests/ -q          # baseline
$ pytest -p randomly tests/ -q             # shuffled order: state leaks now fail
$ git status --porcelain                   # must be empty after a test run

Running the suite in a randomised order is the standard way to surface order-dependence, and filesystem leakage is one of its most common causes. git status after a run catches the other half: a test that wrote into the working tree rather than into tmp_path.

For environment leakage specifically, a teardown assertion is cheaper than debugging the consequence:

import os
import pytest

@pytest.fixture(autouse=True)
def _no_env_leak():
    before = dict(os.environ)
    yield
    assert dict(os.environ) == before, "test modified os.environ without monkeypatch"

Fixtures that build realistic trees

Once several tests need the same directory structure, the setup wants to be a fixture — and a factory fixture is more useful than a fixed one, because tests differ in the details that matter to them.

import pytest
from pathlib import Path

@pytest.fixture
def project_tree(tmp_path):
    """Return a factory that builds a project skeleton with the given files."""
    def build(files: dict[str, str], root: str = "proj") -> Path:
        base = tmp_path / root
        for relative, content in files.items():
            path = base / relative
            path.parent.mkdir(parents=True, exist_ok=True)
            path.write_text(content, encoding="utf-8")
        return base
    return build

def test_finds_nested_configs(project_tree):
    root = project_tree({
        "pyproject.toml": "[tool.app]\n",
        "services/api/app.toml": "port = 8080\n",
        "services/api/.hidden.toml": "ignored = true\n",
    })

    assert [p.name for p in discover(root)] == ["app.toml", "pyproject.toml"]

The factory form keeps the fixture reusable while letting each test declare exactly the tree its assertion depends on — which also makes the test readable, because the input is visible next to the expectation rather than three files away.

Two refinements are worth adding as the suite grows. Return Path objects rather than strings, so tests compose them with / and never build platform-specific separators. And keep the content inline unless it is genuinely large; a test that references fixtures/big.json forces the reader to open another file to understand the assertion, and a test that inlines three lines of TOML does not.

For binary content and permissions, the same factory extends naturally with write_bytes and chmod. What it should not do is grow conditionals: a factory with a mode="broken" parameter is two fixtures wearing one name, and splitting them keeps each test's intent legible.

Troubleshooting

Symptom	Root cause	Fix
Test passes alone, fails in the suite	env var or cwd left over from an earlier test	use `monkeypatch`, add the leak assertion above
`FileNotFoundError` only in CI	relative path resolved against a different cwd	`monkeypatch.chdir(tmp_path)`, or inject the base path
Files appear in the repository after a run	code wrote relative to the working directory	make `tmp_path` the cwd for every test
Permissions differ between machines	umask or a `chmod` that is not reset	set umask explicitly in a fixture
Timestamp assertions fail seasonally	timezone read from the host, DST transition	pin `TZ=UTC` and call `time.tzset()`

An isolation fixture worth copying

Most of this guide collapses into one autouse fixture that every suite can adopt. It establishes a known process state, gives each test its own directory, and fails loudly when something escapes.

import os
import pytest

BASELINE_ENV = {"TZ": "UTC", "LC_ALL": "C.UTF-8", "PYTHONHASHSEED": "0"}
CLEARED_ENV = ("AWS_PROFILE", "APP_DEBUG", "HTTP_PROXY", "HTTPS_PROXY")

@pytest.fixture(autouse=True)
def isolated_process_state(monkeypatch, tmp_path):
    for name in CLEARED_ENV:
        monkeypatch.delenv(name, raising=False)      # nothing inherited from the shell
    for name, value in BASELINE_ENV.items():
        monkeypatch.setenv(name, value)
    if hasattr(os, "tzset"):
        os.tzset()                                    # C-level clocks re-read TZ
    monkeypatch.chdir(tmp_path)                       # never the repository root
    before = dict(os.environ)
    yield tmp_path
    leaked = {k for k in set(before) | set(os.environ)
              if before.get(k) != os.environ.get(k)}
    assert not leaked, f"environment leaked: {sorted(leaked)}"

Three properties make it worth adopting wholesale rather than piecemeal. It is complete: environment, clock configuration and working directory are all covered, so no test needs to remember any of them. It is fail-loud: the teardown assertion names the test that bypassed monkeypatch instead of letting a later test suffer for it. And it is cheap: a few dictionary operations and a directory creation per test, invisible against any real test's runtime.

Two adjustments are worth making per project. Extend CLEARED_ENV with the variables your developers actually export — an AWS_PROFILE or a DATABASE_URL in a shell profile is the classic source of "passes for me". And drop the chdir if a large legacy suite depends on running from the repository root, but treat that dependency as debt: it is exactly the coupling that makes a test write into the working tree.

With this in place, the remaining filesystem work in a suite is choosing between tmp_path and pyfakefs per test, which is the decision the rest of this guide covers.

Four lines of policy in one fixture replace a convention every test author would otherwise have to remember.

Frequently Asked Questions

Should I mock open() or use a real temporary directory? Use a real temporary directory. Real files exercise the encoding, permission and path handling that a mocked open() skips entirely, and tmp_path makes them free to create and automatically cleaned up.

Why do environment variables leak between tests? Because os.environ is process-global and a plain assignment is never undone. monkeypatch.setenv records the previous value and restores it at teardown, including when the test fails.

When is pyfakefs the right tool? When the code under test walks or writes across absolute paths you cannot redirect — a config loader reading /etc, or code that hardcodes a root. pyfakefs replaces the whole filesystem layer so those paths resolve into an in-memory tree.

How do I test code that depends on the current working directory? Change it with monkeypatch.chdir(tmp_path), which restores the original directory at teardown. Better still, refactor the code to take a base path, so the dependency is visible and injectable. A last word on scope. Every technique here isolates process state, and a test that also touches a database, a message broker or a shared cache needs the same discipline applied to those — a per-worker database name, a per-test transaction rollback, a namespaced key prefix. The principle transfers directly: state whose lifetime is longer than the test is state one test can leak into another, and the fix is always either a fresh instance or an explicit reset.

A short adoption order

Retrofitting isolation into an existing suite works best in this order, because each step makes the next one's failures interpretable.

First, add the teardown assertion for environment leakage without changing anything else. It will fail, and the failures are the list of tests to fix — a concrete backlog rather than a vague concern.

Second, replace tempfile with tmp_path everywhere. This is mechanical, cannot break behaviour, and removes the cleanup code that was silently skipped on failure.

Third, make tmp_path the working directory for every test. This will break the tests that depended on running from the repository root, and each break is a genuine coupling worth removing.

Fourth, pin the baseline environment. By now the suite is isolated enough that a timezone or locale change produces a comprehensible failure rather than a cascade.

Only then consider pyfakefs, and only for the paths that cannot be redirected. Teams that start there usually end up with a fake filesystem wrapped around code that never needed one, and with the subprocess and native-extension caveats permanently in their way.

Using tmp_path instead of tempfile in tests — the fixture, its factory sibling, and the cleanup semantics.
Patching environment variables with monkeypatch.setenv — env isolation in detail, including config caches read at import.
Patching builtins and sys.modules safely — why patching open is the wrong tool for this job.
Controlling time and randomness in tests — the other two sources of environment-dependent flakiness.

← Back to Advanced Mocking & Test Doubles in Python