Spinning Up Services with Testcontainers

A test that runs against a real Postgres finds constraint violations, type coercion surprises and isolation-level behaviour that no in-memory substitute reproduces. The obstacle has always been operational: somebody has to install the service, keep its version aligned across every laptop and every CI runner, and clean it between runs. Testcontainers removes that obstacle by making the service a value in the test code — started by a fixture, addressed by a URL the fixture returns, destroyed when the run ends.

Prerequisites

• A working Docker daemon reachable from the test process. Podman works with the Docker-compatible socket enabled.
• testcontainers >= 4.0, which reorganised the package into per-service modules (testcontainers.postgres, testcontainers.redis, and so on).
• pytest >= 8.0, and pytest-xdist if the suite runs in parallel.
• Enough disk on the CI runner for the image cache; a cold pull of postgres:16-alpine is roughly 80 MB and the difference between a two-second and a forty-second startup.

Core concept: the container is a fixture with a URL

Everything Testcontainers does reduces to three operations: start an image, wait until it is genuinely usable, and expose the address it ended up on. The third is not a formality — the library deliberately maps container ports to ephemeral host ports so that two runs on the same machine never collide, which means the connection string is discovered at runtime and can never be hardcoded.

Step-by-step implementation

1. One container for the whole session

import pytest
from testcontainers.postgres import PostgresContainer


@pytest.fixture(scope="session")
def postgres():
    # Context manager: __exit__ stops and removes the container even on failure.
    with PostgresContainer("postgres:16-alpine") as container:
        yield container


@pytest.fixture(scope="session")
def postgres_dsn(postgres):
    # Never hardcode the port; the daemon chose it at start-up.
    return postgres.get_connection_url()

Scope is the single most consequential setting here. At function scope this fixture adds three seconds to every test; at session scope it adds three seconds to the run. Per-test isolation comes from the transactional fixture layered on top, not from restarting the service.

2. Configure the image rather than the test

from testcontainers.postgres import PostgresContainer

container = (
    PostgresContainer("postgres:16-alpine")
    .with_env("POSTGRES_INITDB_ARGS", "--data-checksums")
    # Tuned for a throwaway instance: durability is worthless here and fsync
    # dominates insert-heavy test suites.
    .with_command("postgres -c fsync=off -c full_page_writes=off -c synchronous_commit=off")
)

Turning off fsync for a container that is deleted minutes later typically halves the runtime of an insert-heavy suite. It is safe precisely because the data has no value — the same setting in production would be negligence.

3. Wait on something real

from testcontainers.core.waiting_utils import wait_for_logs
from testcontainers.kafka import KafkaContainer


def start_kafka():
    container = KafkaContainer("confluentinc/cp-kafka:7.6.0")
    container.start()
    # Kafka is "running" long before it is usable; wait for the broker's own
    # readiness line rather than guessing at a duration.
    wait_for_logs(container, r"\[KafkaServer id=\d+\] started", timeout=60)
    return container

Every official module ships a default wait strategy, and the bundled ones are usually right. It is the services without a module — an internal image, a niche broker — where teams reach for time.sleep(10), and where the readiness poll pays for itself immediately. The options and their failure modes are covered in waiting for container readiness without sleep.

4. Isolate parallel workers

import os

import pytest
from testcontainers.redis import RedisContainer


@pytest.fixture(scope="session")
def redis_client():
    with RedisContainer("redis:7-alpine") as container:
        client = container.get_client()
        # One logical database per worker: cheaper than one container each,
        # and Redis gives 16 of them for free.
        worker = os.environ.get("PYTEST_XDIST_WORKER", "gw0")
        client.select(int(worker.removeprefix("gw")) % 16)
        yield client

Under pytest-xdist each worker is a separate process running its own session fixtures, so a naive setup starts one container per worker — eight Postgres instances on an eight-core runner. Sometimes that is exactly right, because it gives complete isolation. More often the cheaper answer is one container with per-worker databases or per-worker Redis indices, which is a second of setup instead of thirty.

5. Compose several services

import pytest
from testcontainers.compose import DockerCompose


@pytest.fixture(scope="session")
def stack():
    # When services must see each other by name, one compose file beats
    # wiring three containers together by hand.
    with DockerCompose("tests/fixtures", compose_file_name="docker-compose.test.yml",
                       pull=True) as compose:
        compose.wait_for("http://localhost:8080/health")
        yield compose

Individual containers are the right unit when services are independent. As soon as one service needs to reach another by hostname — an application container talking to a database container — a compose file expresses the network in one place instead of scattering with_network calls through fixtures.

Verification

Prove the lifecycle is correct before relying on it, with two checks that take a minute.

# 1. The suite starts exactly the containers you expect, and removes them.
docker ps --filter "label=org.testcontainers=true" --format "{{.Image}}\t{{.Status}}"
pytest tests/integration -q
docker ps -a --filter "label=org.testcontainers=true" --format "{{.Image}}\t{{.Status}}"

# during the run
postgres:16-alpine     Up 4 seconds (healthy)
testcontainers/ryuk    Up 5 seconds
# after the run — empty output is the correct result

# 2. The mapped port really is ephemeral, and the suite copes.
pytest tests/integration -q & pytest tests/integration -q; wait

Two concurrent runs of the same suite must both pass. If they collide with "address already in use", something is hardcoding a port; if they interfere through shared data, a container is being reused when it should not be.

Troubleshooting

The reaper, and why containers still leak without it

docker run leaves a container behind if the process that started it dies. A test suite is exactly such a process, and SIGKILL, an OOM kill or a hard ctrl-C all bypass Python's cleanup entirely — which on a developer's machine means a slow accumulation of stopped Postgres containers, and on a shared CI runner means eventual disk exhaustion.

Testcontainers solves this with Ryuk, a small sidecar container started alongside the first real one. The test process opens a TCP connection to Ryuk and holds it for the lifetime of the run. Ryuk's only job is to watch that connection: when it drops — cleanly or otherwise — Ryuk removes every container, network and volume carrying the session's label.

Ryuk is occasionally disabled (TESTCONTAINERS_RYUK_DISABLED=true) because a hardened CI environment forbids the privileged socket mount it needs. That is a legitimate reason, but the consequence must be handled: without Ryuk, cleanup depends entirely on the fixture's teardown running, so the job needs an unconditional docker rm step that runs even on failure.

Making it fast in CI

Three costs dominate, and they respond to different remedies.

The image pull is the largest on a cold runner and is fixed by caching. Most CI platforms can restore a Docker layer cache between jobs; failing that, an explicit docker pull in an earlier step at least moves the cost out of the test timing. Pinning exact tags matters here too: postgres:16-alpine is cacheable, while postgres:latest invalidates whenever upstream publishes.

Container start-up is fixed by scope and by tuning. One session-scoped container with fsync=off is typically three seconds; the same container per test, untuned, is minutes.

Readiness waiting is fixed by polling a real signal. The default strategies poll on a short interval, so a Postgres that is ready in 1.8 seconds is detected at 1.8 seconds. A sleep(5) costs 5 seconds every run and still fails on the day the runner is loaded.

What does not help is running containers on every push. The marker-based split described in integration, database and service testing keeps the fast suite free of Docker entirely, which matters most for contributors who do not have a daemon running at all — their pytest -m "not integration" should pass on a laptop with nothing installed but Python.

Writing a fixture for a service with no module

The bundled modules cover Postgres, MySQL, Redis, Kafka, RabbitMQ, Elasticsearch, MongoDB, LocalStack and a couple of dozen more. Everything else — an internal service, a vendor's image, a mock server — needs a fixture built from the generic container, and the pattern is short enough to write from memory.

import pytest
import requests
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import wait_for_logs


@pytest.fixture(scope="session")
def billing_stub():
    container = (
        DockerContainer("ghcr.io/acme/billing-stub:2.4.1")   # pinned, always
        .with_exposed_ports(8080)                            # container-side port
        .with_env("STUB_MODE", "deterministic")
    )
    with container:
        # The image prints this line once its HTTP listener is bound.
        wait_for_logs(container, "listening on :8080", timeout=30)

        host = container.get_container_host_ip()             # not "localhost" in DinD
        port = container.get_exposed_port(8080)              # the ephemeral host port
        base_url = f"http://{host}:{port}"

        # Belt and braces: one real request before any test runs, so a wait
        # strategy that was satisfied too early fails here rather than mid-suite.
        response = requests.get(f"{base_url}/health", timeout=5)
        response.raise_for_status()

        yield base_url

Four things in that fixture are worth copying verbatim. with_exposed_ports takes the port inside the container, while get_exposed_port returns the mapped one outside it — confusing them produces a connection refused that looks like a readiness problem. get_container_host_ip() rather than a literal localhost is what makes the fixture work under Docker-in-Docker. The log wait is cheap and specific. And the single real request afterwards converts "the log line appeared but the service was still initialising" into a failure during setup, where it is obvious, rather than into a flaky first test.

For images that expose no useful log line, poll the service itself with a bounded loop. wait_for_logs and a health poll are the two primitives; between them they cover every service worth containerising.

Container, fake, or shared instance

Not every external dependency deserves a container, and choosing badly in either direction is expensive. The question is what the tests need to be true.

A fake is right when the tests exercise your code's logic and the dependency's behaviour is incidental. An in-memory repository, a respx route, a stub queue: microseconds per test, no Docker, and the whole suite runs on a laptop with nothing installed.

A container is right when the dependency's own behaviour is under test — SQL semantics, Redis eviction, Kafka consumer-group rebalancing, an S3 API's exact error codes. These are the behaviours a fake reproduces approximately and therefore wrongly.

A shared instance — a long-lived database on the developer's machine or a team server — is almost never right. It reintroduces the version drift and cross-run contamination that containers exist to remove, and it makes the suite fail for people who have not performed a setup ritual nobody wrote down.

Local reuse without CI contamination

Developers run the integration suite dozens of times a day, and three seconds of container startup on every run is a tax worth removing — for them specifically, and never for the pipeline.

import os

import pytest
from testcontainers.postgres import PostgresContainer


@pytest.fixture(scope="session")
def postgres():
    container = PostgresContainer("postgres:16-alpine")

    # Reuse is a local-only optimisation: a container that survives between runs
    # also survives between builds, which is exactly what CI must not have.
    if os.environ.get("TC_REUSE") == "1" and not os.environ.get("CI"):
        container = container.with_reuse(True)

    with container:
        yield container

Reuse requires the Docker daemon to be configured with testcontainers.reuse.enable=true in ~/.testcontainers.properties; without it the flag is ignored rather than failing, which is the right default. When it is active, testcontainers computes a hash of the container's configuration and reattaches to any running container with the same hash, so changing the image tag or an environment variable transparently starts a fresh one.

The trade-off is state. A reused Postgres keeps the rows any previous run committed, which is harmless when every test rolls back and fatal when one does not. That asymmetry is a useful diagnostic in itself: if enabling reuse locally makes tests fail, some test is committing without cleaning up, and that test would eventually have failed in CI under a different ordering. Treating a reuse failure as a bug in the suite rather than a reason to disable reuse tends to find real isolation defects.

Two smaller local conveniences are worth the same treatment. TESTCONTAINERS_RYUK_DISABLED shaves the reaper's own startup, at the cost of the cleanup guarantee — acceptable on a laptop where docker system prune is a keystroke away, never in a shared runner. And keeping a long-lived docker pull of the pinned images in a make setup target means a new contributor's first test run is fast rather than a four-minute download that looks like a hang.

The general rule behind all three is that a local optimisation must be opt-in and must announce itself. An environment variable that a developer sets deliberately is fine; a default that quietly changes isolation semantics depending on which machine the suite runs on is how a pipeline ends up with failures nobody can reproduce.

Frequently Asked Questions

How do I stop containers leaking when a test run is interrupted? Use the container as a context manager so __exit__ stops it, and enable Ryuk, the reaper sidecar testcontainers starts by default. Ryuk holds a connection to the test process and removes every labelled container when that connection drops, which covers SIGKILL and crashed runs where Python's cleanup never executes.

Can containers be reused between runs to save startup time? Yes, with testcontainers' reuse flag plus a Docker daemon that has reuse enabled, which keeps a labelled container alive across runs. It is excellent locally and wrong in CI, where a reused container means state carried between builds. Gate it on an environment variable so developers get the speed and the pipeline gets the isolation.

Why does the container's port keep changing? Because testcontainers maps the container's port to an ephemeral host port on purpose, so parallel runs never collide. Always read the mapped port with get_exposed_port() or the module's connection-URL helper instead of hardcoding it.

Do Testcontainers work inside a CI container? Yes, with either a mounted Docker socket or a Docker-in-Docker service. Mounting the host socket is faster and shares the image cache; Docker-in-Docker is more isolated and needs privileged mode. In both cases the container's ports are reachable from the test process, but the hostname may not be localhost, so read it from the API rather than assuming.

How much startup time should a session-scoped container cost? Two to five seconds for Postgres or Redis with a warm image cache, and longer for Kafka or Elasticsearch, which have real initialisation work. If it is consistently slower, the wait strategy is polling too infrequently or the image is being pulled on every run.

Related guides

• Work through a complete Postgres fixture in starting Postgres with testcontainers-python.
• Replace every startup sleep with waiting for container readiness without sleep.
• Layer per-test isolation on top with database fixtures and transactional tests.
• Debug a container-only failure using attaching debugpy to a container.
• Keep the fast suite Docker-free by faking the transport instead, as in mocking network and HTTP calls.

← Back to Integration, Database & Service Testing