Recording and Replaying HTTP with VCR.py

Hand-written HTTP stubs encode what you believe an API returns. For an API you own, that is fine. For a third-party payment provider whose response contains forty fields, three of which your parser depends on and one of which changed last month, it is a slow-motion outage: the stub keeps returning the old shape and the tests keep passing. VCR.py records the real exchange once, stores it as a cassette, and replays it — so the fixture is a captured fact rather than a remembered one.

Prerequisites

• vcrpy 5.1+, and pytest-recording 0.13+ if you want the pytest integration used below.
• A real credential for the first recording, and a redaction policy before you use it.

Solution

Configure redaction first, record second. Doing it in the other order puts a live token in your git history.

# conftest.py
import pytest

@pytest.fixture(scope="module")
def vcr_config():
    return {
        "filter_headers": [("authorization", "REDACTED"),
                           ("x-api-key", "REDACTED"),
                           ("set-cookie", "REDACTED")],
        "filter_query_parameters": [("api_key", "REDACTED"), ("sig", "REDACTED")],
        "filter_post_data_parameters": [("client_secret", "REDACTED")],
        # Redact anything sensitive inside the body before it is written.
        "before_record_response": _scrub_body,
        "match_on": ["method", "scheme", "host", "port", "path", "query"],
        "record_mode": "none",              # CI default; overridden locally
    }

def _scrub_body(response):
    body = response["body"]["string"]
    if b"account_number" in body:
        response["body"]["string"] = b'{"account_number": "REDACTED"}'
    return response

import pytest

@pytest.mark.vcr                            # cassette name derives from the test id
def test_charge_succeeds(payment_client):
    result = payment_client.charge(amount=500, currency="eur")
    assert result.status == "succeeded"
    assert result.id.startswith("ch_")

Record once, locally, with a mode that allows writing:

$ pytest tests/test_payments.py --record-mode=once      # writes the cassette
$ git diff --stat cassettes/                            # REVIEW before committing
 cassettes/test_charge_succeeds.yaml | 48 ++++++++++++

Reviewing that diff is not optional. It is the moment to confirm the redaction worked, that no pagination token or customer name slipped through, and that the cassette contains the request you intended rather than three retries of a failed one.

Why this works

VCR.py patches the HTTP libraries below your client — urllib3's connection pool for requests, httpcore for httpx — so it sees fully formed requests and complete responses, including status, headers and body. On a cassette hit it constructs the recorded response and hands it back through the same path, which means your client code parses genuine bytes captured from the real service. On a miss the record mode decides: record it, raise, or let it through. That interception depth is why a cassette exercises more of the stack than a hand-written stub, and also why a cassette is coupled to the client library it was recorded with.

Matching rules decide what a cassette proves

The default matcher pairs a request with a recorded interaction on method and URI. That is often too loose and occasionally too strict, and both failure modes are worth understanding.

Too loose: two POSTs to the same endpoint with different bodies match the same interaction, so a test that sends the wrong payload replays the right response and passes. Adding body to match_on fixes it, at the cost of brittleness when the body contains a timestamp or a generated id.

Too strict: a request whose query parameters are ordered differently, or which carries a nonce, never matches its own recording. The answer is a custom matcher that normalises the volatile part rather than loosening the whole rule:

import json

def match_on_json_body(r1, r2):
    # Compare decoded bodies so key ordering and whitespace do not matter,
    # and ignore the client-generated request id.
    def norm(request):
        if not request.body:
            return None
        data = json.loads(request.body)
        data.pop("request_id", None)
        return data
    return norm(r1) == norm(r2)

@pytest.fixture(scope="module")
def vcr(vcr):
    vcr.register_matcher("json_body", match_on_json_body)
    vcr.match_on = ["method", "path", "json_body"]
    return vcr

The general rule: match on everything the API's behaviour depends on, and normalise everything your client generates. A cassette that matches on a nonce is a cassette that has to be re-recorded on every run.

Keeping cassettes from rotting

A cassette is a snapshot, and every snapshot ages. Three practices keep the ageing visible rather than silent.

Re-record on a schedule. A nightly or weekly job that runs the suite with --record-mode=all against a sandbox account, then diffs the cassettes, converts an upstream API change from a production surprise into a pull request. The diff is the alert; the tests passing on old cassettes proves nothing about today's API.

Keep cassettes small and named after the behaviour. One cassette per test, containing one or two interactions, stays reviewable. A single cassette shared by twelve tests becomes an unreadable file that nobody re-records because the diff is impossible to check.

Delete cassettes with the tests they serve. Orphaned cassettes accumulate silently; a short CI check that every .yaml in the cassette directory has a corresponding test keeps the directory honest.

$ pytest --collect-only -q | sed 's/.*:://; s/\[.*//' | sort -u > /tmp/tests.txt
$ ls cassettes | sed 's/\.yaml$//' | sort -u > /tmp/cassettes.txt
$ comm -13 /tmp/tests.txt /tmp/cassettes.txt      # cassettes with no test

Where cassettes prove too heavy for a fast-moving API, the alternative is a contract test against the provider's schema plus hand-written stubs, or a transport-level mock as described in mocking httpx clients with respx. Cassettes are best where the payload is complex and stable; stubs are best where it is simple or changes constantly.

Edge cases and failure modes

• Secrets in the request body are not covered by filter_headers. Use filter_post_data_parameters and a before_record_request hook; a token in a form field is the most common leak.
• Binary and gzipped bodies are stored base64-encoded, which makes the diff unreviewable. Disable compression on the recording run when the body needs to be human-checkable.
• A cassette records timing, not behaviour. Replay is instantaneous, so a test that depends on a timeout firing will never see it; test timeouts with an explicit exception side effect instead.
• Recorded pagination is fixed. A test that follows next links replays exactly the pages captured, so a change in page size upstream is invisible until the next re-record.
• Client library changes break cassettes. Switching from requests to httpx changes the interception layer and can change how headers are recorded; re-record after such a migration rather than adapting the matcher.
• Concurrency is not recorded faithfully. Parallel requests replay in matched order, so a race the real API exhibits will not reproduce.

Reviewing a cassette like a security artefact

A cassette is a file containing real traffic to a real service, committed to a repository that may be public or shared with contractors. Treat the first review of each one as a security review, not a code review.

Four things to check, in order. Headers: Authorization, Cookie, Set-Cookie, X-Api-Key and any vendor-specific auth header must show the redaction placeholder rather than a value. Query strings: signed URLs put credentials in the path, and filter_query_parameters only redacts the parameters you named. Response bodies: account numbers, email addresses, customer names and internal identifiers all arrive in payloads and none are redacted by default. Request bodies: form posts and JSON payloads carry client secrets and personal data on the way out.

Automating the check is better than remembering it. A short test that scans every cassette for high-entropy strings and known key prefixes runs in milliseconds and fails the build when a new recording leaks:

import pathlib, re

SUSPECT = re.compile(rb"(sk_live_|AKIA|-----BEGIN|Bearer [A-Za-z0-9._-]{20,})")

def test_no_secrets_in_cassettes():
    for path in pathlib.Path("cassettes").rglob("*.yaml"):
        body = path.read_bytes()
        assert not SUSPECT.search(body), f"possible secret in {path}"

Add the vendor prefixes your providers use. The check cannot prove a cassette is clean, but it catches the common cases the moment they are committed, which is the only time removing them is cheap.

Frequently Asked Questions

When is a cassette better than a hand-written stub? When the response body is large, structured and not authored by you — a third-party API's payload. A cassette captures the real shape once, including fields you did not know existed, where a hand-written stub captures only what you remembered.

How do I stop secrets ending up in a cassette? Configure filter_headers, filter_query_parameters and before_record_response when the VCR instance is created, so redaction happens before anything is written to disk. Never rely on editing the file afterwards.

What record mode should CI use?none. It replays existing cassettes and fails on any request that has none, so CI can never reach the network or silently re-record. Developers use once or new_episodes locally when adding coverage.

How do I know when a cassette has gone stale? Run a scheduled job with record mode all against the real API and diff the resulting cassettes. A changed cassette is an API change your replayed tests would never have noticed.

Related

• Mocking network and HTTP calls — where cassette replay sits among the interception layers.
• Mocking httpx clients with respx — the hand-declared alternative for fast-changing APIs.
• Mocking requests with the responses library — stub-based interception for the requests stack.
• Controlling time and randomness in tests — freezing the clock so recorded timestamps stay comparable.

← Back to Mocking Network and HTTP Calls