Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Integration & Data

Validating Responses Against an OpenAPI Schema

A provider changes total from an integer to a string, ships it, and every consumer's error rate rises. Each consumer's test suite was green throughout, because none of them checked that the responses they mocked or recorded still looked like the real thing. Validating every response against the provider's OpenAPI document closes that gap for the cost of one wrapper around the test client.

Prerequisites

  • openapi-core >= 0.19, which validates both requests and responses against OpenAPI 3.0 and 3.1.
  • A copy of the provider's OpenAPI document, committed at a known version.
  • pytest >= 8.0, and a test client — requests, httpx, or a framework's own.
  • The wider context in contract testing for HTTP APIs.

Solution

Python
import pytest
from openapi_core import OpenAPI
from openapi_core.contrib.requests import RequestsOpenAPIRequest, RequestsOpenAPIResponse
from openapi_core.exceptions import OpenAPIError


@pytest.fixture(scope="session")
def billing_contract():
    # Vendored at a known version: a remote edit cannot change these results.
    return OpenAPI.from_file_path("contracts/billing-openapi-2.4.0.yaml")


class ValidatingClient:
    """Wraps a requests.Session; every exchange is checked against the contract."""

    def __init__(self, session, contract):
        self._session, self._contract = session, contract

    def request(self, method, url, **kwargs):
        response = self._session.request(method, url, **kwargs)
        try:
            self._contract.validate_response(
                RequestsOpenAPIRequest(response.request),
                RequestsOpenAPIResponse(response),
            )
        except OpenAPIError as exc:
            raise AssertionError(
                f"{method} {url} → {response.status_code} violates the contract: {exc}"
            ) from exc
        return response

    def get(self, url, **kw):  return self.request("GET", url, **kw)
    def post(self, url, **kw): return self.request("POST", url, **kw)


@pytest.fixture
def billing(billing_contract, recorded_session):
    return ValidatingClient(recorded_session, billing_contract)


def test_invoice_total_is_read_correctly(billing):
    invoice = billing.get("https://billing.test/invoices/inv_1").json()
    assert invoice["total_minor"] == 1234
Where validation sits in each exchange A test calls the validating client, which sends the request through a recorded or live session and receives a response. Before the response reaches the test it is checked against the vendored OpenAPI document; a conforming response is returned, and a non-conforming one fails the test with the path, method and violated rule. Every response passes through the contract first test validating client wraps the session transport recorded or live conforms returned to the test violates fails with the rule Because validation lives in the client, no test can forget it and no test has to repeat it.
Recorded cassettes pass through the same check, which is what stops a stale recording from silently asserting against a response shape the provider no longer sends.

Why this works

openapi-core resolves the request's path and method against the document to find the matching operation, then checks the response's status code, headers and body against that operation's declared responses. A status code not listed, a required field missing, a value of the wrong type, an enum value not in the list, a string failing its declared format — each raises with a JSON pointer to the offending location.

Putting the check in the client rather than in each test is what makes it reliable. A validation step that depends on every test author remembering it will be forgotten in the one test that would have caught the change; one that lives in the transport cannot be.

Edge cases and failure modes

  • Permissive published schemas. Many providers publish additionalProperties: true and few required fields, so almost anything validates. Keep a tightened local copy and note the differences.
  • nullable versus type: [string, "null"]. OpenAPI 3.0 and 3.1 express nullability differently; a document mixing them validates inconsistently. Normalise to the version the file declares.
  • Server URLs. The document's servers list may not match the test base URL, so path matching fails. Configure the validator with the test server, or strip the host before matching.
  • Undocumented error responses. A 429 or 503 the document never mentions fails validation. That is usually correct — and a prompt to ask the provider to document it.
  • Recorded responses from an older schema. A cassette recorded against 2.3 validated against 2.4 fails, which is the point: re-record it.

Tightening a loose schema without forking it

A permissive published schema validates almost anything, which means the check passes while the payload drifts. The fix is an overlay: a small local document that tightens specific schemas, merged over the vendored one at load time.

Python
import copy

import yaml
from openapi_core import OpenAPI


def load_tightened(path: str, overlay: dict) -> OpenAPI:
    spec = yaml.safe_load(open(path))
    schemas = spec["components"]["schemas"]
    for name, changes in overlay.items():
        schemas[name] = {**copy.deepcopy(schemas[name]), **changes}
    return OpenAPI.from_dict(spec)


STRICT = {
    "Invoice": {
        "additionalProperties": False,                    # no silent extra fields
        "required": ["id", "total_minor", "currency", "status"],
    },
}

contract = load_tightened("contracts/billing-openapi-2.4.0.yaml", STRICT)

Keeping the overlay separate from the vendored file preserves a clean upgrade path: replacing the provider's document with a new version leaves the local tightening intact, and a diff of the overlay shows exactly which assumptions this consumer makes beyond what the provider promises. Those assumptions are also the natural contents of a consumer-driven contract, covered in consumer-driven contract tests with Pact Python.

Validating mocked responses too

The highest-value place for this check is not the live or recorded response — it is the hand-written mock. A test that stubs the provider with responses or respx asserts against whatever JSON its author typed, and that JSON drifts from reality the day it is written. Routing mocks through the same validator makes them honest.

Python
import pytest
import responses


@pytest.fixture
def mocked_billing(billing_contract):
    with responses.RequestsMock() as rsps:
        original_add = rsps.add

        def add_validated(method, url, json=None, status=200, **kwargs):
            # Validate the MOCK against the contract before registering it.
            body = _fake_response(method, url, json, status)
            billing_contract.validate_response(*body)
            return original_add(method, url, json=json, status=status, **kwargs)

        rsps.add = add_validated
        yield rsps


def test_handles_a_paid_invoice(mocked_billing, billing_client):
    mocked_billing.add(
        "GET", "https://billing.test/invoices/inv_1",
        json={"id": "inv_1", "total_minor": 1234, "currency": "GBP", "status": "paid"},
    )
    assert billing_client.fetch("inv_1").is_paid
Three sources of responses, one validator Live responses, recorded cassettes and hand-written mocks all pass through the same contract validator before reaching the test. The hand-written mock is the source most likely to drift from reality, so validating it gives the largest benefit. Validate wherever a response comes from live sandbox — rarely drifts recorded cassette — ages hand-written mock — drifts first contract validator same document for all the test only ever sees contract-shaped data
A mock validated against the schema cannot claim a field the provider does not send, which is the most common way mocked tests pass against an interface that does not exist.

This inverts the usual weakness of mocked HTTP tests. Instead of the mock being a place where invented shapes accumulate unchecked, it becomes a place where every invented shape is checked against the provider's own description at the moment it is written.

Upgrading the vendored document

Vendoring the schema makes upgrades deliberate, which is the point, but it also means somebody has to perform them. A small routine keeps that cheap and turns each upgrade into a reviewable change rather than a surprise.

First, diff the old and new documents at the level of operations and schemas rather than as text. Tools such as oasdiff classify changes as breaking or non-breaking, which is exactly the question the review needs answered: a new optional field is safe, a removed field or a narrowed enum is not.

Bash
oasdiff breaking contracts/billing-openapi-2.4.0.yaml downloads/billing-openapi-2.5.0.yaml
Plain text
error  [response-property-removed] GET /invoices/{id} 200: removed 'legacy_ref'
warn   [response-property-enum-value-added] GET /invoices/{id} 200: status added 'disputed'

Second, swap the file and run the suite. Every test that fails is now a precise list of places in your code that depend on something the provider changed — the removed legacy_ref will surface wherever it was read, and the new disputed status will surface wherever a match or an if chain assumed the old set was exhaustive.

Third, commit the new document and the fixes together, with the diff output in the commit message. That record answers the question every future reader of the code will ask — "when did the provider start sending this?" — without anyone having to reconstruct it.

A deliberate schema upgrade in three steps Three steps. Diff the old and new documents with a tool that classifies breaking changes. Replace the vendored file and run the suite so every dependency on a changed field fails in place. Commit the new document together with the code fixes and the diff output. The upgrade is a change you review, not an event you suffer 1 · diff oasdiff breaking removed fields, narrowed enums 2 · swap and run every failure is a place your code depended on the changed contract 3 · commit together new document, code fixes, and the diff in the message A schema fetched at runtime skips all three steps — and the review they would have prompted.
Step two is where vendoring pays for itself: the test failures are an exact map of the consumer's exposure to the provider's change.

A scheduled job that downloads the provider's current document and runs the diff against the vendored one — reporting rather than failing — gives advance notice of upcoming changes without letting them destabilise the build. It is the same pattern as the sandbox monitor for third-party providers, applied to the document rather than to live responses.

Finally, validate requests as well as responses where the provider's document describes them. openapi-core checks outgoing request bodies, query parameters and headers against the same operation, which catches the mirror-image bug: a consumer sending a field the provider has deprecated, or omitting one it has made required. A request-side failure in the test suite is far cheaper than the 400 responses it would otherwise produce in production, and it surfaces in exactly the test that exercises the call. Enabling it is one extra call in the wrapper above, made before the request is sent rather than after the response arrives, and it uses the document already parsed for the response checks. Between the two directions, every exchange in the suite is checked against the provider's own description of it, which is as close to a guarantee as a consumer can get without the provider's cooperation.

Frequently Asked Questions

Should the schema be fetched from the provider at test time? No. Vendor a copy at a known version and update it deliberately. A schema fetched at runtime means the provider's edit changes your test results with no commit in your repository, and an unreachable provider breaks your suite entirely.

Does validation catch a field that changed meaning but not type? No. Schema validation checks structure — types, required fields, enums, formats. A price that switched from pounds to pence is still an integer. Semantic changes need consumer-driven contracts or explicit assertions on known values.

Is validating every response too slow? Rarely. Validation is a pure in-memory check against a parsed schema and costs microseconds to low milliseconds per response. Parse the document once per session, and the overhead disappears into the noise of the HTTP call itself.

← Back to Contract Testing for HTTP APIs