Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks
Debugging & Performance

Benchmarking with pytest-benchmark

Profilers explain where time goes; benchmarks measure how much time there is. After a profiling session identifies a hot spot and a fix, the question "is it actually faster, and by how much?" needs a measurement that repeats the operation enough times to average out noise, reports the spread as well as the centre, and can be compared with a previous run. Ad-hoc time.perf_counter() calls around a single execution do none of those things.

pytest-benchmark turns a benchmark into an ordinary test. A benchmark fixture calibrates how many times to call the function, runs it in rounds, and reports minimum, maximum, mean, median, standard deviation and outliers. Results can be saved as JSON, compared against earlier runs, and used to fail a build when performance regresses beyond a threshold. The tool is simple; the craft is in writing benchmarks that measure the right thing and reading their numbers honestly.

Prerequisites

Solution

Python
# benchmarks/test_pricing_bench.py
import copy
import pytest
from pricing import apply_rules, build_index

def test_apply_rules_large_order(benchmark, large_order, all_rules):
    result = benchmark(apply_rules, large_order, all_rules)
    assert result > 0                          # still a correctness check

def test_build_index(benchmark, all_rules):
    benchmark.group = "index"
    benchmark(build_index, all_rules)

def test_apply_rules_mutating(benchmark, large_order, all_rules):
    # Each round needs a fresh copy because apply_rules_inplace mutates the order.
    def setup():
        return (copy.deepcopy(large_order), all_rules), {}
    benchmark.pedantic(apply_rules_inplace, setup=setup, rounds=50, warmup_rounds=3)
Bash
# Run only benchmarks, save the results on main.
pytest benchmarks/ --benchmark-only --benchmark-autosave

# On a branch: compare against the latest saved run and fail on a 10% median regression.
pytest benchmarks/ --benchmark-only \
  --benchmark-compare --benchmark-compare-fail=median:10%

# Keep benchmarks out of the normal test run.
pytest --benchmark-skip
How pytest-benchmark measures The benchmark fixture first calibrates by timing a call and choosing how many iterations to group per round. It runs optional warmup rounds that are discarded, then measured rounds until max_time or min_rounds is reached. Round timings are summarised as min, median, mean, standard deviation and outliers. Calibrate, warm up, measure, summarise calibrate iterations per round warmup discarded rounds measured rounds until max_time statistics min · median · stddev pedantic mode lets you fix rounds, iterations and a per-round setup yourself.
Calibration and repetition are what turn a single noisy timing into a distribution you can compare.

Why this works

A single timing of a fast function is dominated by noise: timer resolution, cache state, an interrupt, the garbage collector choosing that moment to run. pytest-benchmark handles this in two ways. It groups calls into rounds long enough that timer resolution is negligible, and it repeats rounds enough times that a distribution emerges. The report's minimum approximates "the function with nothing in the way", the median is robust to occasional slow outliers, and the standard deviation and interquartile range tell you how much to trust either.

benchmark.pedantic exists for functions that cannot simply be called repeatedly — ones that mutate their input, consume an iterator or depend on fresh state. The setup callable runs before each round, outside the timed region, and returns the arguments. Setting iterations=1 (the default in pedantic mode) ensures each timed call gets fresh input.

Saved results are JSON files under .benchmarks/, keyed by machine and Python version. --benchmark-compare loads the most recent one (or a named one) and prints side-by-side columns; --benchmark-compare-fail turns a comparison into a pass/fail decision based on a statistic and a threshold.

Making CI comparisons trustworthy

Benchmarks in CI are harder than on a quiet laptop. Shared runners throttle, vary in CPU model between jobs, and run other workloads on neighbouring cores. A 10% threshold on the mean can fail on noise alone. Several practices make the signal usable.

Compare like with like: keep baselines per runner type, and on hosted runners prefer a baseline measured in the same workflow run — benchmark main and the branch back-to-back in one job — over a baseline saved days ago on different hardware. Compare the median or the minimum, not the mean, since outliers from interruptions inflate the mean. Set thresholds from observed noise: run the same commit ten times, look at the spread, and set the failure threshold comfortably above it. And isolate the benchmark job: no pytest-xdist, no other test jobs on the same runner, garbage collection disabled during rounds with --benchmark-disable-gc when allocation noise dominates.

Noise band versus regression threshold A horizontal axis shows percentage change from the baseline median. Repeated runs of the same commit fall within a noise band of about plus or minus four percent. The failure threshold is set at ten percent, well outside the band, so real regressions fail while noise does not. Set the threshold outside the noise 0% -15% +15% noise band ±4% (same commit, 10 runs) fail at +10% median real regression noisy but fine
Measure the noise first; a threshold inside the band fails on noise, and people stop trusting a gate that does that.

Reading the report table

A pytest-benchmark run ends with a table that is easy to skim and easy to misread. Each row is one benchmark; the columns are Min, Max, Mean, StdDev, Median, IQR, Outliers, OPS (operations per second, the reciprocal of the mean), Rounds and Iterations. Benchmarks in the same group are sorted and compared within that group, so give related benchmarks a shared group name — "index", "pricing", "serialise" — to get meaningful relative comparisons.

Read the spread before the centre. If StdDev is a large fraction of the Mean, or the IQR is wide, the benchmark is noisy and any comparison built on it is weak. The Outliers column, written as a;b, counts rounds beyond one standard deviation and beyond 1.5 IQR from the quartiles; many outliers suggest interference — garbage collection, other processes, thermal throttling — rather than a property of the code. Check Rounds too: a benchmark that only managed five rounds within max_time has a poorly estimated median, and raising --benchmark-max-time or --benchmark-min-rounds is worth doing before drawing conclusions.

When comparing runs, pytest-benchmark prints the saved and current results side by side, with the percentage change for each statistic in brackets. A change in Min with a stable Median usually means the best case moved — for example, a cache became warmer — while a change in Median with a stable Min usually means the typical case got slower. Both are real, but they point at different causes.

What to read first in a benchmark row A benchmark report row is split into three groups of columns read in order: first the spread columns StdDev, IQR and Outliers to judge noise, then Rounds to judge sample size, and only then the centre columns Median and Min for the actual comparison. Spread, then sample size, then centre 1 · spread StdDev · IQR · Outliers is it noisy? 2 · sample size Rounds · Iterations enough rounds? 3 · centre Median · Min now compare A precise-looking median from five noisy rounds is not a measurement.
Judging noise and sample size first stops a confident-looking number from settling a question it cannot answer.

Choosing what to benchmark

A benchmark suite is only as useful as the operations it measures. The temptation is to benchmark every function that was ever slow; the result is a long, noisy suite that nobody reads. A better rule is to benchmark the operations whose speed someone would notice: the request handler behind the busiest endpoint, the batch job's inner transformation, the serialiser used on every message, the query builder on the reporting path. Five to fifteen well-chosen benchmarks usually cover a service. Each benchmark should use realistic input sizes, taken from production data shapes rather than toy fixtures, because algorithmic problems often only show at scale — a quadratic loop is invisible at ten items and dominant at ten thousand.

It also pays to benchmark at two sizes. A pair of benchmarks, one at a typical size and one at ten times that, shows the scaling behaviour directly: if the larger one takes ten times as long, the operation is linear; if it takes a hundred times as long, something quadratic has crept in. That signal survives CI noise much better than a single absolute number, because both measurements share the same machine and moment. A ratio check — assert that the large benchmark's median is less than, say, fifteen times the small one's — can even be written as an ordinary assertion over the saved JSON, giving a regression gate for algorithmic complexity that is almost immune to runner speed.

Edge cases and failure modes

  • Benchmarks in the normal test run. They slow the suite and produce meaningless numbers under xdist. Keep them in a separate directory and use --benchmark-skip for regular runs.
  • Optimised-away work. If the result is unused, some code paths — especially in JIT-backed libraries — may skip work. Assert on the result, as in the first test.
  • Fixture cost leaking in. Work done inside the benchmarked callable counts. Build inputs in fixtures or pedantic setup, not inside the function passed to benchmark.
  • Caches warming across rounds. A function with an internal cache is fast after the first call, so the benchmark measures the cache hit. Clear the cache in setup if the cold path is what matters.
  • Comparing across Python versions. Saved results are tagged by interpreter; comparing 3.12 to 3.13 is legitimate only as an explicit experiment, not as a regression gate.

Frequently Asked Questions

How does pytest-benchmark decide how many times to run my function? It calibrates: it times a single call, then chooses how many calls to group into each round so a round lasts at least the timer's resolution times a safety factor, and runs rounds until max_time is reached or min_rounds is met.

How do I fail CI when performance regresses? Save a baseline with --benchmark-autosave or --benchmark-save, then run later builds with --benchmark-compare and --benchmark-compare-fail, for example mean:10% to fail when the mean is more than ten percent slower.

Why are my benchmark results so noisy in CI? Shared CI runners have variable CPU frequency, noisy neighbours and throttling. Compare against a baseline measured on the same runner type, use medians or minimums rather than means, and set thresholds wider than the observed noise.

← Back to CPU Profiling with cProfile and py-spy