[{"data":1,"prerenderedAt":817},["ShallowReactive",2],{"page-\u002Fsystematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002Fbenchmarking-with-pytest-benchmark\u002F":3},{"id":4,"title":5,"body":6,"description":783,"extension":784,"meta":785,"navigation":96,"path":813,"seo":814,"stem":815,"__hash__":816},"content\u002Fsystematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002Fbenchmarking-with-pytest-benchmark\u002Findex.md","Benchmarking with pytest-benchmark",{"type":7,"value":8,"toc":772},"minimark",[9,18,25,30,52,56,174,248,384,388,391,405,420,424,427,442,521,525,532,551,554,625,629,632,635,639,684,688,701,724,730,734,763,768],[10,11,12,13,17],"p",{},"Profilers explain where time goes; benchmarks measure how much time there is. After a profiling session identifies a hot spot and a fix, the question \"is it actually faster, and by how much?\" needs a measurement that repeats the operation enough times to average out noise, reports the spread as well as the centre, and can be compared with a previous run. Ad-hoc ",[14,15,16],"code",{},"time.perf_counter()"," calls around a single execution do none of those things.",[10,19,20,21,24],{},"pytest-benchmark turns a benchmark into an ordinary test. A ",[14,22,23],{},"benchmark"," fixture calibrates how many times to call the function, runs it in rounds, and reports minimum, maximum, mean, median, standard deviation and outliers. Results can be saved as JSON, compared against earlier runs, and used to fail a build when performance regresses beyond a threshold. The tool is simple; the craft is in writing benchmarks that measure the right thing and reading their numbers honestly.",[26,27,29],"h2",{"id":28},"prerequisites","Prerequisites",[31,32,33,44],"ul",{},[34,35,36,39,40,43],"li",{},[14,37,38],{},"pytest >= 8.0",", ",[14,41,42],{},"pytest-benchmark >= 4.0",".",[34,45,46,47,43],{},"Background from ",[48,49,51],"a",{"href":50},"\u002Fsystematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002F","CPU profiling with cProfile and py-spy",[26,53,55],{"id":54},"solution","Solution",[57,58,63],"pre",{"className":59,"code":60,"language":61,"meta":62,"style":62},"language-python shiki shiki-themes github-light github-dark","# benchmarks\u002Ftest_pricing_bench.py\nimport copy\nimport pytest\nfrom pricing import apply_rules, build_index\n\ndef test_apply_rules_large_order(benchmark, large_order, all_rules):\n    result = benchmark(apply_rules, large_order, all_rules)\n    assert result > 0                          # still a correctness check\n\ndef test_build_index(benchmark, all_rules):\n    benchmark.group = \"index\"\n    benchmark(build_index, all_rules)\n\ndef test_apply_rules_mutating(benchmark, large_order, all_rules):\n    # Each round needs a fresh copy because apply_rules_inplace mutates the order.\n    def setup():\n        return (copy.deepcopy(large_order), all_rules), {}\n    benchmark.pedantic(apply_rules_inplace, setup=setup, rounds=50, warmup_rounds=3)\n","python","",[14,64,65,73,79,85,91,98,104,110,116,121,127,133,139,144,150,156,162,168],{"__ignoreMap":62},[66,67,70],"span",{"class":68,"line":69},"line",1,[66,71,72],{},"# benchmarks\u002Ftest_pricing_bench.py\n",[66,74,76],{"class":68,"line":75},2,[66,77,78],{},"import copy\n",[66,80,82],{"class":68,"line":81},3,[66,83,84],{},"import pytest\n",[66,86,88],{"class":68,"line":87},4,[66,89,90],{},"from pricing import apply_rules, build_index\n",[66,92,94],{"class":68,"line":93},5,[66,95,97],{"emptyLinePlaceholder":96},true,"\n",[66,99,101],{"class":68,"line":100},6,[66,102,103],{},"def test_apply_rules_large_order(benchmark, large_order, all_rules):\n",[66,105,107],{"class":68,"line":106},7,[66,108,109],{},"    result = benchmark(apply_rules, large_order, all_rules)\n",[66,111,113],{"class":68,"line":112},8,[66,114,115],{},"    assert result > 0                          # still a correctness check\n",[66,117,119],{"class":68,"line":118},9,[66,120,97],{"emptyLinePlaceholder":96},[66,122,124],{"class":68,"line":123},10,[66,125,126],{},"def test_build_index(benchmark, all_rules):\n",[66,128,130],{"class":68,"line":129},11,[66,131,132],{},"    benchmark.group = \"index\"\n",[66,134,136],{"class":68,"line":135},12,[66,137,138],{},"    benchmark(build_index, all_rules)\n",[66,140,142],{"class":68,"line":141},13,[66,143,97],{"emptyLinePlaceholder":96},[66,145,147],{"class":68,"line":146},14,[66,148,149],{},"def test_apply_rules_mutating(benchmark, large_order, all_rules):\n",[66,151,153],{"class":68,"line":152},15,[66,154,155],{},"    # Each round needs a fresh copy because apply_rules_inplace mutates the order.\n",[66,157,159],{"class":68,"line":158},16,[66,160,161],{},"    def setup():\n",[66,163,165],{"class":68,"line":164},17,[66,166,167],{},"        return (copy.deepcopy(large_order), all_rules), {}\n",[66,169,171],{"class":68,"line":170},18,[66,172,173],{},"    benchmark.pedantic(apply_rules_inplace, setup=setup, rounds=50, warmup_rounds=3)\n",[57,175,179],{"className":176,"code":177,"language":178,"meta":62,"style":62},"language-bash shiki shiki-themes github-light github-dark","# Run only benchmarks, save the results on main.\npytest benchmarks\u002F --benchmark-only --benchmark-autosave\n\n# On a branch: compare against the latest saved run and fail on a 10% median regression.\npytest benchmarks\u002F --benchmark-only \\\n  --benchmark-compare --benchmark-compare-fail=median:10%\n\n# Keep benchmarks out of the normal test run.\npytest --benchmark-skip\n","bash",[14,180,181,187,204,208,213,224,232,236,241],{"__ignoreMap":62},[66,182,183],{"class":68,"line":69},[66,184,186],{"class":185},"sJ8bj","# Run only benchmarks, save the results on main.\n",[66,188,189,193,197,201],{"class":68,"line":75},[66,190,192],{"class":191},"sScJk","pytest",[66,194,196],{"class":195},"sZZnC"," benchmarks\u002F",[66,198,200],{"class":199},"sj4cs"," --benchmark-only",[66,202,203],{"class":199}," --benchmark-autosave\n",[66,205,206],{"class":68,"line":81},[66,207,97],{"emptyLinePlaceholder":96},[66,209,210],{"class":68,"line":87},[66,211,212],{"class":185},"# On a branch: compare against the latest saved run and fail on a 10% median regression.\n",[66,214,215,217,219,221],{"class":68,"line":93},[66,216,192],{"class":191},[66,218,196],{"class":195},[66,220,200],{"class":199},[66,222,223],{"class":199}," \\\n",[66,225,226,229],{"class":68,"line":100},[66,227,228],{"class":199},"  --benchmark-compare",[66,230,231],{"class":199}," --benchmark-compare-fail=median:10%\n",[66,233,234],{"class":68,"line":106},[66,235,97],{"emptyLinePlaceholder":96},[66,237,238],{"class":68,"line":112},[66,239,240],{"class":185},"# Keep benchmarks out of the normal test run.\n",[66,242,243,245],{"class":68,"line":118},[66,244,192],{"class":191},[66,246,247],{"class":199}," --benchmark-skip\n",[249,250,253,380],"figure",{"className":251},[252],"diagram",[254,255,262,263,262,267,262,271,262,289,262,297,262,307,262,316,262,322,262,327,262,332,262,336,262,339,262,343,262,347,262,350,262,354,262,358,262,361,262,368,262,372,262,376],"svg",{"viewBox":256,"role":257,"ariaLabelledBy":258,"xmlns":261},"0 0 800 246","img",[259,260],"pb-t","pb-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","\n  ",[264,265,266],"title",{"id":259},"How pytest-benchmark measures",[268,269,270],"desc",{"id":260},"The benchmark fixture first calibrates by timing a call and choosing how many iterations to group per round. It runs optional warmup rounds that are discarded, then measured rounds until max_time or min_rounds is reached. Round timings are summarised as min, median, mean, standard deviation and outliers.",[272,273,274,275,262],"defs",{},"\n    ",[276,277,284],"marker",{"id":278,"viewBox":279,"refX":280,"refY":281,"markerWidth":282,"markerHeight":282,"orient":283},"pb-a","0 0 10 10","9","5","7","auto-start-reverse",[285,286],"path",{"d":287,"fill":288},"M0 0 L10 5 L0 10 z","#81b29a",[290,291],"rect",{"x":292,"y":292,"width":293,"height":294,"rx":295,"fill":296},"0","800","246","14","#fffdf8",[298,299,306],"text",{"x":300,"y":301,"textAnchor":302,"fontSize":303,"fontWeight":304,"fill":305},"400","28","middle","15.5","700","#3d405b","Calibrate, warm up, measure, summarise",[290,308],{"x":309,"y":310,"width":311,"height":310,"rx":312,"fill":313,"stroke":314,"strokeWidth":315},"20","80","170","11","#f7f0da","#f2cc8f","2",[298,317,321],{"x":318,"y":319,"textAnchor":302,"fontSize":320,"fontWeight":304,"fill":305},"105","108","12","calibrate",[298,323,326],{"x":318,"y":324,"textAnchor":302,"fontSize":325,"fill":305},"130","10.5","iterations per round",[290,328],{"x":329,"y":310,"width":311,"height":310,"rx":312,"fill":330,"stroke":305,"strokeWidth":331},"215","#f4f1de","1.5",[298,333,335],{"x":334,"y":319,"textAnchor":302,"fontSize":320,"fontWeight":304,"fill":305},"300","warmup",[298,337,338],{"x":334,"y":324,"textAnchor":302,"fontSize":325,"fill":305},"discarded rounds",[290,340],{"x":341,"y":310,"width":311,"height":310,"rx":312,"fill":342,"stroke":288,"strokeWidth":315},"410","#e6f0ea",[298,344,346],{"x":345,"y":319,"textAnchor":302,"fontSize":320,"fontWeight":304,"fill":305},"495","measured rounds",[298,348,349],{"x":345,"y":324,"textAnchor":302,"fontSize":325,"fill":305},"until max_time",[290,351],{"x":352,"y":310,"width":353,"height":310,"rx":312,"fill":305},"605","175",[298,355,357],{"x":356,"y":319,"textAnchor":302,"fontSize":320,"fontWeight":304,"fill":296},"692","statistics",[298,359,360],{"x":356,"y":324,"textAnchor":302,"fontSize":325,"fill":296},"min · median · stddev",[68,362],{"x1":363,"y1":364,"x2":365,"y2":364,"stroke":288,"strokeWidth":366,"markerEnd":367},"192","120","211","1.8","url(#pb-a)",[68,369],{"x1":370,"y1":364,"x2":371,"y2":364,"stroke":288,"strokeWidth":366,"markerEnd":367},"387","406",[68,373],{"x1":374,"y1":364,"x2":375,"y2":364,"stroke":288,"strokeWidth":366,"markerEnd":367},"582","601",[298,377,379],{"x":300,"y":378,"textAnchor":302,"fontSize":312,"fill":305},"206","pedantic mode lets you fix rounds, iterations and a per-round setup yourself.",[381,382,383],"figcaption",{},"Calibration and repetition are what turn a single noisy timing into a distribution you can compare.",[26,385,387],{"id":386},"why-this-works","Why this works",[10,389,390],{},"A single timing of a fast function is dominated by noise: timer resolution, cache state, an interrupt, the garbage collector choosing that moment to run. pytest-benchmark handles this in two ways. It groups calls into rounds long enough that timer resolution is negligible, and it repeats rounds enough times that a distribution emerges. The report's minimum approximates \"the function with nothing in the way\", the median is robust to occasional slow outliers, and the standard deviation and interquartile range tell you how much to trust either.",[10,392,393,396,397,400,401,404],{},[14,394,395],{},"benchmark.pedantic"," exists for functions that cannot simply be called repeatedly — ones that mutate their input, consume an iterator or depend on fresh state. The ",[14,398,399],{},"setup"," callable runs before each round, outside the timed region, and returns the arguments. Setting ",[14,402,403],{},"iterations=1"," (the default in pedantic mode) ensures each timed call gets fresh input.",[10,406,407,408,411,412,415,416,419],{},"Saved results are JSON files under ",[14,409,410],{},".benchmarks\u002F",", keyed by machine and Python version. ",[14,413,414],{},"--benchmark-compare"," loads the most recent one (or a named one) and prints side-by-side columns; ",[14,417,418],{},"--benchmark-compare-fail"," turns a comparison into a pass\u002Ffail decision based on a statistic and a threshold.",[26,421,423],{"id":422},"making-ci-comparisons-trustworthy","Making CI comparisons trustworthy",[10,425,426],{},"Benchmarks in CI are harder than on a quiet laptop. Shared runners throttle, vary in CPU model between jobs, and run other workloads on neighbouring cores. A 10% threshold on the mean can fail on noise alone. Several practices make the signal usable.",[10,428,429,430,433,434,437,438,441],{},"Compare like with like: keep baselines per runner type, and on hosted runners prefer a baseline measured in the same workflow run — benchmark ",[14,431,432],{},"main"," and the branch back-to-back in one job — over a baseline saved days ago on different hardware. Compare the median or the minimum, not the mean, since outliers from interruptions inflate the mean. Set thresholds from observed noise: run the same commit ten times, look at the spread, and set the failure threshold comfortably above it. And isolate the benchmark job: no ",[14,435,436],{},"pytest-xdist",", no other test jobs on the same runner, garbage collection disabled during rounds with ",[14,439,440],{},"--benchmark-disable-gc"," when allocation noise dominates.",[249,443,445,518],{"className":444},[252],[254,446,262,451,262,454,262,457,262,460,262,463,262,469,262,473,262,476,262,479,262,484,262,489,262,496,262,501,262,507,262,511,262,515],{"viewBox":447,"role":257,"ariaLabelledBy":448,"xmlns":261},"0 0 800 236",[449,450],"pbc-t","pbc-d",[264,452,453],{"id":449},"Noise band versus regression threshold",[268,455,456],{"id":450},"A horizontal axis shows percentage change from the baseline median. Repeated runs of the same commit fall within a noise band of about plus or minus four percent. The failure threshold is set at ten percent, well outside the band, so real regressions fail while noise does not.",[290,458],{"x":292,"y":292,"width":293,"height":459,"rx":295,"fill":296},"236",[298,461,462],{"x":300,"y":301,"textAnchor":302,"fontSize":303,"fontWeight":304,"fill":305},"Set the threshold outside the noise",[68,464],{"x1":465,"y1":466,"x2":467,"y2":466,"stroke":305,"strokeWidth":468},"60","140","740","1.6",[298,470,472],{"x":300,"y":471,"textAnchor":302,"fontSize":325,"fill":305},"164","0%",[298,474,475],{"x":465,"y":471,"textAnchor":302,"fontSize":325,"fill":305},"-15%",[298,477,478],{"x":467,"y":471,"textAnchor":302,"fontSize":325,"fill":305},"+15%",[290,480],{"x":481,"y":482,"width":483,"height":301,"rx":281,"fill":342,"stroke":288,"strokeWidth":468},"309","112","182",[298,485,488],{"x":300,"y":486,"textAnchor":302,"fontSize":312,"fill":487},"104","#2a5f49","noise band ±4% (same commit, 10 runs)",[68,490],{"x1":491,"y1":492,"x2":491,"y2":493,"stroke":494,"strokeWidth":495},"627","70","150","#e07a5f","2.4",[298,497,500],{"x":491,"y":498,"textAnchor":302,"fontSize":312,"fontWeight":304,"fill":499},"62","#8f3d22","fail at +10% median",[502,503],"circle",{"cx":504,"cy":505,"r":282,"fill":506,"stroke":494,"strokeWidth":315},"680","126","#fbe9e3",[298,508,510],{"x":504,"y":509,"textAnchor":302,"fontSize":325,"fill":499},"196","real regression",[502,512],{"cx":513,"cy":505,"r":514,"fill":342,"stroke":288,"strokeWidth":315},"440","6",[298,516,517],{"x":513,"y":509,"textAnchor":302,"fontSize":325,"fill":487},"noisy but fine",[381,519,520],{},"Measure the noise first; a threshold inside the band fails on noise, and people stop trusting a gate that does that.",[26,522,524],{"id":523},"reading-the-report-table","Reading the report table",[10,526,527,528,531],{},"A pytest-benchmark run ends with a table that is easy to skim and easy to misread. Each row is one benchmark; the columns are Min, Max, Mean, StdDev, Median, IQR, Outliers, OPS (operations per second, the reciprocal of the mean), Rounds and Iterations. Benchmarks in the same ",[14,529,530],{},"group"," are sorted and compared within that group, so give related benchmarks a shared group name — \"index\", \"pricing\", \"serialise\" — to get meaningful relative comparisons.",[10,533,534,535,538,539,542,543,546,547,550],{},"Read the spread before the centre. If StdDev is a large fraction of the Mean, or the IQR is wide, the benchmark is noisy and any comparison built on it is weak. The Outliers column, written as ",[14,536,537],{},"a;b",", counts rounds beyond one standard deviation and beyond 1.5 IQR from the quartiles; many outliers suggest interference — garbage collection, other processes, thermal throttling — rather than a property of the code. Check Rounds too: a benchmark that only managed five rounds within ",[14,540,541],{},"max_time"," has a poorly estimated median, and raising ",[14,544,545],{},"--benchmark-max-time"," or ",[14,548,549],{},"--benchmark-min-rounds"," is worth doing before drawing conclusions.",[10,552,553],{},"When comparing runs, pytest-benchmark prints the saved and current results side by side, with the percentage change for each statistic in brackets. A change in Min with a stable Median usually means the best case moved — for example, a cache became warmer — while a change in Median with a stable Min usually means the typical case got slower. Both are real, but they point at different causes.",[249,555,557,622],{"className":556},[252],[254,558,262,563,262,566,262,569,262,572,262,575,262,579,262,585,262,588,262,592,262,595,262,598,262,601,262,605,262,608,262,612,262,615,262,618],{"viewBox":559,"role":257,"ariaLabelledBy":560,"xmlns":261},"0 0 800 226",[561,562],"pbr-t","pbr-d",[264,564,565],{"id":561},"What to read first in a benchmark row",[268,567,568],{"id":562},"A benchmark report row is split into three groups of columns read in order: first the spread columns StdDev, IQR and Outliers to judge noise, then Rounds to judge sample size, and only then the centre columns Median and Min for the actual comparison.",[290,570],{"x":292,"y":292,"width":293,"height":571,"rx":295,"fill":296},"226",[298,573,574],{"x":300,"y":301,"textAnchor":302,"fontSize":303,"fontWeight":304,"fill":305},"Spread, then sample size, then centre",[290,576],{"x":577,"y":578,"width":459,"height":364,"rx":320,"fill":506,"stroke":494,"strokeWidth":315},"26","56",[298,580,584],{"x":581,"y":582,"textAnchor":302,"fontSize":583,"fontWeight":304,"fill":305},"144","84","12.5","1 · spread",[298,586,587],{"x":581,"y":482,"textAnchor":302,"fontSize":325,"fill":305},"StdDev · IQR · Outliers",[298,589,591],{"x":581,"y":590,"textAnchor":302,"fontSize":325,"fill":499},"136","is it noisy?",[290,593],{"x":594,"y":578,"width":459,"height":364,"rx":320,"fill":313,"stroke":314,"strokeWidth":315},"282",[298,596,597],{"x":300,"y":582,"textAnchor":302,"fontSize":583,"fontWeight":304,"fill":305},"2 · sample size",[298,599,600],{"x":300,"y":482,"textAnchor":302,"fontSize":325,"fill":305},"Rounds · Iterations",[298,602,604],{"x":300,"y":590,"textAnchor":302,"fontSize":325,"fill":603},"#8a5a00","enough rounds?",[290,606],{"x":607,"y":578,"width":459,"height":364,"rx":320,"fill":342,"stroke":288,"strokeWidth":315},"538",[298,609,611],{"x":610,"y":582,"textAnchor":302,"fontSize":583,"fontWeight":304,"fill":305},"656","3 · centre",[298,613,614],{"x":610,"y":482,"textAnchor":302,"fontSize":325,"fill":305},"Median · Min",[298,616,617],{"x":610,"y":590,"textAnchor":302,"fontSize":325,"fill":487},"now compare",[298,619,621],{"x":300,"y":620,"textAnchor":302,"fontSize":312,"fill":305},"204","A precise-looking median from five noisy rounds is not a measurement.",[381,623,624],{},"Judging noise and sample size first stops a confident-looking number from settling a question it cannot answer.",[26,626,628],{"id":627},"choosing-what-to-benchmark","Choosing what to benchmark",[10,630,631],{},"A benchmark suite is only as useful as the operations it measures. The temptation is to benchmark every function that was ever slow; the result is a long, noisy suite that nobody reads. A better rule is to benchmark the operations whose speed someone would notice: the request handler behind the busiest endpoint, the batch job's inner transformation, the serialiser used on every message, the query builder on the reporting path. Five to fifteen well-chosen benchmarks usually cover a service. Each benchmark should use realistic input sizes, taken from production data shapes rather than toy fixtures, because algorithmic problems often only show at scale — a quadratic loop is invisible at ten items and dominant at ten thousand.",[10,633,634],{},"It also pays to benchmark at two sizes. A pair of benchmarks, one at a typical size and one at ten times that, shows the scaling behaviour directly: if the larger one takes ten times as long, the operation is linear; if it takes a hundred times as long, something quadratic has crept in. That signal survives CI noise much better than a single absolute number, because both measurements share the same machine and moment. A ratio check — assert that the large benchmark's median is less than, say, fifteen times the small one's — can even be written as an ordinary assertion over the saved JSON, giving a regression gate for algorithmic complexity that is almost immune to runner speed.",[26,636,638],{"id":637},"edge-cases-and-failure-modes","Edge cases and failure modes",[31,640,641,652,658,669,678],{},[34,642,643,647,648,651],{},[644,645,646],"strong",{},"Benchmarks in the normal test run."," They slow the suite and produce meaningless numbers under xdist. Keep them in a separate directory and use ",[14,649,650],{},"--benchmark-skip"," for regular runs.",[34,653,654,657],{},[644,655,656],{},"Optimised-away work."," If the result is unused, some code paths — especially in JIT-backed libraries — may skip work. Assert on the result, as in the first test.",[34,659,660,663,664,666,667,43],{},[644,661,662],{},"Fixture cost leaking in."," Work done inside the benchmarked callable counts. Build inputs in fixtures or pedantic ",[14,665,399],{},", not inside the function passed to ",[14,668,23],{},[34,670,671,674,675,677],{},[644,672,673],{},"Caches warming across rounds."," A function with an internal cache is fast after the first call, so the benchmark measures the cache hit. Clear the cache in ",[14,676,399],{}," if the cold path is what matters.",[34,679,680,683],{},[644,681,682],{},"Comparing across Python versions."," Saved results are tagged by interpreter; comparing 3.12 to 3.13 is legitimate only as an explicit experiment, not as a regression gate.",[26,685,687],{"id":686},"frequently-asked-questions","Frequently Asked Questions",[10,689,690,693,694,696,697,700],{},[644,691,692],{},"How does pytest-benchmark decide how many times to run my function?","\nIt calibrates: it times a single call, then chooses how many calls to group into each round so a round lasts at least the timer's resolution times a safety factor, and runs rounds until ",[14,695,541],{}," is reached or ",[14,698,699],{},"min_rounds"," is met.",[10,702,703,706,707,546,710,713,714,716,717,719,720,723],{},[644,704,705],{},"How do I fail CI when performance regresses?","\nSave a baseline with ",[14,708,709],{},"--benchmark-autosave",[14,711,712],{},"--benchmark-save",", then run later builds with ",[14,715,414],{}," and ",[14,718,418],{},", for example ",[14,721,722],{},"mean:10%"," to fail when the mean is more than ten percent slower.",[10,725,726,729],{},[644,727,728],{},"Why are my benchmark results so noisy in CI?","\nShared CI runners have variable CPU frequency, noisy neighbours and throttling. Compare against a baseline measured on the same runner type, use medians or minimums rather than means, and set thresholds wider than the observed noise.",[26,731,733],{"id":732},"related","Related",[31,735,736,742,749,756],{},[34,737,738,741],{},[48,739,740],{"href":50},"CPU Profiling with cProfile and py-spy"," — finding what to benchmark.",[34,743,744,748],{},[48,745,747],{"href":746},"\u002Fsystematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002Fline-level-profiling-with-line-profiler\u002F","Line-Level Profiling with line_profiler"," — locating the slow line.",[34,750,751,755],{},[48,752,754],{"href":753},"\u002Fsystematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002Fprofiling-async-code-with-yappi\u002F","Profiling Async Code with yappi"," — when the hot path is a coroutine.",[34,757,758,762],{},[48,759,761],{"href":760},"\u002Fsystematic-debugging-performance-profiling\u002Fmemory-profiling-with-tracemalloc\u002Fcatching-per-test-memory-growth-in-pytest\u002F","Catching Per-Test Memory Growth in pytest"," — the memory counterpart.",[10,764,765,766],{},"← Back to ",[48,767,740],{"href":50},[769,770,771],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}",{"title":62,"searchDepth":75,"depth":75,"links":773},[774,775,776,777,778,779,780,781,782],{"id":28,"depth":75,"text":29},{"id":54,"depth":75,"text":55},{"id":386,"depth":75,"text":387},{"id":422,"depth":75,"text":423},{"id":523,"depth":75,"text":524},{"id":627,"depth":75,"text":628},{"id":637,"depth":75,"text":638},{"id":686,"depth":75,"text":687},{"id":732,"depth":75,"text":733},"Measure Python performance reliably with pytest-benchmark: the benchmark fixture, rounds and warmup, pedantic mode, saving and comparing runs, failing on regressions, and keeping CI numbers trustworthy.","md",{"slug":786,"type":787,"breadcrumb":788,"datePublished":789,"dateModified":789,"faq":790,"howto":797},"benchmarking-with-pytest-benchmark","article","pytest-benchmark","2026-09-18",[791,793,795],{"q":692,"a":792},"It calibrates: it times a single call, then chooses how many calls to group into each round so a round lasts at least the timer's resolution times a safety factor, and runs rounds until max_time is reached or min_rounds is met.",{"q":705,"a":794},"Save a baseline with --benchmark-autosave or --benchmark-save, then run later builds with --benchmark-compare and --benchmark-compare-fail, for example mean:10% to fail when the mean is more than ten percent slower.",{"q":728,"a":796},"Shared CI runners have variable CPU frequency, noisy neighbours and throttling. Compare against a baseline measured on the same runner type, use medians or minimums rather than means, and set thresholds wider than the observed noise.",{"name":798,"description":799,"steps":800},"How to benchmark with pytest-benchmark","Write benchmark tests with the fixture, control setup and rounds, save baselines and compare against them with a regression threshold.",[801,804,807,810],{"name":802,"text":803},"Use the benchmark fixture","Call benchmark(func, *args) inside a test to time func and return its result.",{"name":805,"text":806},"Separate setup from measurement","Use benchmark.pedantic with a setup function when each round needs fresh input.",{"name":808,"text":809},"Save a baseline","Run with --benchmark-autosave on the main branch to store results.",{"name":811,"text":812},"Compare and fail on regressions","Run with --benchmark-compare and --benchmark-compare-fail=median:10% in CI.","\u002Fsystematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002Fbenchmarking-with-pytest-benchmark",{"title":5,"description":783},"systematic-debugging-performance-profiling\u002Fcpu-profiling-with-cprofile-and-py-spy\u002Fbenchmarking-with-pytest-benchmark\u002Findex","Ht9zrUJJHqmMV7gzXjTEhyBdCbULyF7bryTAGEvRr0Y",1789718767464]