Running Fuzz Targets in CI with Time Budgets

A fuzzer with no budget runs forever, which is why fuzzing so often stops at the experiment stage. The fix is to run two different things: a short, deterministic regression pass on every pull request, and a long discovery run on a schedule. They share a target and a corpus, and nothing else.

Prerequisites

A working target as built in writing your first Atheris fuzz target.
A CI system with caching and scheduled jobs.

Solution

The pull-request run replays the corpus and stops. -runs=0 executes each corpus entry exactly once and exits, which is the cheapest possible proof that no known-bad input crashes:

$ python fuzz_decode.py corpus/ -runs=0 -print_final_stats=1
Running 37 inputs 1 time(s) each.
Done 37 runs in 0 second(s)

The scheduled run searches. Give it a real budget, and keep whatever it learns:

name: fuzz
on:
  schedule: [{ cron: "0 3 * * *" }]
jobs:
  discover:
    runs-on: ubuntu-latest
    timeout-minutes: 90
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: corpus
          key: fuzz-corpus-${{ github.run_id }}
          restore-keys: fuzz-corpus-          # reuse the most recent corpus
      - run: pip install atheris -e .
      - run: |
          python fuzz_decode.py corpus/ \
            -max_total_time=3600 \
            -timeout=10 \
            -artifact_prefix=artifacts/ \
            -print_final_stats=1
      - if: failure()
        uses: actions/upload-artifact@v4
        with: { name: fuzz-crashes, path: artifacts/ }

Three flags carry the weight. -max_total_time bounds the run so the job cannot hang. -timeout turns a single slow input into a reported finding rather than a stalled job — an input that takes ten seconds in a decoder is a denial-of-service bug. And -artifact_prefix puts crash files somewhere the upload step can find them; without it they land in the working directory and are lost with the runner.

Only the left column belongs on the critical path; conflating the two is why fuzzing gets removed from pipelines.

Why this works

libFuzzer's corpus is a set of inputs, each of which reached some coverage the others did not. Replaying it is therefore a coverage-targeted regression suite that was assembled automatically — every entry exists because it exercised something new. Running it with -runs=0 is a deterministic pass over that set, with no mutation and no randomness, which is exactly the property a merge gate needs. The scheduled run does the opposite: it mutates, discovers and grows the corpus, and its output is the next regression suite rather than a verdict on this change.

The loop only closes if crashes become committed tests: a corpus in a cache is not a regression suite anyone can rely on.

Turning a crash into a test

A crash artefact is bytes. Committing it and adding a five-line pytest case makes the finding permanent, reviewable, and independent of the fuzzing infrastructure:

import pathlib
import pytest
from myapp.wire import decode_frame, FrameError

CRASHES = sorted((pathlib.Path(__file__).parent / "crashes").glob("*.bin"))

@pytest.mark.parametrize("path", CRASHES, ids=[p.stem for p in CRASHES])
def test_known_crashers_are_rejected_cleanly(path):
    data = path.read_bytes()
    # The contract: malformed input raises FrameError and nothing else.
    with pytest.raises(FrameError):
        decode_frame(data[0] if data else 0, data[1:])

This test runs in the normal suite, needs no Atheris, and states the contract explicitly. It also fails for the right reason if a refactor reintroduces the bug — with the original bytes, not a regenerated approximation.

Keep the crash directory small by minimising each artefact first. libFuzzer will do it:

$ python fuzz_decode.py -minimize_crash=1 -runs=10000 artifacts/crash-8f14e45f

A minimised input is usually a few bytes, which makes the committed fixture readable and the failure obvious.

Budgeting across several targets

Once there are five targets, the scheduled job needs a policy or it spends its whole budget on the first one. Two arrangements work.

Round-robin with equal shares is simplest: divide the total budget by the number of targets and run each in turn. It is fair and it under-serves the targets that are still finding things.

Coverage-weighted gives more time to targets whose coverage is still growing, which is where new bugs are. A short probe run reports cov:, and the scheduler allocates proportionally to the growth since the last run. This is more code than most teams need, but the underlying signal is worth watching manually: a target whose coverage has been flat for a month is either exhausted or blocked, and both deserve investigation rather than more hours.

Whichever policy, log what each target received and found. A fuzzing job whose output nobody reads is a job that will be deleted in the next pipeline cleanup, and the log line — "decode: 1h, 0 new crashes, cov 412 → 418" — is what justifies its cost.

Reporting that keeps the programme alive

A fuzzing job produces no output when it finds nothing, which is most of the time — and a job that is silent for six weeks is a job somebody deletes. A three-line summary per run turns invisible work into visible work.

$ python fuzz_decode.py corpus/ -max_total_time=3600 -print_final_stats=1 \
    | tee run.log
$ python - <<'EOF' >> summary.md
import re, pathlib
log = pathlib.Path("run.log").read_text()
cov = re.findall(r"cov: (\d+)", log)
execs = re.search(r"stat::number_of_executed_units:\s+(\d+)", log)
print(f"- decode: {execs.group(1) if execs else '?'} execs, "
      f"coverage {cov[0] if cov else '?'} → {cov[-1] if cov else '?'}, "
      f"{len(list(pathlib.Path('artifacts').glob('crash-*')))} crash(es)")
EOF

Posted to a channel or appended to a dashboard, that line answers the only questions anyone asks: is it still running, is it still learning, and did it find anything. Coverage that grew from 412 to 418 over an hour is a healthy target still exploring; coverage flat for three weeks is a target that has either exhausted its input space or is blocked, and both deserve a look.

Track two further numbers over time. Time to first crash on a deliberately broken build validates that the pipeline can still detect failures — a fuzzing job that cannot fail is worse than none. And corpus size trending flat while coverage is flat confirms exhaustion rather than a stuck fuzzer.

Finally, give the crashes an owner and a deadline like any other bug. A crash artefact committed as a regression test and left failing is a red build nobody trusts; a crash triaged, fixed and pinned is the outcome that justifies the whole programme.

Edge cases and failure modes

An unbounded run in a pull-request job blocks merges and teaches the team to distrust fuzzing; always pass a budget on the critical path.
Cache keys that never rotate accumulate a stale corpus. Use a run-id key with a prefix restore so the corpus carries forward but can be rebuilt.
Crash artefacts contain real input. If the fuzzed function processes user data in production, treat artefacts as potentially sensitive before committing them.
A flaky target produces phantom crashes. Non-determinism inside TestOneInput — a clock, a random seed, a shared cache — makes artefacts unreproducible; seed and isolate exactly as for property tests.
-timeout interacts with slow CI runners. A value tuned on a laptop will fire spuriously on a loaded runner; set it well above the worst legitimate input.
Corpus growth is not linear. A target can stall for hours and then discover a whole subtree; judge productivity over weeks, not runs. One scheduling nuance: run the discovery job at a fixed time and keep its duration constant. Coverage growth is only comparable between runs of equal length, and a job whose budget varies with runner availability produces a trend line nobody can interpret. A fixed hour, a fixed budget and a logged summary make the programme's output a time series rather than a sequence of anecdotes.

The last row is what keeps the first three funded: a silent job is a job that gets deleted.

Frequently Asked Questions

Should fuzzing block a pull request? Only the regression run should. A short run over the stored corpus proves no known input crashes and finishes in under a minute; discovery runs belong on a schedule where a multi-hour budget is acceptable.

Where should the corpus live? In CI cache for the working copy and in the repository for a small curated seed set. Caching keeps the accumulated coverage between runs; committing the seeds makes a fresh clone productive immediately.

What do I do with a crash artefact? Commit it as a regression fixture and add a pytest case that feeds it to the same function. The fuzzer found it once; the test keeps it found. Can the regression run replace an integration test? No, and it should not try. Replaying the corpus proves that a set of historically interesting inputs no longer crashes; it says nothing about correct output, because the oracle is only 'does not raise'. Keep the corpus replay as a robustness gate and let the ordinary test suite carry the behavioural assertions.

What if the corpus cache is lost? The next run rebuilds from the committed seeds, more slowly and with less coverage until it catches up. That is survivable, which is exactly why the seeds belong in the repository — a programme whose entire state lives in a CI cache is one cache eviction away from starting over.

Coverage-guided fuzzing with Atheris — the search loop this schedule feeds.
Writing your first Atheris fuzz target — the target being budgeted.
Coverage measurement and enforcement — the other coverage signal in the pipeline, and how it is gated.
Reproducing Hypothesis failures with @example — the same convert-the-finding discipline on the property-testing side.

← Back to Coverage-Guided Fuzzing with Atheris