Seeding random and NumPy for Reproducible Tests

A test that exercises randomised code — shuffling, sampling, jitter, simulated data — has two failure modes. If the randomness is not controlled, the test passes on most runs and fails on the one where the generator produced an edge case, and nobody can reproduce it. If the randomness is controlled with a single global random.seed(42), the test is reproducible until someone adds a test before it that also draws random numbers, at which point every later test sees a different sequence and some of them fail for reasons unrelated to any change.

The robust arrangement has three parts: randomised code receives an explicit generator rather than calling module-level functions, each test derives its own seed from a printed session seed and its node id, and assertions check properties of the output rather than exact values wherever possible. With those in place, a failure caused by an unlucky draw is reproducible with one command and stays put when the suite changes around it.

Prerequisites

• Python 3.9+; random in the standard library, NumPy 1.17+ for default_rng.
• pytest >= 8.0, and optionally pytest-randomly.
• The same approach for fake data is in generating reproducible fake data with Faker.

Solution

import hashlib
import os
import random

import numpy as np
import pytest


@pytest.fixture(scope="session")
def session_seed(request):
    seed = int(os.environ.get("TEST_SEED", random.SystemRandom().randrange(2**32)))
    print(f"\ntest session seed: {seed}  (re-run with TEST_SEED={seed})")
    return seed


def _derive(session_seed: int, nodeid: str) -> int:
    digest = hashlib.sha256(f"{session_seed}:{nodeid}".encode()).digest()
    return int.from_bytes(digest[:8], "big")


@pytest.fixture
def rng(request, session_seed) -> random.Random:
    # An explicit, per-test generator: order-independent and parallel-safe.
    return random.Random(_derive(session_seed, request.node.nodeid))


@pytest.fixture
def np_rng(request, session_seed) -> np.random.Generator:
    return np.random.default_rng(_derive(session_seed, request.node.nodeid))

def sample_customers(customers, k, *, rng: random.Random):
    return rng.sample(customers, k)          # randomness is a parameter, not a global


def test_sample_never_repeats_a_customer(rng):
    customers = [f"cus_{i}" for i in range(100)]
    chosen = sample_customers(customers, 10, rng=rng)

    assert len(chosen) == len(set(chosen)) == 10     # a property, not exact values
    assert set(chosen) <= set(customers)

Why this works

A random.Random instance carries its own state. Seeding it affects nothing else in the process, and nothing else can advance it, so the sequence a test sees depends only on the seed it was created with. Deriving that seed by hashing the session seed with the node id gives every test a distinct, stable starting point: the same test always gets the same stream for a given session seed, regardless of which tests ran before it, whether it was selected alone with -k, or which pytest-xdist worker executed it.

NumPy's default_rng returns a Generator with the same property, and it is the interface NumPy recommends for new code. The legacy np.random.seed controls a single global RandomState shared by everything in the process, which has exactly the ordering problem described above. It also means that a library you call — a data-augmentation routine, a sampling utility — can silently consume numbers from the same global stream, so the values your own code sees depend on what the library did first. Explicit generators remove that coupling entirely: the library gets its own generator, your code gets its own, and neither affects the other.

Edge cases and failure modes

• Library code using module-level random. Code you cannot change that calls random.random() directly still reads global state. Seed the global generator per test as well — pytest-randomly does this — as a fallback, not a substitute.
• Python's hash() for deriving seeds. String hashing is salted per process unless PYTHONHASHSEED is set, so seeds derived with hash() differ between runs and between workers. Use hashlib.
• Exact-value assertions. assert rng.random() == 0.6394… breaks when the algorithm or the library version changes. Prefer invariants: bounds, uniqueness, distribution properties.
• Threads sharing one generator. random.Random is not safe for concurrent use without a lock. Give each thread its own instance, derived from the test's seed.
• Cryptographic randomness. secrets and os.urandom are deliberately unseedable. Code that uses them for non-security purposes should take a generator instead; code that uses them for security should be tested by property, not by value.

Asserting on randomised output

The hardest part of testing randomised code is choosing what to assert. Pinning exact values makes the test a snapshot of one library version's algorithm; asserting nothing makes it useless. The middle ground is properties that hold for every valid output, and most randomised functions have several.

Structural properties are the easiest: a sample has the requested size, contains no duplicates, draws only from the population; a shuffle is a permutation of its input; jittered delays fall within their bounds. These never depend on the seed and survive any change to the underlying algorithm.

Statistical properties need more care but catch different bugs. Drawing ten thousand values from a generator that should be uniform on [0, 1) and checking the mean lies within a generous band around 0.5 catches an off-by-one that biases every draw. Seeding makes such a test deterministic — it either always passes or always fails for a given seed — and choosing a wide tolerance keeps it from failing on legitimate variation.

Relational properties compare two runs: the same seed gives the same output, different seeds give different output. These pin down that randomness is actually wired through, which is the property that breaks when someone replaces an injected generator with a call to the module-level function.

Wiring generators through an application

Injecting a generator into one function is easy; doing it across an application needs a convention, or the parameter gets threaded through a dozen call sites by hand. The pattern that scales is the same one used for clocks: services receive a generator in their constructor, defaulting to a fresh unseeded instance, and pass it down to the functions they call.

In production nothing changes — each service creates its own generator, which is as random as the module-level functions and has no shared state. In tests the factory that builds services passes a generator derived from the test's seed, so every random decision the service makes is reproducible. And a reviewer reading a service's constructor sees immediately that its behaviour involves randomness, which is information a hidden random.choice deep in a helper never gives.

The convention also makes one class of bug impossible: two components accidentally sharing a generator and interfering with each other's sequences. Each owns its instance, derived from a common seed when a test needs determinism, and independent otherwise.

Replaying a failure

The printed session seed is what turns a flaky-looking failure into a reproducible one. When CI reports a failure in test_sample_never_repeats_a_customer, the log header carries the seed, and re-running locally with TEST_SEED=<value> pytest -k test_sample_never_repeats_a_customer produces exactly the same random values for exactly that test, because the derivation depends only on the seed and the node id.

That makes the investigation ordinary: run it, see it fail, add a print or a breakpoint, fix the code. Once fixed, the specific input that exposed the bug is worth preserving as an explicit, non-random test case — the seed will change on the next run, and a regression test that depends on reproducing a particular random draw is fragile. A plain test with the problematic input written out keeps the bug fixed regardless of future seeds.

For randomised code where finding such inputs matters — shuffles that must preserve some property, samplers that must respect constraints — property-based testing with Hypothesis is the stronger tool, because it searches for failing inputs deliberately and shrinks them to minimal cases, as covered in property-based and fuzz testing strategies. Seeded randomness in ordinary tests is for determinism; Hypothesis is for exploration.

Frequently Asked Questions

Is calling random.seed() at the top of a test enough? It works for code using the module-level functions, but it is global state: any other code consuming random numbers in between shifts the sequence, and parallel workers each have their own global generator. Passing an explicit random.Random or NumPy Generator instance is more robust and makes the dependency visible.

What does pytest-randomly do with seeds? It shuffles test order and reseeds random, NumPy's legacy global generator and Faker at the start of each test from a seed printed in the header. Re-running with -p randomly -p "randomly_seed=N" reproduces both the order and the random values.

Should NumPy code use np.random.seed? No. np.random.seed controls the legacy global RandomState. New code should use np.random.default_rng(seed), which returns an independent Generator with better statistical properties, and pass that generator to the functions that need randomness.

Related

• Controlling Time and Randomness in Tests — the wider approach to non-deterministic inputs.
• Generating Reproducible Fake Data with Faker — the same per-test derivation for Faker.
• Bisecting Test-Order Dependencies — what shuffled order reveals.
• Generating DataFrames and Arrays with Hypothesis Extras — exploring random inputs rather than fixing them.

← Back to Controlling Time and Randomness in Tests