Finding Reference Cycles with gc and objgraph

tracemalloc reports that 40 MB was allocated at parsers.py:88, and the line is a perfectly ordinary object construction. That is the allocation site, not the bug. Something is still holding those objects, and finding it needs a different tool: one that walks backwards from an object to whatever refers to it.

Prerequisites

• Python 3.9+; gc is standard library, objgraph (3.6+) needs pip install objgraph and graphviz for images.
• A snapshot diff that already named the growing type — see finding memory leaks with tracemalloc snapshots.

Solution

Work in three steps: collect, count, then walk backwards.

import gc
import objgraph

gc.collect()                                    # 1. remove merely-uncollected noise
objgraph.show_growth(limit=10)                  # 2. which types grew since last call
# ParsedFrame     18422    +18422
# dict            24019     +6104
# Line            18422    +18422

frame = objgraph.by_type("ParsedFrame")[0]      # 3. take one and walk backwards
objgraph.show_backrefs([frame], max_depth=4, filename="backrefs.png")

show_growth is the fastest orientation: called twice around a workload, it prints only the types whose live-instance count increased. show_backrefs renders the chain from a sample object up to the roots, and the picture answers the question directly — a dict that belongs to a module, a list on a long-lived singleton, a closure cell in a registered callback.

When graphviz is unavailable, the same walk works in text with the standard library alone:

import gc

def who_holds(obj, depth: int = 2) -> None:
    """Print the containers referring to obj, up to `depth` levels back."""
    seen = {id(obj)}
    frontier = [(obj, 0)]
    while frontier:
        current, level = frontier.pop()
        if level >= depth:
            continue
        for ref in gc.get_referrers(current):
            if id(ref) in seen or isinstance(ref, type(who_holds.__code__)):
                continue                        # skip our own frames and code objects
            seen.add(id(ref))
            kind = type(ref).__name__
            size = len(ref) if hasattr(ref, "__len__") else "-"
            print(f"{'  ' * level}held by {kind} (len={size})")
            frontier.append((ref, level + 1))

The output usually identifies the container within two levels: a dict of length 18,000 is the cache nobody bounded, and a list growing by one per request is the registry nobody unregisters from.

Why this works

CPython frees an object when its reference count reaches zero. A cycle keeps every member's count above zero, so the cyclic collector exists to find groups that are unreachable from any root and reclaim them — which means a cycle among unreachable objects is a delay, not a leak. A genuine leak is an object still reachable from a live root: a module global, a class attribute, a running frame, a registered callback. gc.get_referrers walks the reference graph in reverse and therefore answers exactly that question, and objgraph is a presentation layer over the same data with cycle handling and filtering.

Uncollectable cycles

Most cycles are collected. A few are not, and gc.DEBUG_SAVEALL makes them visible by moving everything the collector could not free into gc.garbage instead of discarding it:

import gc

gc.set_debug(gc.DEBUG_SAVEALL)
run_workload()
gc.collect()
print(f"{len(gc.garbage)} uncollectable object(s)")
for obj in gc.garbage[:5]:
    print(type(obj).__name__, repr(obj)[:80])
gc.set_debug(0)
gc.garbage.clear()                      # otherwise this list is itself a leak

The historical cause — a __del__ method on a cycle member — was fixed in Python 3.4, so objects appearing here today usually come from a C extension that does not implement traversal correctly, or from a frame captured in a traceback that is itself stored somewhere. That second case is common and easy to fix: storing exc.__traceback__ on an object keeps every frame, and every frame's locals, alive indefinitely.

Note that DEBUG_SAVEALL is a diagnostic only. Leaving it enabled turns every collected cycle into retained memory, so it belongs in a scratch reproduction rather than in a service.

Choosing the right fix

The reference chain determines the fix, and there are only four shapes.

An unbounded cache becomes a bounded one: functools.lru_cache(maxsize=...), a TTL cache, or an explicit eviction policy. The cache key's cardinality is worth checking at the same time — a cache keyed on a request id never hits and is pure leak.

A registry with no unregister gets one, and the object that registered becomes responsible for removing itself, ideally through a context manager so the removal cannot be skipped.

A callback holding a strong reference becomes a weakref.ref or a WeakMethod, so the callback does not keep its subject alive. weakref.WeakValueDictionary and WeakSet are the collection forms.

A closure capturing more than it needs is rewritten to capture the small value rather than the large object — a common accident when a lambda references self only to read one attribute.

import weakref

class EventBus:
    def __init__(self):
        self._subscribers: weakref.WeakSet = weakref.WeakSet()

    def subscribe(self, handler) -> None:
        self._subscribers.add(handler)          # does not keep handler alive

    def publish(self, event) -> None:
        for handler in list(self._subscribers):  # copy: the set may shrink mid-iteration
            handler(event)

Edge cases and failure modes

• gc.get_referrers sees your own frame. The list includes the frame that called it, so filtering frames and the function's own locals is required or every object appears to be held by the debugger.
• objgraph.show_backrefs on a hot type is slow. The graph explodes; sample one object and cap max_depth at three or four.
• Interned and cached objects look retained. Small ints, short strings and empty tuples are shared process-wide; growth in int counts is almost never meaningful.
• A generator's frame keeps its locals alive until it is exhausted or closed, so an abandoned generator retains everything it referenced.
• WeakSet iteration during collection can shrink mid-loop. Copy to a list before iterating, as in the example above.
• Weak references cannot point at every type. list, dict, int and str do not support them; wrap the value in a small object when you need a weak reference to plain data.

A worked retention hunt

Putting the tools in order on a real symptom shows how few steps are needed. The symptom: a worker's RSS grows by roughly 30 MB per hour and never falls.

Step one — is it Python-level? Compare the process RSS with the tracemalloc total. If tracemalloc accounts for the growth, the object graph will explain it; if not, the growth is native or fragmentation and the tools on this page will find nothing.

import tracemalloc, psutil, os
tracemalloc.start(5)
run_workload(minutes=10)
current, peak = tracemalloc.get_traced_memory()
rss = psutil.Process(os.getpid()).memory_info().rss
print(f"traced {current/1e6:.1f} MB, peak {peak/1e6:.1f} MB, rss {rss/1e6:.1f} MB")

Step two — which type? objgraph.show_growth() around one cycle of the workload names it: ParsedFrame +18422 is unambiguous.

Step three — what holds it? gc.get_referrers on a sample, filtered to skip frames, reports a dict of 18,422 entries. One more level up names the module that owns the dict.

Step four — why is it unbounded? Reading that dict's key expression is usually the end of the investigation: keyed on a request id, so no key is ever reused and no entry is ever evicted.

The fix is a bounded cache, and the confirmation is the same measurement repeated: run the workload again and check that the traced total is flat. Four steps, three tools, and the only genuinely difficult part is knowing that step one exists — teams that skip it spend hours looking for a Python reference to memory that a C extension allocated. The investigation order matters more than the tools, so it is worth stating as a rule: measure whether the growth is Python-level, then name the type, then name the referrer, then fix ownership. Skipping straight to the referrer walk on a process whose growth is native produces hours of searching for a Python reference that does not exist.

Frequently Asked Questions

Does Python leak memory when objects form a cycle? Not permanently — the cyclic collector reclaims cycles whose members are unreachable. Cycles matter because collection is deferred and because an object reachable from a live root is never collected at all, cycle or not.

What is the difference between tracemalloc and objgraph?tracemalloc says where memory was allocated; objgraph says what still refers to it. Growth diagnosis needs both: the allocation site names the code that created the object, the reference chain names the code that will not let go.

Is gc.collect() in production a reasonable fix? Rarely. It hides growth caused by deferred collection but does nothing for objects held by a live reference, and it pauses the process. Use it in diagnosis to distinguish the two cases, not as a remedy. Is weakref always the right fix for a retaining reference? No. A weak reference is correct when the referrer genuinely does not own the object's lifetime — a cache, an observer list, a back-pointer. Where the referrer should keep the object alive, a weak reference converts a memory leak into a much worse bug: an object that disappears while still in use. Fix ownership first and reach for weakref only when the answer is that nothing there owns it.

Does gc.freeze() help with growth? It helps with copy-on-write memory in forked workers, not with leaks: it moves existing objects into a permanent generation so the collector stops touching them, which keeps shared pages shared after a fork. It has no effect on objects your code still references.

Related

• Memory profiling with tracemalloc — the allocation half of the investigation.
• Finding memory leaks with tracemalloc snapshots — producing the diff that names the type.
• Comparing tracemalloc snapshots to locate growth — reading a diff without being misled.
• Post-mortem debugging with pdb.pm() — why a stored traceback retains an entire stack.

← Back to Memory Profiling with tracemalloc