Python Testing & Debugging Beyond unittestAdvanced Python testing playbooks

Debugging & Performance

Profiling a Running Process with py-spy

You have a Python worker pegging a CPU in production and restarting it to add instrumentation is not an option — you need to see what it is doing right now, from outside, without touching its code. py-spy does exactly this: it attaches to a running interpreter by PID, reads the call stacks through OS debugging facilities, and renders them as a live view, a flame graph, or a one-shot dump. This guide covers the three subcommands and the Linux ptrace_scope permission wall that blocks the first attempt on most hosts.

Prerequisites

  • py-spy >= 0.3.14 (pip install py-spy); 0.3.x is required for reliable --pid attach, dump, and --native.
  • A running CPython process you can identify by PID.
  • On Linux: CAP_SYS_PTRACE, root, or kernel.yama.ptrace_scope=0 to attach to a process you do not own. In Docker, run the container with --cap-add SYS_PTRACE.
  • macOS requires running py-spy with sudo.

Solution

First locate the worker's PID — pick the actual interpreter, not a shell or supervisor parent:

Bash
pgrep -f 'gunicorn.*worker' || ps -eo pid,cmd | grep '[p]ython'

py-spy top gives a live, auto-refreshing view of the hottest functions in that process:

Bash
# Live top-style view; %Own is self time, %Total includes subcalls.
py-spy top --pid 48291

py-spy record samples for a window and writes an interactive flame graph SVG:

Bash
# Sample for 30s and write a flame graph. --native unwinds C/Rust frames
# (e.g. numpy internals) so they are not collapsed into one built-in row.
py-spy record --pid 48291 --duration 30 --native --output flame.svg

py-spy dump prints a single snapshot of every thread's current stack — the fastest answer to "what is this hung process stuck on":

Bash
# One-shot stack dump of all threads; no sampling window needed.
py-spy dump --pid 48291

If attach is refused on Linux, the cause is almost always ptrace_scope:

Bash
# Option A (per session, needs root): allow any process to attach.
sudo sysctl -w kernel.yama.ptrace_scope=0
# Option B (no sysctl change): just run py-spy elevated.
sudo py-spy dump --pid 48291

Why this works

py-spy is written in Rust and reads the target process's memory through process_vm_readv (Linux), vm_read (macOS), or ReadProcessMemory (Windows), reconstructing Python frame objects without pausing the interpreter for more than a few microseconds per sample. Because it never injects code or imports a module into the target, it cannot corrupt application state and adds negligible overhead, which is what makes it safe to point at a live production worker. The ptrace_scope setting exists precisely because reading another process's memory is a privileged operation; relaxing it or granting CAP_SYS_PTRACE is what authorizes the attach.

Edge cases and failure modes

  • Wrong PID under a process manager. Gunicorn, uWSGI, and Celery fork worker processes; profiling the master shows almost no work. Attach to a worker PID, or use py-spy dump --pid <master> once to discover children.
  • Container PID namespaces. A PID inside a container differs from the host PID. Either run py-spy inside the container (with SYS_PTRACE added) or translate the PID before attaching from the host.
  • C extensions hide time. Without --native, time inside numpy, pandas, or a compiled dependency collapses into an opaque built-in frame. Add --native, which needs debug symbols to be most useful.
  • Idle / I/O-bound threads. A worker blocked on a socket shows wait-style frames. Use --idle to include idle threads in record, and remember py-spy measures wall-clock stacks, not CPU — cross-check with cProfile and pstats if you need exact CPU attribution.
  • Static or stripped Python builds. py-spy may fail to locate the interpreter struct (Failed to find python interpreter). Match the py-spy version to the CPython version and avoid heavily stripped builds.

Frequently Asked Questions

Do I need to restart my process to profile it with py-spy? No. py-spy attaches to an already-running interpreter by PID and reads its stacks from outside the process, so there is no restart and no code change. This is why it is safe for production workers.

Why does py-spy fail with Operation not permitted on Linux? Linux ptrace_scope restricts which processes may attach to another. Run py-spy with sudo, grant the binary CAP_SYS_PTRACE, or set kernel.yama.ptrace_scope to 0. In a container, add the SYS_PTRACE capability.

What does the --native flag do in py-spy?--native unwinds native C and Rust stack frames alongside Python frames, so time spent inside C extensions like numpy or a compiled library shows up instead of appearing as an opaque built-in call.

← Back to CPU Profiling with cProfile and py-spy