Memory Profiling¶

SGN has built-in memory profiling that uses Python's tracemalloc module. Enable it via the SGNLOGLEVEL environment variable.

Enable Memory Profiling¶

export SGNLOGLEVEL="pipeline:MEMPROF"

Then run your pipeline normally:

from sgn import Pipeline, IterSource, NullSink

src = IterSource(name="src", source_pad_names=["H1"],
                 iters={"H1": range(100)})
snk = NullSink(name="snk", sink_pad_names=["H1"])

p = Pipeline()
p.connect(src, snk)
p.run()

Output¶

When MEMPROF is active, each pipeline iteration logs:

Top memory consumers: Which code lines allocate the most memory
Memory growth: How allocations change between iterations

Example output:

Top 10 memory allocations:
  /path/to/sgn/base.py:140: 1024 B
  /path/to/your_code.py:42: 512 B
  ...

Memory growth since last snapshot:
  /path/to/your_code.py:42: +256 B
  ...

Log Level Syntax¶

The SGNLOGLEVEL variable follows the pattern logger:LEVEL:

Value	Effect
`pipeline:MEMPROF`	Enable memory profiling for the pipeline logger
`pipeline:DEBUG`	Enable debug logging (pad-level frame details)
`pipeline:INFO`	Default - shows iteration counts and timing

Tips¶

Memory profiling adds overhead. Disable it in production.
Look for lines with consistent positive growth - these suggest memory leaks.
Combine with DEBUG logging to correlate memory usage with specific frames.