Abdou Seck
05/31/2026, 2:28 PMMarvin
05/31/2026, 2:28 PMMarvin
05/31/2026, 2:32 PMFlowRun data exposed via the client/API does not include a peak-memory field
- The OSS/server database schema does not store that UI metric as normal flow-run metadata
- The Cloud UI metric comes from telemetry, not a public flow-run attribute you can aggregate through the client
So if your goal is percentile aggregation, the practical options are:
1. Send telemetry to your own observability backend
- Prefect emits resource metrics via OpenTelemetry
- If you export those metrics to your own OTLP-compatible backend, you can compute avg / median / p95 / p99 there
2. Capture memory usage in the flow yourself
- Use something like psutil inside the flow / task runtime
- Record peak memory as an artifact, result, event, or external metric
- Then query and aggregate it yourself
If you want a simple Prefect-native workaround, you can record a custom artifact per run, e.g. peak RSS in MB:
from prefect import flow
from prefect.artifacts import create_markdown_artifact
import psutil
import os
import threading
import time
def sample_peak_memory(stop_event, interval=1.0):
process = psutil.Process(os.getpid())
peak = 0
while not stop_event.is_set():
rss = process.memory_info().rss / 1024 / 1024
peak = max(peak, rss)
time.sleep(interval)
return peak
@flow
def my_flow():
stop_event = threading.Event()
peak_holder = {"peak": 0}
def monitor():
process = psutil.Process(os.getpid())
while not stop_event.is_set():
rss = process.memory_info().rss / 1024 / 1024
peak_holder["peak"] = max(peak_holder["peak"], rss)
time.sleep(1.0)
thread = threading.Thread(target=monitor, daemon=True)
thread.start()
try:
# flow logic here
data = [i for i in range(1_000_000)]
return len(data)
finally:
stop_event.set()
thread.join(timeout=2)
create_markdown_artifact(
key="peak-memory-mb",
markdown=f"Peak memory: {peak_holder['peak']:.2f} MB"
)
Then you could query those artifacts and compute your own aggregates.
If you want, I can help with either of these:
- a Cloud API + artifacts approach to aggregate memory per flow/deployment
- an OpenTelemetry/Prometheus/Grafana approach for proper p95/p99 reporting
Relevant docs / references:
- Prefect Cloud: Prefect Cloud
- Prefect docs: Prefect docs
- OpenTelemetry exporter env vars: OTLP exporter configuration