Tristan Moeller
09/26/2025, 11:18 AMMarvin
09/26/2025, 11:18 AMMarvin
09/26/2025, 11:25 AMprefecthq/prefect:3.4.20-python3.11
)?
- Python version in the image?
- Version of prefect-kubernetes
(if installed) or any other Prefect extras?
- Deployment/Helm
- Are you using the Prefect Helm chart? If so, which chart version and values for the worker?
- Logs and timing
- Does the memory spike before or after the worker prints that it has started and is polling the pool?
- Any unusual log lines at DEBUG on startup?
- Does downgrading the worker back to 3.3.6 immediately stop the spike?
Fast isolation steps you can try now
1) Quick bisect to find the first bad version
- Try 3.4.0 and 3.4.10 for the worker image. If 3.3.6 is fine and 3.4.20 spikes, this will tell us where it started.
2) Run the worker with DEBUG logging to see if we can spot where it blows up
prefect worker start --pool <your-kubernetes-pool> --log-level DEBUG
3) Minimal local repro (optional)
- If you can, run a worker container locally with the same tag and env (pointed at your Cloud/Server) to see if the spike reproduces off-cluster. That helps us determine if this is cluster/client related vs. package-level.
4) Lightweight memory trace during startup (optional)
- If you can wrap the start command to sample memory during the first 60s, it helps narrow down the exact moment it spikes:
python - <<'PY'
import os, time, subprocess, tracemalloc, shutil
tracemalloc.start(25)
p = subprocess.Popen(["prefect","worker","start","--pool", os.environ.get("PREFECT_POOL")])
for i in range(60):
time.sleep(1)
current, peak = tracemalloc.get_traced_memory()
print(f"[{i:02d}s] current={current/1e6:.1f}MB peak={peak/1e6:.1f}MB", flush=True)
p.wait()
PY
Immediate workaround
- If you’re blocked, pin the worker image back to 3.3.6 while we investigate.
If you can provide the details above (image tag, startup command, worker type/pool, logs around startup, and whether a bisect isolates a specific 3.4.x), I’ll dig in further and escalate. If we confirm a regression, I’ll open an issue with a minimal repro to get it addressed quickly.