Joseph Wibowo
11/24/2025, 3:49 AMMarvin
11/24/2025, 3:49 AMMarvin
11/24/2025, 3:56 AMdocker ps -a
docker inspect <container_id> --format '{{.State.OOMKilled}} {{.State.ExitCode}}'
If OOMKilled is true, you need more memory (or lower usage).
- Watch memory in real time:
docker stats
If you’re using a Docker work pool (Prefect 3.x)
- Set a higher memory limit for the job container via job_variables on your deployment. Example:
from prefect import flow
@flow
def my_flow():
...
if __name__ == "__main__":
my_flow.deploy(
name="mem-test",
work_pool_name="my-docker-pool",
image="your/image:tag",
job_variables={
"mem_limit": "4g",
"memswap_limit": "6g" # optional; total = mem + swap
},
)
- Or set it per-deployment in prefect.yaml:
deployments:
- name: mem-test
entrypoint: flows/my_flow.py:my_flow
work_pool:
name: my-docker-pool
job_variables:
mem_limit: "4g"
memswap_limit: "6g"
- Or set defaults at the pool level by editing the base job template (add "mem_limit": "4g" in the job_configuration), via UI or CLI:
# See the default template
prefect work-pool get-default-base-job-template --type docker
# Update a pool with a JSON template file
prefect work-pool update --base-job-template base-job-template.json my-docker-pool
If you’re running your flow inside a general-purpose Docker container (not a Docker work pool)
- Increase Docker Desktop’s memory allocation (Preferences > Resources).
- If you start the container yourself, run it with a higher limit:
docker run --memory=4g --memory-swap=6g ...
Other mitigations
- Reduce memory usage in the flow (chunk data, stream instead of loading whole datasets, avoid large in-memory objects, write intermediates to disk).
- If using parallelism, lower concurrency so fewer tasks run at once.
Can you confirm:
- Are you using a Docker work pool to execute your flow, or running Prefect inside a general Docker container?
- Roughly how much memory your workload needs?
With that, I can suggest exact job_variables or template changes.