Matthew McKay
10/08/2025, 1:16 AMMarvin
10/08/2025, 1:16 AMMarvin
10/08/2025, 1:23 AMcpu (CPU units; 1024 = 1 vCPU) and memory (MiB)
- Where to set:
- Work pool (Base job template): sets defaults for all deployments using the pool
- Deployment (job variables): overrides the pool defaults for that deployment
Example in prefect.yaml:
deployments:
- name: my-ecs-deployment
entrypoint: flows/my_flow.py:my_flow
work_pool:
name: my-ecs-push-pool
job_variables:
cpu: 1024 # 1 vCPU
memory: 2048 # 2 GiB
Notes:
- Values must follow AWS ECS constraints, especially for Fargate where CPU/memory must be a valid combination. See AWS ECS task size combos.
- You can also set these in the Prefect UI under the deployment’s Job Variables.
Docs:
- ECS integration guide: Prefect AWS ECS guide
- Customizing job variables: Customize job variablesMatthew McKay
10/08/2025, 1:46 AMMarvin
10/08/2025, 1:46 AMMarvin
10/08/2025, 1:49 AMjob_variables.
- Prefect supplies the container command/entrypoint to execute the flow run; you usually don’t need to change it.
- For ECS specifically, Prefect will register or reuse a task definition that references your image and settings (cpu, memory, env, etc.) and then RunTask.
How to include your code and dependencies
You’ve got two common patterns:
1) Bake everything into the image (recommended for speed and reproducibility)
- Build a Docker image that has:
- Python + Prefect
- Your project code (or package) and all Python deps
- Any system-level deps (e.g., libpq, gcc, git if needed)
- Push it to your registry (ECR/GCR/Docker Hub).
- Set the image in the work pool base job template or per-deployment job_variables.image.
Example deployment snippet:
deployments:
- name: my-flow
entrypoint: flows/my_flow.py:flow
work_pool:
name: my-ecs-push-pool
job_variables:
image: myrepo/myflow:2025-10-08
cpu: 1024
memory: 2048
2) Pull source at runtime (image is “thin”)
- Use flow.from_source(...) so your code is pulled into the container at runtime (e.g., from Git).
- The image must still contain Prefect, Python, and any tools needed to fetch/install your code (git, build tools).
- If you install Python deps at runtime (e.g., pip install -r requirements.txt), expect longer cold starts and ensure the container has network access and credentials.
Where to configure this
- Work pool base job template: Set defaults like image, cpu, memory, env, etc.
- Deployment `job_variables`: Override per-deployment.
- You can also edit these in the UI under the deployment’s Job Variables.
Good to know
- Prefect does not do an image build for you at deploy time.
- For ECS push pools, CPU and memory must follow AWS ECS valid combos (especially for Fargate). See AWS ECS task size.
- Ensure your image includes Prefect that matches your deployment’s Prefect version.
Docs
- ECS push work pool guide: Prefect AWS ECS guide
- Customizing job variables: Customize job variables
If you share which push pool you’re using (ECS, Kubernetes, or Docker), I can provide a provider-specific example for the base job template and deployment config.Matthew McKay
10/08/2025, 2:06 AMMarvin
10/08/2025, 2:06 AMMarvin
10/08/2025, 2:08 AMprefect.deployments.deploy(..., image=..., build=True, push=True) Prefect will build and push a Docker image as part of deployment.
- Prefect does not “magically” discover and install your dependencies at run time. They get into the image through the Docker build step.
How dependencies get included
- If you provide a Dockerfile: Prefect uses it as-is. Your Dockerfile should install Prefect, your app code, and all dependencies (e.g., pip install -r requirements.txt or pip install . for a package).
- If you don’t provide a Dockerfile: Prefect uses prefect.docker.DockerImage (the image parameter can be a string or a DockerImage), which by default will try to build with a simple, auto-generated Dockerfile based on your project directory. You still need to follow common conventions (e.g., include a requirements.txt) so the build can install your deps. For anything non-trivial, we recommend supplying your own Dockerfile to control the environment.
What happens at run time with push work pools
- Push pools (ECS/Kubernetes/Docker) run the image you specified; Prefect does not install additional packages at run time. If your flow needs it, it must be baked into the image (or your image must be capable of pulling your code and installing deps as part of its entrypoint/startup).
- If you use flow.from_source(...), your container still needs Prefect (and optionally tools like git) installed; and if you plan to install Python deps on container startup, add that logic to your Dockerfile/entrypoint.
Common patterns
- Recommended: Build a project image with all deps pinned
- Dockerfile installs Prefect + your code + deps
- deploy(..., image="myrepo/myapp:tag", build=True, push=True, work_pool_name="...")
- “Thin” image + pull source at runtime:
- Image contains Prefect and tooling (e.g., git, pip); your startup installs deps
- More flexible but slower cold starts; you own the startup script
Example
from prefect import flow
from prefect.deployments import deploy
from prefect.docker import DockerImage
@flow
def my_flow():
...
# Either pass a string "repo/name:tag" or a DockerImage with a Dockerfile
deploy(
my_flow,
work_pool_name="my-ecs-push-pool",
image=DockerImage(name="myrepo/myapp", tag="2025-10-08", dockerfile="Dockerfile"),
build=True,
push=True,
)
Helpful docs
- Customize job variables: Customize job variables
- ECS push work pool guide: Prefect AWS ECS guide
- Managing flow code in deployments: Store flow code
- Prefect YAML/deployment config: prefect.yaml
If you share your project setup (requirements.txt vs pyproject, private repos, etc.), I can suggest a concrete Dockerfile and deployment snippet tailored to your push pool.