<@ULVA73B9P> When using an ECS push work pool, is ...
# ask-marvin
m
@Marvin When using an ECS push work pool, is it possible to configure the memory and CPU of the task that gets run in ECS?
m
thought for 397.4 seconds
Yes. With an ECS Push work pool you can set the CPU and memory for the ECS task via job variables. You can define defaults on the work pool’s base job template and/or override them per deployment. - Keys to use:
cpu
(CPU units; 1024 = 1 vCPU) and
memory
(MiB) - Where to set: - Work pool (Base job template): sets defaults for all deployments using the pool - Deployment (job variables): overrides the pool defaults for that deployment Example in prefect.yaml:
Copy code
deployments:
  - name: my-ecs-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-ecs-push-pool
    job_variables:
      cpu: 1024       # 1 vCPU
      memory: 2048    # 2 GiB
Notes: - Values must follow AWS ECS constraints, especially for Fargate where CPU/memory must be a valid combination. See AWS ECS task size combos. - You can also set these in the Prefect UI under the deployment’s Job Variables. Docs: - ECS integration guide: Prefect AWS ECS guide - Customizing job variables: Customize job variables
m
@Marvin How do the containers get generated for flows operating in a push work pool? Does Prefect generate them when deploying the flows? How are appropriate dependencies included?
m
thought for 198.7 seconds
Short answer: Prefect does not build container images for push work pools. You provide the image, and Prefect uses the provider API (ECS/Kubernetes/Docker) to run it with your deployment’s job variables. How it works - Push work pools (ECS/Kubernetes/Docker) submit a job directly to the provider using the image you specify in the work pool’s base job template or per-deployment
job_variables
. - Prefect supplies the container command/entrypoint to execute the flow run; you usually don’t need to change it. - For ECS specifically, Prefect will register or reuse a task definition that references your image and settings (cpu, memory, env, etc.) and then RunTask. How to include your code and dependencies You’ve got two common patterns: 1) Bake everything into the image (recommended for speed and reproducibility) - Build a Docker image that has: - Python + Prefect - Your project code (or package) and all Python deps - Any system-level deps (e.g., libpq, gcc, git if needed) - Push it to your registry (ECR/GCR/Docker Hub). - Set the image in the work pool base job template or per-deployment
job_variables.image
. Example deployment snippet:
Copy code
deployments:
  - name: my-flow
    entrypoint: flows/my_flow.py:flow
    work_pool:
      name: my-ecs-push-pool
    job_variables:
      image: myrepo/myflow:2025-10-08
      cpu: 1024
      memory: 2048
2) Pull source at runtime (image is “thin”) - Use
flow.from_source(...)
so your code is pulled into the container at runtime (e.g., from Git). - The image must still contain Prefect, Python, and any tools needed to fetch/install your code (git, build tools). - If you install Python deps at runtime (e.g.,
pip install -r requirements.txt
), expect longer cold starts and ensure the container has network access and credentials. Where to configure this - Work pool base job template: Set defaults like
image
,
cpu
,
memory
, env, etc. - Deployment `job_variables`: Override per-deployment. - You can also edit these in the UI under the deployment’s Job Variables. Good to know - Prefect does not do an image build for you at deploy time. - For ECS push pools, CPU and memory must follow AWS ECS valid combos (especially for Fargate). See AWS ECS task size. - Ensure your image includes Prefect that matches your deployment’s Prefect version. Docs - ECS push work pool guide: Prefect AWS ECS guide - Customizing job variables: Customize job variables If you share which push pool you’re using (ECS, Kubernetes, or Docker), I can provide a provider-specific example for the base job template and deployment config.
m
@Marvin I thought when flows were deployed Prefect built a Docker image for them automatically if an image was specified in the deploy function. If so, how does Prefect include all the necessary dependencies?
m
thought for 106.7 seconds
Great question — and you’re right to notice that deployments can build images for you. In Prefect 3.x: - If you call
prefect.deployments.deploy(..., image=..., build=True, push=True)
Prefect will build and push a Docker image as part of deployment. - Prefect does not “magically” discover and install your dependencies at run time. They get into the image through the Docker build step. How dependencies get included - If you provide a Dockerfile: Prefect uses it as-is. Your Dockerfile should install Prefect, your app code, and all dependencies (e.g.,
pip install -r requirements.txt
or
pip install .
for a package). - If you don’t provide a Dockerfile: Prefect uses
prefect.docker.DockerImage
(the
image
parameter can be a string or a
DockerImage
), which by default will try to build with a simple, auto-generated Dockerfile based on your project directory. You still need to follow common conventions (e.g., include a
requirements.txt
) so the build can install your deps. For anything non-trivial, we recommend supplying your own Dockerfile to control the environment. What happens at run time with push work pools - Push pools (ECS/Kubernetes/Docker) run the image you specified; Prefect does not install additional packages at run time. If your flow needs it, it must be baked into the image (or your image must be capable of pulling your code and installing deps as part of its entrypoint/startup). - If you use
flow.from_source(...)
, your container still needs Prefect (and optionally tools like
git
) installed; and if you plan to install Python deps on container startup, add that logic to your Dockerfile/entrypoint. Common patterns - Recommended: Build a project image with all deps pinned - Dockerfile installs Prefect + your code + deps -
deploy(..., image="myrepo/myapp:tag", build=True, push=True, work_pool_name="...")
- “Thin” image + pull source at runtime: - Image contains Prefect and tooling (e.g., git, pip); your startup installs deps - More flexible but slower cold starts; you own the startup script Example
Copy code
from prefect import flow
from prefect.deployments import deploy
from prefect.docker import DockerImage

@flow
def my_flow():
    ...

# Either pass a string "repo/name:tag" or a DockerImage with a Dockerfile
deploy(
    my_flow,
    work_pool_name="my-ecs-push-pool",
    image=DockerImage(name="myrepo/myapp", tag="2025-10-08", dockerfile="Dockerfile"),
    build=True,
    push=True,
)
Helpful docs - Customize job variables: Customize job variables - ECS push work pool guide: Prefect AWS ECS guide - Managing flow code in deployments: Store flow code - Prefect YAML/deployment config: prefect.yaml If you share your project setup (requirements.txt vs pyproject, private repos, etc.), I can suggest a concrete Dockerfile and deployment snippet tailored to your push pool.