Short answer: don’t put your data science stack into the Prefect image. Orchestrate SageMaker with a small Prefect image and let SageMaker pull and run your 7GB ECR image. Prefect 3.x doesn’t need to build/push that big image at deploy time. Two solid approaches: 1) Orchestrate SageMaker from a small Prefect image (recommended) - Your Prefect deployment can be tiny (Prefect + boto3 + prefect-aws). The large ECR image is only referenced by SageMaker’s Processing job. - This completely avoids long Prefect image builds/pushes and startup pulls. Example flow (boto3) with AwsCredentials block:

from prefect import flow, task
from prefect_aws import AwsCredentials

@task
def submit_processing(job_name, image_uri, role_arn, inputs, outputs):
    client = AwsCredentials.load("aws").get_boto3_session().client("sagemaker")
    resp = client.create_processing_job(
        ProcessingJobName=job_name,
        RoleArn=role_arn,
        ProcessingInputs=inputs,
        ProcessingOutputConfig=outputs,
        ProcessingResources={"ClusterConfig": {"InstanceCount": 1, "InstanceType": "ml.m5.xlarge", "VolumeSizeInGB": 50}},
        AppSpecification={"ImageUri": image_uri},
    )
    return resp["ProcessingJobArn"]

@flow
def sagemaker_processing_flow():
    return submit_processing(
        job_name="my-processing-job",
        image_uri="<http://123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/my-processor:sha-abcdef|123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/my-processor:sha-abcdef>",
        role_arn="arn:aws:iam::123456789012:role/MySageMakerRole",
        inputs=[...],
        outputs={...},
    )

if __name__ == "__main__":
    sagemaker_processing_flow.deploy(
        name="sm-processing",
        work_pool_name="ecs-ec2-pool",
        # Use a small image for the Prefect runtime
        image="your-small-prefect-image:tag",
        build=False,
        push=False,
        job_variables={"image_pull_policy": "IfNotPresent"},
    )

- Use immutable tags or digests for the SageMaker image (

:sha-...

) to benefit from caching and avoid surprises. - Ensure your SageMaker execution role can pull from ECR (ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer). - Docs: boto3 create_processing_job, Prefect AWS integration 2) If you really need heavy libs inside the Prefect/ECS container - Pre-build and push your large ECR image outside Prefect, then reference it in `.deploy()`:

my_flow.deploy(
  name="ml-deployment",
  work_pool_name="ecs-ec2-pool",
  image="<http://123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/ml-image:v1|123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/ml-image:v1>",
  build=False,
  push=False,
  job_variables={"image_pull_policy": "IfNotPresent"}
)

- Keep your flow code out of the image so you don’t rebuild: use

flow.from_source(...).deploy(...)

with Git/S3 for code retrieval. - On your single ECS EC2 host, pre-pull the image once (e.g., at boot via user-data) so tasks start fast and don’t re-pull. - Use multi-stage builds and slim bases, pin immutable tags, and avoid

:latest

. Extras for your setup (single EC2 in ap-southeast-1, ECS launch type EC2): - Prefect Server + worker can run on the same box. For the ECS work pool, set small defaults for CPU/mem for the Prefect container; the heavy compute happens in SageMaker anyway. - Set

image_pull_policy: IfNotPresent

on the Prefect deployment to avoid re-pulls on that host. - If you do use a large ECS image, warm the cache with

docker pull

and use immutable tags; consider ECR repo policies and lifecycle rules. Quick sanity check

- Are you planning to run the actual processing inside SageMaker (preferred), or inside the Prefect/ECS container? If it’s SageMaker, the first pattern is ideal and you won’t be waiting on Prefect to build/push a 7GB image. If you need help wiring the AwsCredentials block or your ECS EC2 work pool, I can sketch those steps next.

Great question! Here’s how to deploy with a prebuilt image using prefect.yaml in Prefect 3.x, plus a minimal example you can copy. Quick plan - Create a Docker work pool - Start a worker on your EC2 - Use a prefect.yaml that references your prebuilt image and includes a minimal pull step - Deploy with prefect deploy 1) Create a Docker work pool and start a worker

prefect work-pool create my-docker-pool --type docker
prefect worker start -p my-docker-pool

2) Example: prebuilt image (code baked into the image) Use this if your flow code and dependencies are inside the image. The required pull step can just set the working directory.

prefect-version: "3.0"
name: my-project

# No build/push because the image is prebuilt
build: null
push: null

# Minimal required pull step: set working directory inside the container
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: "/app"  # wherever your code lives in the image

deployments:
  - name: my-flow-prod
    description: "Runs using a prebuilt Docker image"
    entrypoint: "flows/main.py:my_flow"  # relative to /app
    parameters:
      some_param: "value"

    # Example schedule (optional)
    # schedule:
    #   cron: "0 * * * *"
    #   timezone: "Asia/Singapore"

    work_pool:
      name: my-docker-pool
      job_variables:
        image: "<http://123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/my-flows:1.0.0|123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/my-flows:1.0.0>"
        env:
          # Example: supply DB URL via a Prefect secret you created
          # Create this with: `prefect block create -t secret -n rds-url`
          DATABASE_URL: "{{ prefect.blocks.secret.rds-url }}"
          PREFECT_LOGGING_LEVEL: "INFO"

Then deploy:

prefect deploy -n my-flow-prod

Notes for your EC2 + RDS setup - Your worker will run on the EC2 instance in ap-southeast-1 and pull the prebuilt image. Make sure the EC2 has permissions to pull from your registry (e.g., ECR auth or Docker Hub creds). - Since you’re storing DB credentials in Prefect, the simplest pattern is a single secret containing the full Postgres URL and injecting it via env (as shown above). - You can also use a SQLAlchemy connector block if you prefer; then pass connection params in your flow code via block.load(...). 3) Alternative: prebuilt image but pull code from Git at runtime If you don’t bake code into the image, add a git clone pull step and set the working directory:

prefect-version: "3.0"
name: my-project

build: null
push: null

pull:
  - prefect.deployments.steps.git_clone:
      repository: "<https://github.com/my-org/my-repo.git>"
      branch: "main"
      # access_token optional if private:
      # access_token: "{{ prefect.blocks.github-credentials.gh-creds }}"
  - prefect.deployments.steps.set_working_directory:
      directory: "./my-repo"

deployments:
  - name: my-flow-prod
    entrypoint: "flows/main.py:my_flow"
    work_pool:
      name: my-docker-pool
      job_variables:
        image: "<http://123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/my-flows:1.0.0|123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/my-flows:1.0.0>"
        env:
          DATABASE_URL: "{{ prefect.blocks.secret.rds-url }}"

Deploy:

prefect deploy -n my-flow-prod

4) If you want to build during deploy (not prebuilt) Only if you change your mind and want Prefect to build/push the image. Requires the docker integration installed. ``` prefect-version: "3.0" name: my-project build: - prefect_docker.deployments.steps.build_docker_image: id: build_image image_name: "123456789012.dkr.ecr.ap-southeast-1.amazonaws.com/my-flows" dockerfile: "Dockerfile" tag: "1.0.1" ignore_cache: false push: - prefect_docker.deployments.steps.push_docker_image: image_name: "{{ build_image.image_name }}" tag: "{{ build_image.tag }}" # credentials: "{{ prefect.blocks.docker-registry-credentials.ecr }}" # if needed pull: - prefect.deployments.steps.set_working_directory: directory: "/app" deployments: - name: my-flow-dev entrypoint: "flows/main.py:my_flow"

work_pool: name: my-docker-pool job_variables: image: "{{ build_image.image_name }}:{{ build_image.tag }}"

Deploy:

# If you use build/push steps, ensure docker extras are available in your environment # e.g. via uv: # uv run --with 'prefect[docker]' prefect deploy -n my-flow-dev prefect deploy -n my-flow-dev

CLI commands you’ll use
- Initialize a template (optional):

prefect init

- Deploy from YAML:

prefect deploy -n <deployment-name>

- Deploy all in the file:

prefect deploy --all

- Create a work pool:

prefect work-pool create my-docker-pool --type docker

- Start a worker:

prefect worker start -p my-docker-pool ``` Docs for more detail - prefect.yaml (3.x) - Docker deployments - Customize job variables Want me to tailor the YAML to your exact image name and code layout? Also, are you baking the code into the image or pulling from Git at runtime?

Thanks for sharing the YAML and the error. The crash is happening because the worker starts your container with the command “prefect flow-run execute”, but the container image does not have the Prefect CLI in its PATH. Hence: exec: "prefect": executable file not found in $PATH. What to fix 1) Make sure your image installs Prefect 3.4.11 and exposes the “prefect” console script in PATH - If you’re using a Python base (e.g., python:3.11-slim), a simple pip install is enough. - If you’re using a conda-based image (e.g., some SageMaker images), ensure PATH includes the conda bin (e.g., /opt/conda/bin) so the “prefect” command is found. - If you set a custom ENTRYPOINT, ensure it preserves CMD/command args (exec "$@"). Example Dockerfile that works

FROM python:3.11-slim

# Install build tools if needed for your deps
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc build-essential git && \
    rm -rf /var/lib/apt/lists/*

# Install Prefect (and any extras you need)
RUN pip install --no-cache-dir "prefect==3.4.11"

# Optional: AWS extras if you use them in your flow code
# RUN pip install --no-cache-dir "prefect-aws>=0.4.0"

# Copy code into the container
WORKDIR /opt/prefect/src
COPY . /opt/prefect/src

# Do not override CMD/ENTRYPOINT; Prefect workers inject the command:
# `prefect flow-run execute`

If you must stick to a conda-based image (e.g., a SageMaker image), ensure:

ENV PATH="/opt/conda/bin:${PATH}"
RUN pip install --no-cache-dir "prefect==3.4.11"

2) Fix the YAML templating and Slack link artifacts Your pasted YAML shows Slack link formatting and HTML entities; these must not be present in the actual file. Also fix the image templating line. - Replace things like: -

requires: prefect-docker>=0.3.1

->

requires: prefect-docker>=0.3.1

-

image: <http://...|...>

-> just the plain image string (no angle brackets) -

image: <...:{{

-> should be

image: "<http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:>{{ git_sha.stdout }}"

Here’s your YAML corrected (keeping your structure) ``` name: src prefect-version: 3.4.11 build: - prefect.deployments.steps.utility.run_shell_script: id: git_sha script: git rev-parse --short HEAD - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.3.1 image_name: 010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect tag: "{{ git_sha.stdout }}" additional_tags: - "{{ prefect.variables.app_env }}" dockerfile: Dockerfile.prefect_flows_run ignore_cache: true push: - prefect_docker.deployments.steps.push_docker_image: requires: prefect-docker>=0.3.1 image_name: "{{ build_image.image_name }}" tag: "{{ git_sha.stdout }}" additional_tags: - "{{ prefect.variables.app_env }}" pull: - prefect.deployments.steps.set_working_directory: directory: /opt/prefect/src deployments: - name: "{{ prefect.variables.app_env }}" version: "{{ build_image.tag }}" tags: - "{{ prefect.variables.app_env }}" - ebay entrypoint: extraction/ebay/flow.py:run parameters: seller_id: "d480fd76-f59e-4ea0-95cb-4e22db0164e0" selling_partner_id: "ed442813-afe7-4026-bc14-190ca41bfb23" data_cycle: "origination" work_pool: name: ds-worker work_queue_name: default job_variables: image: "010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{ git_sha.stdout }}" cpu: 2048 memory: 8192 env: APP_ENV: "{{ prefect.variables.app_env }}" SOURCE: "ebay" - name: forecast description: "test ml flow" entrypoint: extraction/amazon/forecast.py:run work_pool: name: ds-worker work_queue_name: default job_variables: cpu: 4096 memory: 16384 image: "010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/sagemaker-forecasting:prefect"

Optional quick workaround (not recommended for prod)
If you need an immediate test and can’t rebuild the image, you can override the container command in job variables to install Prefect at startup:

job_variables: image: "your-image" command: - /bin/sh - -lc - | pip install --no-cache-dir "prefect==3.4.11" \ && prefect flow-run execute ``` This is slower and brittle but can confirm the root cause. A couple more checks - Worker type: The presence of cpu/memory in vCPU/MiB suggests an ECS pool. Please confirm ds-worker is an ECS work pool. If you’re actually using a Docker work pool on your single EC2, these ECS-specific fields won’t apply; use Docker job variables instead. - ECR auth: Ensure your worker environment can pull from ECR in ap-southeast-1. - Code path: Your pull sets /opt/prefect/src; make sure that’s where your flow code lives inside the image. Docs - Docker deployments: https://docs.prefect.io/v3/how-to-guides/deployment_infra/docker/ - YAML deployments: https://docs.prefect.io/v3/how-to-guides/deployments/prefect-yaml/ - Customize job variables: https://docs.prefect.io/v3/how-to-guides/deployments/customize-job-variables/ Can you share: - What worker type is ds-worker (ECS or Docker)? - A snippet of your Dockerfile.prefect_flows_run? I can adjust the YAML/Dockerfile precisely once I see those.