<@ULVA73B9P> I am trying to deploy my first flow t...
# ask-marvin
m
@Marvin I am trying to deploy my first flow to prefect server in ecs , but it get is not moving onwards the last step prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml Running deployment build steps... > Running build_docker_image step... Step 1/7 : FROM prefecthq/prefect:3-python3.12 ---> af96b39899cd Step 2/7 : WORKDIR /opt/prefect/src ---> Using cache ---> 00063cb97921 Step 3/7 : RUN mkdir -p /opt/prefect/src/extraction ---> Using cache ---> 58ccd50efd79 Step 4/7 : COPY requirements.txt . ---> Using cache ---> d52bf2d905a9 Step 5/7 : RUN pip install --no-cache-dir -r requirements.txt ---> Using cache ---> 2ffe9e2dd425 Step 6/7 : COPY . . ---> Using cache ---> 7a1d57b1bb33 Step 7/7 : LABEL io.prefect.version=3.4.13 ---> Using cache ---> c66692a7a628 Successfully built c66692a7a628 Running deployment push steps... > Running push_docker_image step...
m
thought for 303.0 seconds
Thanks for the details — it looks like your deployment is hanging at the “push_docker_image” step. That step invokes Docker to push your built image to a registry (typically ECR). If Docker cannot authenticate to the registry, cannot reach it, or is pushing a large image without streaming progress, it can appear to “hang” at that line. What the step expects - Step function:
prefect_docker.deployments.steps.push_docker_image(image_name, tag=None, credentials=None, additional_tags=None, ignore_cache=False)
- If
credentials
is omitted, it relies on your local Docker login/credential helper. - If provided,
credentials
should be a DockerRegistryCredentials block (or a dict with equivalent fields). Common causes of hanging at push - Not logged in to the target registry (ECR) from the machine running
prefect deploy
- Image name does not include the registry prefix, so Docker tries Docker Hub (rate-limited or blocked) - Missing ECR permissions (GetAuthorizationToken, PutImage, InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload, BatchCheckLayerAvailability, CreateRepository if you rely on auto-create) - ECR repository doesn’t exist yet - Network egress blocked (no NAT Gateway and no VPC endpoints to ECR/S3) - Large image upload with no progress output (the step does not stream Docker’s layer progress) Quick checks 1) Confirm the image name you’re building matches your ECR registry: - It must be like
<account-id>.<http://dkr.ecr.ap-southeast-1.amazonaws.com/<repo>:<tag|dkr.ecr.ap-southeast-1.amazonaws.com/<repo>:<tag>>
. - If your
image_name
is just
my-repo/image:tag
, Docker will try Docker Hub. 2) Try a manual push from the same EC2 instance:
Copy code
aws ecr get-login-password --region ap-southeast-1 | docker login --username AWS --password-stdin <account-id>.<http://dkr.ecr.ap-southeast-1.amazonaws.com|dkr.ecr.ap-southeast-1.amazonaws.com>

# if the repo doesn’t exist, create it
aws ecr create-repository --repository-name <repo> --region ap-southeast-1 || true

# push the image you built during deploy
docker push <account-id>.<http://dkr.ecr.ap-southeast-1.amazonaws.com/<repo>:<tag|dkr.ecr.ap-southeast-1.amazonaws.com/<repo>:<tag>>
- If this fails, fix the underlying Docker/ECR issue first. If it succeeds, Prefect’s push should work too. 3) Ensure the EC2 instance has AWS credentials with ECR permissions: - Prefer an instance profile role with ECR permissions (GetAuthorizationToken, PutImage, UploadLayerPart, CompleteLayerUpload, etc.). - Or ensure
~/.aws/credentials
is configured for the user running the deploy. 4) Networking (since you’re in ap-southeast-1): - If the instance is in a private subnet without NAT, add VPC endpoints: - Interface endpoints: com.amazonaws.ap-southeast-1.ecr.api, com.amazonaws.ap-southeast-1.ecr.dkr - S3 Gateway endpoint (ECR uses S3 for layer storage) Ways to configure auth for the Prefect push Option A — Use the ECR Docker credential helper (recommended) - Install and configure the helper so Docker auto-authenticates:
Copy code
sudo yum install -y amazon-ecr-credential-helper || sudo apt-get install -y amazon-ecr-credential-helper

mkdir -p ~/.docker
cat > ~/.docker/config.json <<EOF
{
  "credHelpers": {
    "<account-id>.<http://dkr.ecr.ap-southeast-1.amazonaws.com|dkr.ecr.ap-southeast-1.amazonaws.com>": "ecr-login"
  }
}
EOF
- Ensure the EC2 has IAM permissions (or valid AWS credentials). Then you can omit
credentials
in
push_docker_image
. Option B — Add a login step before push in your prefect.ecs.yaml
Copy code
build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      image_name: "<account-id>.<http://dkr.ecr.ap-southeast-1.amazonaws.com/<repo|dkr.ecr.ap-southeast-1.amazonaws.com/<repo>>"
      tag: "latest"
      dockerfile: "Dockerfile"

push:
  - prefect.deployments.steps.utility.run_shell_script:
      id: ecr_login
      script: |
        aws ecr get-login-password --region ap-southeast-1 | \
        docker login --username AWS --password-stdin <account-id>.<http://dkr.ecr.ap-southeast-1.amazonaws.com|dkr.ecr.ap-southeast-1.amazonaws.com>

  - prefect_docker.deployments.steps.push_docker_image:
      image_name: "{{ build_image.image_name }}"
      tag: "{{ build_image.tag }}"
Option C — Use a DockerRegistryCredentials block (okay for testing, not ideal for ECR since passwords are short-lived) - Create a Docker Registry Credentials block with: - registry_url:
https://<account-id>.<http://dkr.ecr.ap-southeast-1.amazonaws.com|dkr.ecr.ap-southeast-1.amazonaws.com>
- username:
AWS
- password: output of
aws ecr get-login-password
(this expires, so you’ll need to refresh it) - Reference it in your YAML:
Copy code
- prefect_docker.deployments.steps.push_docker_image:
    image_name: "{{ build_image.image_name }}"
    tag: "{{ build_image.tag }}"
    credentials: "{{ prefect.blocks.docker-registry-credentials.<your-block-name> }}"
Get more visibility when it “hangs” - Run a manual
docker push --debug <image>
to see precise Docker-side behavior. - Run Prefect with debug logging:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml
A couple of notes - Because your EC2 is in ap-southeast-1, make sure all commands use that region. - If your repo doesn’t exist, create it first. Docker cannot auto-create ECR repos. If you can, please share the relevant bits of your
prefect.ecs.yaml
(the build and push steps, especially the
image_name
and any
credentials
you set). I can sanity-check the configuration. References - Prefect 3 deployment config overview: Prefect YAML guide - Storing flow code and images in deployments: Store flow code - AWS ECR authentication: ECR registry authentication - Amazon ECR credential helper: ecr-login helper