Thanks for sharing your prefect.ecs.yaml. The behavior you’re seeing is expected with ECS on EC2 when using a static tag like dev: your EC2 host caches the image locally and ECS will often reuse the cached image instead of pulling the updated one from ECR. That’s why changes only show up after you manually push and “refresh” the image. Two reliable ways to fix this: 1) Recommended: Use immutable image tags (or digests) for each deploy - Keep publishing dev as a moving tag if you like, but set the deployment to run the uniquely tagged image for that build. - Generate a unique tag (e.g., git SHA) in a step, build/push with that tag, and point job_variables.image at that immutable tag. Example:

name: src
prefect-version: 3.4.11

build:
- prefect.deployments.steps.utility.run_shell_script:
    id: git_sha
    script: git rev-parse --short HEAD
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1
    image_name: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect>
    tag: '{{ git_sha.stdout | trim }}'      # immutable tag for this build
    additional_tags:
    - dev                                   # optional moving tag for humans
    dockerfile: Dockerfile.prefect_flows_run
    ignore_cache: true                      # rebuild layers on your build host

push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.1
    image_name: '{{ build_image.image_name }}'
    tag: '{{ git_sha.stdout | trim }}'
    additional_tags:
    - dev

pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/src

deployments:
- name: dev
  version: '{{ git_sha.stdout | trim }}'
  tags: ['dev']
  entrypoint: extraction/ebay/flow.py:run
  parameters: {"seller_id":"d480fd76-f59e-4ea0-95cb-4e22db0164e0","selling_partner_id":"ed442813-afe7-4026-bc14-190ca41bfb23","data_cycle":"origination"}
  work_pool:
    name: my-ecs-pool
    work_queue_name: default
    job_variables:
      # Use the immutable tag so ECS must pull it
      image: '{{ build_image.image }}'
      env:
        APP_ENV: "dev"
        SOURCE: "ebay"

- This ensures each flow run uses a new image reference, so the ECS host must pull it. If you prefer, you can pin to an image digest (repo@sha256:...) for even stronger immutability. 2) Alternative: Tell the ECS agent to always pull - If you must keep a static tag like dev, configure the ECS agent on your EC2 host to always pull the image: - Add

ECS_IMAGE_PULL_BEHAVIOR=always

to

/etc/ecs/ecs.config

and restart the ECS agent. - On Amazon Linux 2, for example:

sudo bash -lc 'echo ECS_IMAGE_PULL_BEHAVIOR=always >> /etc/ecs/ecs.config'
    sudo systemctl restart ecs

- AWS docs: ECS agent image pull behavior A few extra notes: - Your build/push steps will run when you call

prefect deploy --prefect-file ...

. The

requires: prefect-docker>=0.3.1

line will auto-install the step’s dependency if needed. -

ignore_cache: true

on the build step skips local Docker layer cache on the build machine; it does not affect ECS pulling. The immutable tag (or ECS_IMAGE_PULL_BEHAVIOR=always) fixes the ECS caching issue. - ECS does not have a task definition setting analogous to Kubernetes

imagePullPolicy: Always

. That’s why the two approaches above are the standard solutions. Relevant docs: - Prefect YAML / steps: Prefect YAML and steps - Versioning best practices: Deployment versioning - ECS work pools: Prefect ECS guide

Would you like me to help you update your YAML to use immutable tags, or do you prefer flipping the ECS agent setting on your EC2 host?

Got it — if you need to keep using a mutable tag like

dev

, the most reliable fix on ECS EC2 is to tell the ECS agent to always pull the image before starting a task. Do this on the EC2 instance(s) that run your ECS tasks: 1) Set the ECS agent image pull behavior

sudo bash -lc 'echo ECS_IMAGE_PULL_BEHAVIOR=always >> /etc/ecs/ecs.config'
sudo systemctl restart ecs

2) Verify it took effect - Check the value is present:

grep ECS_IMAGE_PULL_BEHAVIOR /etc/ecs/ecs.config

- After you push a new

dev

image to ECR, start a flow run; in the ECS task logs you should see it pulling the image each time. Notes: - This setting is per EC2 host. If you ever add more instances to your ECS cluster, add it to each one (via user data or AMI). - Expect slightly slower task starts since the agent will always re-pull. - Your current prefect.ecs.yaml is fine for mutable tags — the key is ensuring the ECS agent is configured to always pull. AWS docs for the setting: ECS agent image pull behavior If you want, I can also tweak your YAML to keep publishing

dev

plus a unique tag (for history/rollbacks) while your tasks still run

dev

.

Short answer: You can use the tag

latest

, but I don’t recommend it for dev/stage/prod because any push to

latest

will immediately affect every deployment that references it. On ECS EC2, that can lead to surprise upgrades across environments and caching weirdness unless you force pulls. If you still want a mutable tag approach, here are safer patterns: - Per-environment mutable tags (recommended for mutable tags) - Use

dev

,

stage

,

prod

as separate tags. - Set ECS to always pull so your EC2 host doesn’t reuse a stale cached image. - Only move each env’s tag when you intend to promote changes. - Hybrid (immutable builds + mutable pointers) - Build once with an immutable tag (e.g., git SHA), and push additional tags for convenience: - Push

dev

during dev deploys, later retag and push

stage

, then

prod

when you promote. - Dev deployment can reference `dev`; prod can reference the immutable digest for extra safety. Example YAML sketch using per-env tags with a single build step:

build:
- prefect.deployments.steps.utility.run_shell_script:
    id: git_sha
    script: git rev-parse --short HEAD
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1
    image_name: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect>
    tag: '{{ git_sha.stdout | trim }}'       # immutable build tag
    additional_tags:                         # mutable pointers per env
    - dev
    - stage
    - prod
    dockerfile: Dockerfile.prefect_flows_run
    ignore_cache: true

push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.1
    image_name: '{{ build_image.image_name }}'
    tag: '{{ git_sha.stdout | trim }}'
    additional_tags:
    - dev
    - stage
    - prod

deployments:
- name: dev
  entrypoint: extraction/ebay/flow.py:run
  work_pool:
    name: my-ecs-pool
    work_queue_name: default
    job_variables:
      image: '<http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:dev|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:dev>'

- name: stage
  entrypoint: extraction/ebay/flow.py:run
  work_pool:
    name: my-ecs-pool
    work_queue_name: default
    job_variables:
      image: '<http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:stage|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:stage>'

- name: prod
  entrypoint: extraction/ebay/flow.py:run
  work_pool:
    name: my-ecs-pool
    work_queue_name: default
    job_variables:
      # For maximum safety in prod, consider pinning to the immutable tag or digest
      image: '<http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:prod|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:prod>'

If you use a single

latest

tag: - Any environment pointing to

:latest

will start running whatever you most recently pushed. - On ECS EC2 you must configure the agent to always pull to avoid stale cache:

sudo bash -lc 'echo ECS_IMAGE_PULL_BEHAVIOR=always >> /etc/ecs/ecs.config'
  sudo systemctl restart ecs

AWS docs: ECS agent image pull behavior Related Prefect docs: - Versioning best practices: Deployment versioning - ECS work pools: Prefect ECS guide If you confirm which pattern you prefer (single

latest

vs per-env tags vs hybrid), I can tailor your prefect.ecs.yaml accordingly.

Great question. There isn’t a CLI flag to pass arbitrary variables into build steps, but you can parameterize your prefect.ecs.yaml with environment variables (or Prefect Variables) and then set them when you run prefect deploy. Here’s how to wire it up so you can change the Docker tag from the command: 1) Update your prefect.ecs.yaml to read the tag from an env var

name: src
prefect-version: 3.4.11

build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1
    image_name: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect>
    tag: '{{ $DOCKER_TAG | default("dev") }}'
    dockerfile: Dockerfile.prefect_flows_run

push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.1
    image_name: '{{ build_image.image_name }}'
    tag: '{{ $DOCKER_TAG | default("dev") }}'

pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/src

deployments:
- name: dev
  version: '{{ build_image.tag }}'
  tags: ['{{ build_image.tag }}']
  entrypoint: extraction/ebay/flow.py:run
  parameters: {"seller_id":"d480fd76-f59e-4ea0-95cb-4e22db0164e0","selling_partner_id":"ed442813-afe7-4026-bc14-190ca41bfb23","data_cycle":"origination"}
  work_pool:
    name: my-ecs-pool
    work_queue_name: default
    job_variables:
      image: '{{ build_image.image }}'
      env:
        APP_ENV: "dev"
        SOURCE: "ebay"

2) Pass the tag via an environment variable when you run deploy - Use whatever tag you need (dev, stage, prod, latest, etc.):

DOCKER_TAG=latest prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml
# or
DOCKER_TAG=stage prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml
# or
DOCKER_TAG=prod prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml

Notes: - If you only need to change the image used by runs (not the build/push), you can also override the job variable at deploy time:

prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml \
  -jv image=<http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:latest|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:latest>

This does not change the build/push tag—just the image your ECS tasks will use. - Because you’re on ECS EC2 and want mutable tags, make sure your ECS agent is set to always pull or you’ll hit caching:

sudo bash -lc 'echo ECS_IMAGE_PULL_BEHAVIOR=always >> /etc/ecs/ecs.config'
sudo systemctl restart ecs

AWS docs: ECS agent image pull behavior Docs for reference: - Prefect YAML and templating: Prefect YAML - Customize job variables: Customize job variables - Prefect Variables (optional alternative to env vars): Prefect Variables

Thanks for sharing the traceback — that 400 is happening because Docker is being asked to tag the image with the literal string

{{ git_sha.stdout | trim }}

. In other words, the placeholder wasn’t templated, so Docker sees invalid characters in the tag. Why this happens: - In Prefect 3.x, step outputs are referenced with simple placeholders like

{{ step_id.field }}

— it’s not full Jinja and filters like

| trim

are not supported. That’s why

| trim

wasn’t replaced. - You also need to ensure the step has an

id

so you can reference it, and that the reference is quoted in YAML (to avoid YAML interpreting the braces). How to fix it: - Remove the filter and reference the step output directly:

{{ git_sha.stdout }}

- Make sure your shell step prints a value without a trailing newline, since Docker tags can’t include newline characters. Use

printf

or

tr -d '\n'

. Example prefect.yaml using an immutable ECR tag based on the short git SHA:

name: credilinq-prefect
prefect-version: ">=3.0.0"

deployments:
- name: my-flow-deploy
  entrypoint: flows/my_flow.py:flow
  work_pool:
    name: ecs-ec2-pool
    job_variables:
      # Use the immutable SHA tag for the runtime image too
      image: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{> git_sha.stdout }}

  build:
    # 1) Get commit sha without trailing newline
    - prefect.deployments.steps.utility.run_shell_script:
        id: git_sha
        script: 'printf "%s" "$(git rev-parse --short HEAD)"'

    # 2) Build the Docker image, tag with the sha and optionally 'latest'
    - prefect_docker.deployments.steps.build_docker_image:
        id: build_image
        image_name: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect>
        tag: "{{ git_sha.stdout }}"
        additional_tags:
          - latest

  push:
    # 3) Log in to ECR (if your docker daemon isn’t already logged in)
    - prefect.deployments.steps.utility.run_shell_script:
        id: ecr_login
        script: |
          aws ecr get-login-password --region ap-southeast-1 \
            | docker login --username AWS --password-stdin <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com>

    # 4) Push both tags
    - prefect_docker.deployments.steps.push_docker_image:
        image_name: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect>
        tag: "{{ git_sha.stdout }}"
        additional_tags:
          - latest

Key points: - Use

{{ git_sha.stdout }}

(no

| trim

). - Keep the placeholder in quotes in YAML. - Ensure the

run_shell_script

outputs a clean value with no newline (via

printf

or

tr -d '\n'

). - Make sure your Docker daemon is reachable where you run

prefect deploy

(the error shows it’s using the local Docker daemon via http+docker://localhost, which is correct). If you can share your current

prefect.yaml

build/push sections, I can spot-check them. References: - Prefect 3 deployment steps and templating overview: prefect.yaml guide - Step utilities (run_shell_script): run_shell_script API - Docker steps (build/push): prefect-docker steps

Got it — if the deploy now succeeds but your flow code isn’t changing at runtime, the most common cause with ECS is that the worker is running a fixed ECS Task Definition that pins an image tag, so your deployment’s new image/tag never gets used. What I see in your task definition: - The container image is hard-coded to …/credilinq-prefect:prefect-flow-run. - The task def ARN is arnawsecs…task-definition/prefect-flow-run:1. - requiresCompatibilities is set to ["FARGATE"]. How this interacts with Prefect ECS workers (Prefect 3.x): - If your ECS work pool is configured with a task_definition_arn, the worker uses that task definition as-is. Prefect will not override the container image from your deployment’s job_variables. That means you’ll keep running whatever image/tag is in the task definition (in your case, :prefect-flow-run), even if your deployment builds and pushes a new immutable tag every time. - To let Prefect use the image you set in prefect.yaml (e.g. the git SHA tag), do NOT pin a task_definition_arn in the work pool. Instead, let Prefect render/register a task definition for each run using the job variables (including image). - Also, you said you want to use ECS EC2, but the posted task definition has requiresCompatibilities ["FARGATE"]. If your work pool is set to EC2 launch type, that’s a mismatch. Use a task definition suitable for EC2 or let Prefect generate one with launch_type=EC2. What to change Option A (recommended): Let Prefect manage the task definition so your image updates take effect 1) Remove task_definition_arn from the ECS work pool. 2) Ensure your deployment sets the image on the work pool job variables to the immutable tag:

deployments:
- name: my-flow-deploy
  work_pool:
    name: ecs-ec2-pool
    job_variables:
      image: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{> git_sha.stdout }}
      launch_type: EC2
      # set cluster, subnets, security groups, etc., as needed

3) Keep your build/push steps producing that tag as you already do. Option B: Keep a pinned task definition, but you must update it every deploy - If you choose to keep task_definition_arn on the pool, Prefect will not change its container image. You’d need your deploy process to register a new task definition revision with the new image tag and then update the work pool (or the task_definition_arn) to point at the new revision. This is more operational overhead and easy to forget. Other checks to ensure updates are picked up - Build cache: if you suspect Docker cached layers and baked old code into your image, set

ignore_cache: true

in your build step:

- prefect_docker.deployments.steps.build_docker_image:
    image_name: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect>
    tag: "{{ git_sha.stdout }}"
    ignore_cache: true

- Entry point path: make sure the deployment’s

entrypoint

matches the file path inside the container image. - Worker logs: check the ECS worker logs for the image it’s launching. Prefect will log the job variables. If you still see

:prefect-flow-run

, that confirms a pinned task def or wrong job variables. If you can share: - The ECS work pool configuration (especially whether

task_definition_arn

is set, launch type, and any custom task definition template). - The

deployments[..].work_pool.job_variables

section of your prefect.yaml. I’ll pinpoint the exact fix. Docs for reference: - ECS worker guide: Prefect ECS worker (prefect-aws) - prefect.yaml and templating: prefect.yaml guide - Customizing job variables: Customize job variables

- Docker steps: prefect-docker steps