Short answer: there isn’t a built-in retry/backoff for image pulls in the Prefect 3.x Docker worker. If a pull fails (like your transient ACR EOF), the exception bubbles up and the run fails. Good news: there are a few practical ways to make this resilient. What you can do now - Avoid “latest” and pull less often - With

:latest

, the Docker worker defaults to always pulling, which increases the chance you hit a flaky registry. - Pin an immutable tag (or digest) and set

image_pull_policy: "IfNotPresent"

so the worker only pulls when the image isn’t cached. - Docs: Run flows in Docker and Customize job variables - Pre-pull and cache the image on the host (recommended) - Pre-pull the image where the Docker daemon runs, and let the worker use the cached image. - Example wrapper to start your worker with a retrying pre-pull:

#!/usr/bin/env bash
  set -euo pipefail

  IMAGE="<http://mytest.azurecr.io/prefect-test:latest|mytest.azurecr.io/prefect-test:latest>"

  # Ensure you're logged in to ACR first (docker login or via your secret manager)
  # docker login <http://mytest.azurecr.io|mytest.azurecr.io> -u "$ACR_USER" -p "$ACR_PASS"

  for i in {1..5}; do
    if docker pull "$IMAGE"; then
      break
    fi
    echo "Pull failed (attempt $i), retrying in $((i*5))s..."
    sleep $((i*5))
  done

  # Start the worker
  exec prefect worker start --pool my-docker-pool

- Then use

image_pull_policy: "IfNotPresent"

in your deployment/work pool template so Prefect won’t try to re-pull each run. - Automatically resubmit failed/crashed runs (infrastructure-level retry via Automations) - If a run fails due to a registry blip, have an Automation resubmit the same deployment after a short delay:

from datetime import timedelta
  from prefect.automations import Automation
  from prefect.events.actions import RunDeployment
  from prefect.events.schemas.automations import EventTrigger, Posture

  Automation(
      name="Resubmit on infra errors",
      trigger=EventTrigger(
          expect={"prefect.flow-run.Failed", "prefect.flow-run.Crashed"},
          for_each={"prefect.resource.id"},
          posture=Posture.Reactive,
      ),
      actions=[RunDeployment(source="inferred", schedule_after=timedelta(seconds=30))],
  ).create()

- Docs: Automations and Detect zombie flows - Ensure registry auth is solid - Configure

DockerRegistryCredentials

on the Docker work pool base job template so the worker always logs in before pulling. - Integration hub: Prefect Docker integration - API reference (job variables including

image_pull_policy

): Docker worker API - Optional infra tweaks - Run a pull-through cache/registry mirror to smooth over transient cloud registry issues. - Script a systemd/cron job to periodically pull the image with retries on the host. Quick examples for setting image_pull_policy - In a deployment:

flow.deploy(
      name="my-deployment",
      work_pool_name="my-docker-pool",
      job_variables={
          "image": "<http://mytest.azurecr.io/prefect-test:1.2.3|mytest.azurecr.io/prefect-test:1.2.3>",
          "image_pull_policy": "IfNotPresent",
      },
  )

- In prefect.yaml:

deployments:
    - name: my-deployment
      work_pool:
        name: my-docker-pool
        job_variables:
          image: "<http://mytest.azurecr.io/prefect-test:1.2.3|mytest.azurecr.io/prefect-test:1.2.3>"
          image_pull_policy: "IfNotPresent"

A couple questions to tailor this: - Are you okay switching off

:latest

to a pinned tag (or digest)? That plus

IfNotPresent

is the most effective change. - Where is your Docker daemon running (VM, k8s node, container)? I can suggest a pre-pull approach that fits your setup.

- Do you already have ACR credentials configured in a

DockerRegistryCredentials

block on your Docker work pool?

Great question — if you stick with

:latest

+

image_pull_policy: "Always"

, the cleanest way to avoid infinite resubmit loops is to track and bound “infra retry attempts” via a parameter that your Automation increments and your flow reads. Pattern to avoid loops 1) Add a hidden counter param to your flow and bail out when the limit is reached - This ensures a resubmitted run will exit cleanly (Completed), so it won’t trigger the automation again.

from prefect import flow, get_run_logger

@flow
def my_flow(_infra_retry: int = 0, MAX_INFRA_RETRIES: int = 3):
    logger = get_run_logger()

    # Stop the loop after N resubmissions caused by infra issues
    if _infra_retry > MAX_INFRA_RETRIES:
        logger.warning(
            f"Infra retry limit reached (_infra_retry={_infra_retry}); not retrying further."
        )
        return

    # Your normal flow logic
    ...

2) Automation: resubmit on Failed/Crashed, increment the counter, and add a cooldown - Use templating to carry the counter forward (+1 each time). - A small delay gives the registry time to recover and reduces thrash.

from datetime import timedelta
from prefect.automations import Automation
from prefect.events.actions import RunDeployment
from prefect.events.schemas.automations import EventTrigger, Posture

Automation(
    name="Resubmit on infra errors (bounded)",
    trigger=EventTrigger(
        expect={"prefect.flow-run.Failed", "prefect.flow-run.Crashed"},
        for_each={"prefect.resource.id"},
        posture=Posture.Reactive,
    ),
    actions=[
        RunDeployment(
            source="inferred",
            parameters={
                "_infra_retry": {
                    "__prefect_kind": "jinja",
                    "template": "{{ (flow_run.parameters.get('_infra_retry') or 0) + 1 }}"
                }
            },
            schedule_after=timedelta(seconds=60),  # cooldown/backoff
        )
    ],
).create()

Notes and tips - Prefer triggering on Crashed and/or specific infra failure cases if you want to avoid resubmitting after “real” code failures. When in doubt, use both Failed and Crashed, but the bounded attempt counter will still protect you. - You can also send a notification when

_infra_retry

exceeds the limit (inside the flow) so you know when the automation has “given up.” - With

:latest

, the Docker worker will default to always pulling. Keeping a cooldown (

schedule_after

) is a good idea to avoid hammering the registry. Docs - Automations concepts: Automations - Creating automations: Create automations - Passing/templating parameters in automations: Access parameters in templates If you want, share your current deployment/flow signature and I can tailor the parameter and automation snippets to your setup.