Short answer: in Prefect 2.x there isn’t a built-in retry mechanism for deployment “pull” steps (e.g.,

git_clone

or a

git pull

run via steps). When that step fails, the flow run typically crashes before your flow code starts, so flow/task retries don’t apply. Good options you can use today: - Prefect Cloud Automation to resubmit on crash/failure - Create an Automation that triggers when a flow run enters Crashed or Failed for your deployment, then performs a “resubmit/create flow run” action (optionally with a delay/backoff). - This requeues the run so the whole process (including the git step) is attempted again. - Docs: Automations (Prefect 2) - Add retry logic around the git command in your pull steps - Replace the stock `git_clone`/`git pull` with a

run_shell_script

pull step that retries with backoff. Example

prefect.yaml

snippet:

pull:
    - prefect.deployments.steps.utility.set_working_directory:
        directory: "/opt/prefect/flows"  # adjust as needed
    - prefect.deployments.steps.utility.run_shell_script:
        # Exponential backoff retry for a transient git failure
        script: |
          set -euo pipefail
          REPO_URL="<https://github.com/your-org/your-repo.git>"
          TARGET_DIR="repo"
          MAX_RETRIES=5

          clone_or_pull() {
            if [ -d "$TARGET_DIR/.git" ]; then
              echo "Repo exists; pulling latest..."
              git -C "$TARGET_DIR" fetch --all --prune
              git -C "$TARGET_DIR" reset --hard origin/main
              git -C "$TARGET_DIR" submodule update --init --recursive
            else
              echo "Cloning repo..."
              git clone --recurse-submodules "$REPO_URL" "$TARGET_DIR"
            fi
          }

          n=0
          until clone_or_pull; do
            n=$((n+1))
            if [ $n -ge $MAX_RETRIES ]; then
              echo "git operation failed after $MAX_RETRIES attempts"
              exit 1
            fi
            sleep_time=$(( 5 * n ))
            echo "git failed (attempt $n/$MAX_RETRIES); retrying in ${sleep_time}s..."
            sleep "$sleep_time"
          done
        stream_output: true

- Step reference: prefect.yaml and deployment steps - Avoid runtime git entirely - Bake your flow code into a container image used by the deployment’s infrastructure, so there’s no

git

call at run-time. This removes a common transient failure point. - Use infrastructure-level restarts (if applicable) - If you’re running in Kubernetes, set a Job

backoffLimit

on your work pool’s job template so the container restarts on non-zero exit (the git failure) and retries the whole run. - Similarly, if you’re using Docker or ECS, enable restart/on-failure semantics where available in your deployment environment. This re-attempts the whole process including the pull step. A couple of clarifying questions so I can tailor a precise recommendation: - Are you using Prefect Cloud or Prefect Server? - Which worker/work pool type are you using (Process, Docker, Kubernetes, ECS)? - Could you share the

pull:

section of your

prefect.yaml

(sanitized)? That will help me suggest the cleanest retry pattern for your setup.

Ottimo — l’approccio con uno shell script nel

pull:

è perfetto per aggiungere retry e backoff al passo Git. Qui sotto trovi un esempio plug-and-play che sostituisce il tuo

git_clone

. Fa: - clone iniziale con token e branch - pull successivi con

fetch + reset --hard

sul branch richiesto - retry con backoff lineare (puoi portarlo a esponenziale se vuoi) - imposta la working directory alla cartella del repo, così i tuoi

entrypoint

restano invariati Nota: assicurati che l’immagine/ambiente del worker abbia

git

installato. Sostituisci la sezione

pull:

così:

pull:
  # 1) esegue git clone/pull con retry
  - prefect.deployments.steps.utility.run_shell_script:
      script: |
        set -euo pipefail

        REPO_URL="<https://github.com/prometeia-wam/da-data-pipeline-customer.git>"
        BRANCH="${GIT_BRANCH:-main}"
        TARGET_DIR="repo"        # cartella di checkout
        MAX_RETRIES=5

        # Costruisce URL con token se presente
        auth_url="$REPO_URL"
        if [ -n "${GIT_TOKEN:-}" ]; then
          auth_url="$(echo "$REPO_URL" | sed -E "s#https://#https://${GIT_TOKEN}@#")"
        fi

        clone_or_pull() {
          if [ -d "$TARGET_DIR/.git" ]; then
            echo "Repo esistente: fetch/reset su $BRANCH"
            git -C "$TARGET_DIR" remote set-url origin "$auth_url" >/dev/null 2>&1 || true
            git -C "$TARGET_DIR" fetch --all --prune
            git -C "$TARGET_DIR" checkout "$BRANCH"
            git -C "$TARGET_DIR" reset --hard "origin/$BRANCH"
            git -C "$TARGET_DIR" submodule update --init --recursive || true
          else
            echo "Clono $BRANCH in $TARGET_DIR"
            rm -rf "$TARGET_DIR"
            git clone --branch "$BRANCH" --single-branch --recurse-submodules "$auth_url" "$TARGET_DIR"
          fi
        }

        n=0
        until clone_or_pull; do
          n=$((n+1))
          if [ $n -ge $MAX_RETRIES ]; then
            echo "git fallito dopo $MAX_RETRIES tentativi"
            exit 1
          fi
          sleep_time=$(( 5 * n ))  # backoff lineare (5s,10s,15s,...)
          echo "git fallito (tentativo $n/$MAX_RETRIES); riprovo tra ${sleep_time}s..."
          sleep "$sleep_time"
        done
      env:
        # prende i valori dai tuoi templating Prefect
        GIT_TOKEN: "{{ prefect.blocks.secret.github-tech-user-token }}"
        GIT_BRANCH: "{{ prefect.variables.git_branch }}"
      stream_output: true

  # 2) imposta la working directory al repo clonato
  - prefect.deployments.steps.pull.set_working_directory:
      directory: "repo"

Cose da tenere a mente: - Se vuoi un backoff esponenziale, sostituisci

sleep_time=$(( 5 * n ))

con

sleep_time=$(( 5 * 2**(n-1) ))

e magari aggiungi un po’ di jitter. - Questo script evita di stampare il token; non fa echo delle variabili. - Se usate submodules, il comando

submodule update --init --recursive

è già previsto. - Se preferisci non cambiare la working dir, puoi clonare direttamente nella dir corrente (ma attenzione: il clone in

.

richiede una dir vuota). Riferimenti utili: - prefect.yaml e deployment steps (Prefect 2) - Automations (Prefect 2) per un eventuale “piano B” che rimandi automaticamente i run in stato Failed/Crashed Se mi confermi che state usando Cloud o Server e il tipo di worker/work pool (Process, Docker, Kubernetes, …), posso suggerirti anche eventuali retry a livello di infrastruttura come ulteriore rete di sicurezza.