<@ULVA73B9P> Flow run crashed when github down, ...
# ask-marvin
p
@Marvin Flow run crashed when github down, how to handle it?
m
thought for 218.9 seconds
Short answer: in Prefect 3.x, when you use
flow.from_source("<https://github.com/...>").deploy(...)
, the worker pulls your code at run start. If GitHub is down, the git pull step retries a few times, then the flow run fails before your flow code ever starts. Flow-level retries won’t help because nothing in your flow has executed yet. Good options to handle this: - Rely on built‑in git retries (already enabled) - Prefect’s git pull step automatically retries 3 times with exponential backoff. If the outage is brief, the run will recover. - This is on by default when you use
flow.from_source(...)
or the
git_clone
pull step. - Docs: DeploymentsStore and retrieve flow code - Automatically reschedule failed runs via Automations - Create an Automation that triggers on “Flow run failed” (optionally filter on state message like “Failed to clone”) and action “Run deployment” with a delay/backoff. - This gives you hands‑off recovery when GitHub has a longer outage. - Docs: Automations - Remove the runtime GitHub dependency: bake code into a Docker image - Build/push an image in CI and run your deployment on that image. No git needed at runtime. - Example (prefect-docker steps in prefect.yaml):
Copy code
build:
      - prefect_docker.deployments.steps.build_docker_image:
          id: build-image
          requires: prefect-docker>=0.7.1
          image_name: my_registry/my_image
          tag: latest
          dockerfile: auto

    push:
      - prefect_docker.deployments.steps.push_docker_image:
          image_name: "{{ build-image.image_name }}"
          tag: "{{ build-image.tag }}"

    pull: null  # No git pull during runs

    deployments:
      - name: my-deployment
        entrypoint: my_flow.py:my_flow
        work_pool:
          name: my-work-pool
          job_variables:
            image: "{{ build-image.image }}"
- This is the most reliable pattern for production. - Use alternative storage (S3/GCS/Azure) and pull from there - Push a snapshot of your code to blob storage in CI; pull it at runtime. Often more available or under your control. - Example (S3):
Copy code
push:
      - prefect_aws.deployments.steps.push_to_s3:
          id: push_code
          requires: prefect-aws>=0.5
          bucket: my-bucket
          folder: my-app/code

    pull:
      - prefect_aws.deployments.steps.pull_from_s3:
          requires: prefect-aws>=0.5
          bucket: "{{ push_code.bucket }}"
          folder: "{{ push_code.folder }}"
- Docs: Store and retrieve flow code - Add a fallback if git fails (git → S3 backup) - Use a custom pull sequence to attempt git first, then copy a prebuilt code snapshot from storage:
Copy code
pull:
      - prefect.deployments.steps.git_clone:
          id: git
          repository: <https://github.com/org/repo.git>
          branch: main
      - prefect.deployments.steps.run_shell_script:
          script: |
            if [ ! -f "my_flow.py" ]; then
              echo "Git clone missing/failed, pulling backup from S3..."
              aws s3 cp <s3://my-backup-bucket/my-app/code> . --recursive
            fi
- Cache code on the runtime if you control the environment - Run workers in a container with a persistent volume or a base image that already contains your repo. If a transient git error occurs, you still have a usable copy of the code on disk. - You can keep periodic syncs by using a scheduled CI job to refresh the image/volume, or rely on your own bootstrap scripts. Notes and references - Built-in git retries: the
git_clone
pull step wraps the repo pull in retry logic (3 attempts, exponential backoff) so short outages are retried automatically. - API references:
- Pull steps: prefect.deployments.steps.pull - Runner storage: prefect.runner.storage If you share how you’re running (worker type, Docker/Kubernetes?, and how you deploy), I can suggest the most direct fix and give you a ready-to-use snippet. My default recommendation for production is: build a Docker image with your flow code and pin your deployment to that image, plus an Automation to re-run on failure for extra resilience.
p
Is there config so it only clone or pull once? @Marvin
m
thought for 142.3 seconds
Yes. In Prefect 3.x you can make it clone/pull only once by setting the GitRepository pull_interval to None. What it does - pull_interval=None: Prefect will do a single pull when the runtime initializes the storage and won’t do any periodic or pre-run pulls after that. - Effect depends on your runtime: - Long-lived worker (e.g., Process worker on a VM): one pull for the life of the worker process. - Ephemeral jobs (e.g., Kubernetes or Docker job per run): one pull per job container (effectively once per run, since each job is a fresh environment). How to configure it - Use a GitRepository explicitly with flow.from_source:
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

storage = GitRepository(
    url="<https://github.com/org/repo.git>",
    branch="main",
    pull_interval=None  # pull only once; no periodic or pre-run pulls
)

my_flow = flow.from_source(
    source=storage,
    entrypoint="flows.py:my_flow",
)

my_flow.deploy(
    name="my-deployment",
    work_pool_name="my-pool",
)
Notes - If you pass a plain string to flow.from_source (e.g., "https://github.com/org/repo.git"), Prefect uses a default pull_interval=60. To change this behavior you must construct GitRepository yourself as shown above (or via create_storage_from_source with pull_interval=None). - You cannot set pull_interval in prefect.yaml pull steps; those are for deployment build-time steps, not runtime pull cadence. Related docs: - Store and retrieve flow code - Runner storage API
p
Why there git_clone step in my flow run even though i already set the pull_intervall to none? @Marvin
m
thought for 45.9 seconds
Great question. Seeing a git_clone step doesn’t necessarily mean pull_interval=None isn’t respected. What’s happening - pull_interval=None disables periodic pulls and the pre-run “adhoc” pull. It does not suppress the initial checkout. Prefect still has to fetch the code once so it can import your entrypoint. - On ephemeral infrastructure (Kubernetes/Docker job per run), each run starts in a fresh container, so you’ll see one git clone per run (that’s still “once,” but per job). - If your deployment was created from a prefect.yaml that includes a pull section with git_clone, that step will always run at runtime regardless of pull_interval on GitRepository. How to get the behavior you want - Long‑lived worker, single initial clone: - Ensure you’re using a long‑lived worker (e.g., Process/Local). - Create the deployment with a GitRepository and pull_interval=None. - Do not include a pull: git_clone step in prefect.yaml. - Example:
Copy code
from prefect import flow
    from prefect.runner.storage import GitRepository

    storage = GitRepository(
        url="<https://github.com/org/repo.git>",
        branch="main",
        pull_interval=None  # one-time pull when the worker starts
    )

    my_flow = flow.from_source(
        source=storage,
        entrypoint="flows.py:my_flow",
    )

    my_flow.deploy(
        name="my-deployment",
        work_pool_name="my-pool",
    )
- Zero git pulls at runtime: - Bake code into a Docker image and set pull: null in prefect.yaml, or mount a persistent volume with your code. This removes the runtime dependency on git entirely. Why you might still see git_clone - It’s the first/only checkout (expected with pull_interval=None). - You have a pull: section in prefect.yaml (e.g., prefect.deployments.steps.git_clone) which always executes; remove it if you’re using GitRepository storage. - Log wording: the initial checkout is internally labeled “git_clone,” so it can look like a recurring step even when it’s just the first pull. If you can share how your deployment is defined (your deployment code or prefect.yaml) and what worker type you’re using (long‑lived vs job-per-run), I can pinpoint exactly why you’re seeing the step and provide a precise fix. Docs for reference: - Deployments - Store and retrieve flow code