pradana adnyana
03/12/2026, 3:49 PMMarvin
03/12/2026, 3:49 PMMarvin
03/12/2026, 3:53 PMflow.from_source("<https://github.com/...>").deploy(...), the worker pulls your code at run start. If GitHub is down, the git pull step retries a few times, then the flow run fails before your flow code ever starts. Flow-level retries won’t help because nothing in your flow has executed yet.
Good options to handle this:
- Rely on built‑in git retries (already enabled)
- Prefect’s git pull step automatically retries 3 times with exponential backoff. If the outage is brief, the run will recover.
- This is on by default when you use flow.from_source(...) or the git_clone pull step.
- Docs: Deployments • Store and retrieve flow code
- Automatically reschedule failed runs via Automations
- Create an Automation that triggers on “Flow run failed” (optionally filter on state message like “Failed to clone”) and action “Run deployment” with a delay/backoff.
- This gives you hands‑off recovery when GitHub has a longer outage.
- Docs: Automations
- Remove the runtime GitHub dependency: bake code into a Docker image
- Build/push an image in CI and run your deployment on that image. No git needed at runtime.
- Example (prefect-docker steps in prefect.yaml):
build:
- prefect_docker.deployments.steps.build_docker_image:
id: build-image
requires: prefect-docker>=0.7.1
image_name: my_registry/my_image
tag: latest
dockerfile: auto
push:
- prefect_docker.deployments.steps.push_docker_image:
image_name: "{{ build-image.image_name }}"
tag: "{{ build-image.tag }}"
pull: null # No git pull during runs
deployments:
- name: my-deployment
entrypoint: my_flow.py:my_flow
work_pool:
name: my-work-pool
job_variables:
image: "{{ build-image.image }}"
- This is the most reliable pattern for production.
- Use alternative storage (S3/GCS/Azure) and pull from there
- Push a snapshot of your code to blob storage in CI; pull it at runtime. Often more available or under your control.
- Example (S3):
push:
- prefect_aws.deployments.steps.push_to_s3:
id: push_code
requires: prefect-aws>=0.5
bucket: my-bucket
folder: my-app/code
pull:
- prefect_aws.deployments.steps.pull_from_s3:
requires: prefect-aws>=0.5
bucket: "{{ push_code.bucket }}"
folder: "{{ push_code.folder }}"
- Docs: Store and retrieve flow code
- Add a fallback if git fails (git → S3 backup)
- Use a custom pull sequence to attempt git first, then copy a prebuilt code snapshot from storage:
pull:
- prefect.deployments.steps.git_clone:
id: git
repository: <https://github.com/org/repo.git>
branch: main
- prefect.deployments.steps.run_shell_script:
script: |
if [ ! -f "my_flow.py" ]; then
echo "Git clone missing/failed, pulling backup from S3..."
aws s3 cp <s3://my-backup-bucket/my-app/code> . --recursive
fi
- Cache code on the runtime if you control the environment
- Run workers in a container with a persistent volume or a base image that already contains your repo. If a transient git error occurs, you still have a usable copy of the code on disk.
- You can keep periodic syncs by using a scheduled CI job to refresh the image/volume, or rely on your own bootstrap scripts.
Notes and references
- Built-in git retries: the git_clone pull step wraps the repo pull in retry logic (3 attempts, exponential backoff) so short outages are retried automatically.
- API references:Marvin
03/12/2026, 3:53 PMpradana adnyana
03/12/2026, 3:55 PMMarvin
03/12/2026, 3:55 PMMarvin
03/12/2026, 3:57 PMfrom prefect import flow
from prefect.runner.storage import GitRepository
storage = GitRepository(
url="<https://github.com/org/repo.git>",
branch="main",
pull_interval=None # pull only once; no periodic or pre-run pulls
)
my_flow = flow.from_source(
source=storage,
entrypoint="flows.py:my_flow",
)
my_flow.deploy(
name="my-deployment",
work_pool_name="my-pool",
)
Notes
- If you pass a plain string to flow.from_source (e.g., "https://github.com/org/repo.git"), Prefect uses a default pull_interval=60. To change this behavior you must construct GitRepository yourself as shown above (or via create_storage_from_source with pull_interval=None).
- You cannot set pull_interval in prefect.yaml pull steps; those are for deployment build-time steps, not runtime pull cadence.
Related docs:
- Store and retrieve flow code
- Runner storage APIpradana adnyana
03/12/2026, 3:58 PMMarvin
03/12/2026, 3:58 PMMarvin
03/12/2026, 3:59 PMfrom prefect import flow
from prefect.runner.storage import GitRepository
storage = GitRepository(
url="<https://github.com/org/repo.git>",
branch="main",
pull_interval=None # one-time pull when the worker starts
)
my_flow = flow.from_source(
source=storage,
entrypoint="flows.py:my_flow",
)
my_flow.deploy(
name="my-deployment",
work_pool_name="my-pool",
)
- Zero git pulls at runtime:
- Bake code into a Docker image and set pull: null in prefect.yaml, or mount a persistent volume with your code. This removes the runtime dependency on git entirely.
Why you might still see git_clone
- It’s the first/only checkout (expected with pull_interval=None).
- You have a pull: section in prefect.yaml (e.g., prefect.deployments.steps.git_clone) which always executes; remove it if you’re using GitRepository storage.
- Log wording: the initial checkout is internally labeled “git_clone,” so it can look like a recurring step even when it’s just the first pull.
If you can share how your deployment is defined (your deployment code or prefect.yaml) and what worker type you’re using (long‑lived vs job-per-run), I can pinpoint exactly why you’re seeing the step and provide a precise fix.
Docs for reference:
- Deployments
- Store and retrieve flow code