Hi <@ULVA73B9P>, how to make when run using proces...
# ask-marvin
p
Hi @Marvin, how to make when run using process pool with github, only clone when there any changes
m
thought for 310.8 seconds
Great question! Assuming you’re on Prefect 3.x with a Process work pool: - Prefect will clone your Git repo once to a local cache on the worker. - On subsequent runs, it does a shallow
git pull
and only updates if there are changes (it does not re-clone each run). - You control how often Prefect checks for updates with
pull_interval
on
GitRepository
(default is 60s). If the interval hasn’t elapsed, it reuses the cached code. Examples: - Check for changes every 5 minutes
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

flow.from_source(
    source=GitRepository(
        url="<https://github.com/org/repo.git>",
        pull_interval=300  # pull at most every 5 minutes
    ),
    entrypoint="flows.py:my_flow",
).deploy(name="gh-deployment", work_pool_name="my-process-pool")
- Clone once and never pull again
Copy code
from prefect.runner.storage import GitRepository

flow.from_source(
    source=GitRepository(
        url="<https://github.com/org/repo.git>",
        pull_interval=None  # clone once; no periodic pulls
    ),
    entrypoint="flows.py:my_flow",
).deploy(name="no-refresh", work_pool_name="my-process-pool")
- Pin to an exact commit (no updates unless the commit isn’t present locally)
Copy code
from prefect.runner.storage import GitRepository

flow.from_source(
    source=GitRepository(
        url="<https://github.com/org/repo.git>",
        commit_sha="abc123def456"
    ),
    entrypoint="flows.py:my_flow",
).deploy(name="pinned", work_pool_name="my-process-pool")
Notes: - Prefect uses shallow clones by default (
--depth 1
) and reuses the local repo between runs. - If there are no upstream changes, the
git pull
is a no-op. Docs and references: - Deploy via Python - Store flow code (Git) - Runner storage API (GitRepository) Would you like to pull only once at startup, or periodically (e.g., every N minutes)?
p
It will pull on run or we can set outside @Marvin?
m
thought for 263.1 seconds
Do you mean pulling the container image, or pulling your flow code from Git? Here’s how both work in Prefect 3.x: - Container image pulling - Docker worker: The worker pulls at run time based on
image_pull_policy
. - Defaults: If tag is
latest
→ `Always`; otherwise →
IfNotPresent
. - You can disable pulling by setting
image_pull_policy: "Never"
and pre-pulling the image on the host(s). - Example:
Copy code
my_flow.deploy(
        name="my-deployment",
        work_pool_name="my-docker-pool",
        job_variables={"image_pull_policy": "Never"},
        image="my-registry/app:1.2.3",
        push=False
      )
- Docs: Docker worker guide - Kubernetes worker: Kubernetes enforces
imagePullPolicy
at run time (default
IfNotPresent
). Set via job variables to `Always`/`IfNotPresent`/`Never`. Pre-pull images on nodes if you set
Never
. - Docs: Kubernetes worker guide - Flow code pulling (from Git via
flow.from_source(...)
) - The worker fetches code at run time via deployment “pull steps” (e.g.,
prefect.deployments.steps.git_clone
) before executing the flow. - To avoid runtime Git pulls, bake the code into your image (build your image with the code) and don’t use a Git storage pull step. - Docs: Store flow code, Pull steps (git_clone) If you can share which one you’re targeting (image vs code), I can give the exact config you need.
p
What if github down, how to make it use existing @Marvin?
m
thought for 152.7 seconds
Short answer: with the default settings, if GitHub is down the run will fail. Prefect’s Process worker tries to pull before execution; if that pull fails, it doesn’t fall back to the cached copy. How to make it use the existing code - Disable runtime pulls so it never tries to update just before a run:
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

flow.from_source(
    source=GitRepository(
        url="<https://github.com/org/repo.git>",
        pull_interval=None,     # do not pull at runtime
        # optional but recommended for stability:
        # commit_sha="abc123..."  # pin to a known-good commit
    ),
    entrypoint="flows.py:my_flow",
).deploy(name="stable-offline", work_pool_name="my-process-pool")
This uses whatever was cloned previously. Make sure the initial clone succeeds while GitHub is reachable. After that, the worker will reuse the local copy and won’t try to pull again during runs. Other robust options - Bake code into an image and use Docker/Kubernetes work pools. No git at runtime. - Use Local/Remote storage (e.g., S3/GCS/Azure) instead of Git as the runtime source. - Mirror GitHub to an internal git server and point
GitRepository.url
at the mirror. - Operationally, only roll workers (or re-deploy) when you want to refresh code. Important notes - With the default
pull_interval
(60s), a failed pull before a run will crash the run; there’s no built-in “use last good code” fallback. - Pinning a
commit_sha
improves determinism, but you still need to disable runtime pulls (
pull_interval=None
) to avoid failures when the remote is down. Docs for reference: - Store flow code (Git) - Deploy via Python - Runner storage API (GitRepository) Do you want periodic updates most of the time but strict fallback to cached code when GitHub is down? If so, we can discuss patterns to handle updates out-of-band (e.g., a controlled refresh job or a local mirror) while keeping
pull_interval=None
for runtime stability.