<@ULVA73B9P> When using a git clone pull step on a...
# marvin-ai
j
@Marvin When using a git clone pull step on a process worker deployment all flow runs will crash if the source provider is down. Does there exist a way to cache the git repository on disk when spawning a flow run so it doesn't have to clone from the git host each time? Ideally validated by the commit ref. If no could this be easily extended?
m
thought for 290.7 seconds
Short answer: yes—on a Process worker you can cache repos between runs so you’re not cloning every time, and if you pin to a commit SHA Prefect will skip any network call when the local checkout already matches that commit. How to get caching in practice - Use a persistent working directory for the worker so clones are reused: - Configure the Process work pool’s “Working directory” (UI → Work Pools → your pool → Edit) to a host path like
/opt/prefect/flows
. The worker will run every flow in that same directory. - Alternatively, add a pull step to set the working directory:
Copy code
pull:
      - prefect.deployments.steps.set_working_directory:
          directory: /opt/prefect/flows
Note: this only helps if that path is actually persistent on the machine/volume where the worker runs. - Pin your deployment to a commit SHA in the git pull step:
Copy code
pull:
    - prefect.deployments.steps.set_working_directory:
        directory: /opt/prefect/flows
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/owner/repo.git>
        commit_sha: "<commit-sha>"
With a persistent working dir: - First run clones and checks out the commit. - Subsequent runs detect the repo is already at that commit and no-op locally (no remote call), so runs won’t crash if the git host is down. Notes and caveats - If you pull by branch (not commit), the step will attempt a
git pull
and will fail if the provider is down. For maximum resilience, pin to a commit SHA. - The built-in step does shallow clones by default and reuses existing repos in the working directory. It validates by commit SHA when provided. - If you run workers in ephemeral environments (e.g., new containers/pods each run) you’ll need a persistent volume mount for the working directory to get the benefits. If you need even stronger offline behavior - You can create a small custom pull step that: - Checks for
.git
and whether the requested commit exists locally. - If it does, returns early without any git network commands. - If not, it fetches only as needed. This is a straightforward extension because pull steps are just Python callables. References - Process worker working directory (source): prefect/workers/process.py - Git pull step implementation (source): prefect/deployments/steps/pull.py - Deployments and pull steps overview: Deployments docs A couple quick questions to make sure this will work for you: - Are you able to set a persistent working directory on the machine (or mount a persistent volume if containerized)? - Are your deployments pinned to commit SHAs, or are you currently pulling by branch?