https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> How do I resolve the error "The exist...
# ask-marvin
r
@Marvin How do I resolve the error "The existing repository at /opt/prefect/nil_etl-main does not match the configured repository" when using Prefect Cloud managed infrastructure?
m
✅ thought for 456.8 seconds
Short answer: you’re hitting a Git cache collision in the managed runtime. Prefect’s managed work pool keeps a cached checkout of your repo under /opt/prefect to speed up runs. If a flow run tries to pull a different repository (or the same repo via a differently-formatted URL) into a directory that already contains a Git repo, Prefect aborts with “The existing repository at /opt/prefect/<dir> does not match the configured repository”. Why it happens - Different repo in the same path: Two deployments (or a changed deployment) try to use the same local path but point to different Git URLs. - URL format drift: Same logical repo but different remote string (e.g., https://github.com/org/repo vs git@github.com:org/repo), which Prefect treats as different. - Directory reuse: Managed infrastructure reuses a cache directory under /opt/prefect between runs; a prior run may have created a repo there with a different origin. How to fix it (pick one) 1) Keep the exact same Git URL string across versions - Use a single canonical URL format everywhere (prefer HTTPS or SSH consistently). Even minor differences can trigger the mismatch. 2) Give each repo a unique local name/path - If you deploy via flow.from_source(...), set a unique name on the GitRepository so it uses a unique cache directory:
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

repo = GitRepository(
    url="<https://github.com/org/nil_etl.git>",
    branch="main",
    name="org-nil-etl-main",  # unique local directory name under /opt/prefect
)

@flow
def my_flow():
    ...

flow.from_source(
    source=repo,
    entrypoint="flows/etl.py:my_flow",
).deploy(
    name="prod",
    work_pool_name="<your-managed-pool>",
)
- Or set a unique working directory before cloning in your pull steps:
Copy code
pull:
  - prefect.deployments.steps.pull.set_working_directory:
      directory: /opt/prefect/repos/org-nil-etl-main
  - prefect.deployments.steps.pull.agit_clone:
      repository: <https://github.com/org/nil_etl.git>
      branch: main
3) Use agit_clone instead of git_clone in pull steps - agit_clone performs an atomic clone and tends to be more robust around pre-existing directories:
Copy code
pull:
  - prefect.deployments.steps.pull.agit_clone:
      repository: <https://github.com/org/nil_etl.git>
      branch: main
4) One-time cleanup if the cache is already “poisoned” - Temporarily add a pre-clone shell step to remove the path, then remove it after a successful run:
Copy code
pull:
  - prefect.deployments.steps.utility.run_shell_script:
      script: rm -rf /opt/prefect/nil_etl-main || true
  - prefect.deployments.steps.pull.agit_clone:
      repository: <https://github.com/org/nil_etl.git>
      branch: main
- Alternatively, change GitRepository.name (or the working directory) to a new unique name to bypass the old cache. A few notes - Multiple deployments that use the same repo but different branches are OK as long as the remote URL matches; the mismatch is about the remote identity, not the branch. - If you changed how you authenticate (SSH vs HTTPS), align all deployments on a single canonical URL to avoid conflicts. - In Prefect 3.x, the recommended pattern is flow.from_source(...).deploy(...). Avoid legacy 2.x methods. Docs and refs - Managed work pools: Managed infrastructure - Storing flow code with Git (from_source, GitRepository): Store flow code - prefect.yaml and pull steps: prefect.yaml and pull step API Pull steps API If you can share: - The exact repository URL in your deployment vs the one that created /opt/prefect/nil_etl-main previously - How you’re defining the deployment (from_source snippet or prefect.yaml pull steps)
…I can suggest the smallest, safest change for your setup.
r
@Marvin I deployed my flows using this command: 
ron@Mac nil_etl % prefect-cloud deploy src/flows/sportradar.py:sportradar --name "Sportradar Sync" --from Marut-AI/nil_etl/
When I first deployed, a few of them were deployed using a different branch, but they are now all on the same branch. How can I clear the cache?
m
✅ thought for 94.0 seconds
You’re running into a cached Git checkout under /opt/prefect from an earlier deployment that used a different Git URL/branch. In Prefect Cloud managed pools, that cache is reused between runs for speed. To clear it, the easiest path is to add a one-time cleanup pull step that deletes the cached directory before cloning, then remove it after a successful run. Option A — one-time cleanup via pull steps (prefect.yaml) 1) Add a prefect.yaml to your repo root (or edit your existing one) with a cleanup step before cloning, then redeploy:
Copy code
name: nil-etl

pull:
  - prefect.deployments.steps.utility.run_shell_script:
      # adjust the path to match your error message
      script: rm -rf /opt/prefect/nil_etl-main || true
  - prefect.deployments.steps.pull.agit_clone:
      repository: <https://github.com/Marut-AI/nil_etl>
      branch: main
      # include_submodules: false   # if you need it, set to true
      # credentials: <optional block>  # if you use a block for auth

deployments:
  - name: Sportradar Sync
    entrypoint: src/flows/sportradar.py:sportradar
    work_pool:
      name: <your-managed-pool-name>
2) Redeploy (your 
prefect-cloud deploy
wrapper or 
prefect deploy
will pick it up) and kick off one run. 3) Once the run succeeds, remove the cleanup step (the run_shell_script) so you don’t delete the directory on every run, and redeploy again. Option B — avoid the collision going forward - Use a unique local directory name for this repo so it won’t collide with any previous cache: - If you deploy in Python, set GitRepository.name so Prefect uses a distinct path:
Copy code
from prefect import flow
    from prefect.runner.storage import GitRepository

    repo = GitRepository(
        url="<https://github.com/Marut-AI/nil_etl>",
        branch="main",
        name="marut-ai-nil-etl-main"  # unique local dir under /opt/prefect
    )

    @flow
    def sportradar():
        ...

    sportradar.from_source(
        source=repo,
        entrypoint="src/flows/sportradar.py:sportradar",
    ).deploy(
        name="Sportradar Sync",
        work_pool_name="<your-managed-pool-name>",
    )
- Or in prefect.yaml, set a working directory before cloning to isolate repos:
Copy code
pull:
    - prefect.deployments.steps.pull.set_working_directory:
        directory: /opt/prefect/repos/marut-ai-nil-etl-main
    - prefect.deployments.steps.pull.agit_clone:
        repository: <https://github.com/Marut-AI/nil_etl>
        branch: main
Tips - Keep the Git URL string identical across deployments (e.g., always use HTTPS or always SSH). Even small differences (https vs ssh) are treated as different remotes and can trigger this error. - Prefer agit_clone over git_clone in managed environments; it’s more robust with existing directories. Docs - Managed infrastructure: Managed infrastructure - Storing flow code (from_source, GitRepository): Store flow code - prefect.yaml and pull steps: prefect.yaml and Pull steps API If you want, paste the exact error path (e.g., /opt/prefect/nil_etl-main) and I’ll tailor the cleanup snippet precisely to your deployment.
r
@Marvin There is no way to clear the cache aside from redeploying code multiple times?
m
✅ thought for 27.1 seconds
Short answer: there isn’t a “clear cache” button/command for Prefect Cloud managed infrastructure. Because you can’t SSH into the managed runtime, the way to clear or avoid the cached repo is to change what the runtime does at startup. Good news: you do not need to redeploy multiple times unless you want to remove the cleanup afterward. Here are your options: - One redeploy with a permanent cleanup step - Add a pre-clone shell step that removes the cached path. Leave it in place so every run starts clean (slightly slower, but simple and only one redeploy).
Copy code
pull:
    - prefect.deployments.steps.utility.run_shell_script:
        script: rm -rf /opt/prefect/nil_etl-main || true
    - prefect.deployments.steps.pull.agit_clone:
        repository: <https://github.com/Marut-AI/nil_etl>
        branch: main
- This works even if previous runs used a different branch/URL. - One redeploy with a new working directory (no cleanup needed) - Give this repo a unique path so it doesn’t collide with the old cache:
Copy code
pull:
    - prefect.deployments.steps.pull.set_working_directory:
        directory: /opt/prefect/repos/marut-ai-nil-etl-main
    - prefect.deployments.steps.pull.agit_clone:
        repository: <https://github.com/Marut-AI/nil_etl>
        branch: main
- Or, if you deploy in Python, set 
GitRepository.name="marut-ai-nil-etl-main"
to get a unique cache directory automatically. - Create a new managed work pool (fresh environment) - If convenient, pointing the deployment at a new managed pool also avoids any existing cache with a single redeploy. Notes - Prefer 
agit_clone
over 
git_clone
for robustness. - Keep your Git URL format consistent across deployments (HTTPS vs SSH). A different remote string will trigger the “does not match the configured repository” error even for the same repo. Docs - Managed infrastructure: Managed infrastructure - Storing flow code (from_source, GitRepository): Store flow code - prefect.yaml and pull steps: prefect.yaml and Pull steps API If you share the exact cached path from the error (e.g., 
/opt/prefect/nil_etl-main
), I can give you a copy/paste cleanup snippet tailored to your setup.
r
@Marvin Can I just delete the work pool and move my deployments to a new work pool?
m
✅ thought for 136.9 seconds
Yes — creating a new Managed work pool and pointing your deployments to it will give you a fresh environment with no prior repo cache. That’s a perfectly valid way to bypass the old cache. Recommended steps - Create a new Managed work pool (UI: Work Pools > Create > Managed). - Update each deployment to use the new pool: - Easiest: In the UI, open the deployment > Edit > change Work pool to the new one > Save. - Or redeploy from code, pointing to the new pool (e.g., via your normal deploy command or by setting 
work_pool.name
in prefect.yaml). - After all deployments are moved and running, delete the old pool:
Copy code
prefect work-pool delete "<old-pool-name>"
Notes - Update deployments first to avoid runs targeting a deleted pool. - This approach avoids needing cleanup steps; you’ll start with a clean cache. - If you prefer to keep the existing pool, you can still fix this by adding a one-time cleanup pull step or by setting a unique working directory, as outlined earlier. Docs - Managed infrastructure overview: Managed infrastructure
r
@Marvin After redeploying all of my flows, they all ran once, but now do not run and I get this error: Runner failed to retrieve flow to execute on_crashed hooks for flow run UUID('068fbc84-629c-7341-8000-70d79e10dfe1'). Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/prefect/runner/storage.py", line 451, in _clone_repo await run_process(cmd) File "/usr/local/lib/python3.12/site-packages/anyio/_core/_subprocesses.py", line 123, in run_process raise CalledProcessError(cast(int, process.returncode), command, output, errors) subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/Marut-AI/nil_etl', '--branch', 'main', '--depth', '1', '/opt/prefect/nil_etl-main']' returned non-zero exit status 128. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps step_output = await run_step(step, upstream_outputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 137, in run_step result = await from_async.call_soon_in_new_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult return await asyncio.wrap_future(self.future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/pull.py", line 94, in agit_clone await _pull_git_repository_with_retries(storage) File "/usr/local/lib/python3.12/site-packages/prefect/_internal/retries.py", line 57, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/pull.py", line 47, in _pull_git_repository_with_retries await repo.pull_code() File "/usr/local/lib/python3.12/site-packages/prefect/runner/storage.py", line 415, in pull_code await self._clone_repo() File "/usr/local/lib/python3.12/site-packages/prefect/runner/storage.py", line 460, in _clone_repo raise RuntimeError( RuntimeError: Failed to clone repository 'https://github.com/Marut-AI/nil_etl' with exit code 128. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 1548, in _run_on_crashed_hooks flow = await load_flow_from_flow_run( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run output = await run_steps( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps raise StepExecutionError(f"Encountered error while running {fqn}") from exc prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.pull.agit_clone I have successfully configured prefect-cloud by running github setup. What could be causing the flows to fail?
m
✅ thought for 287.4 seconds
Thanks for sharing the stack trace — the key part is this:
Copy code
git clone <https://github.com/Marut-AI/nil_etl> --branch main --depth 1 ...
...
RuntimeError: Failed to clone repository ... with exit code 128.
That means the runner could not clone your repo at runtime. The “on_crashed hooks” message is just a follow-on effect: when the run crashed, Prefect tried to reload your flow to execute hooks and failed again because the repo couldn’t be pulled. A few likely causes and how to fix them: - Private repo without runtime credentials: The Prefect Cloud GitHub “setup” does not give your workers permission to clone code. You must provide Git credentials in the deployment so the worker can pull your repo at run time. - Token issues: Expired/rotated token, missing repo access, or org SSO not authorized for the token, all lead to git exit 128. - URL/branch mismatch or repo moved/renamed. - Network/egress restrictions from your worker environment. How to provide credentials in Prefect 3.x 1) Using the Python API (recommended for private repos) - Create a GitHub Personal Access Token (fine‑grained with read access to this repo or classic with repo scope) and, if your org enforces SSO, authorize it for the org. - Store it as a Prefect Secret block (e.g., “github-pat”). - Use a GitRepository with credentials when deploying:
Copy code
from prefect.blocks.system import Secret
from prefect.runner.storage import GitRepository, GitCredentials
from prefect.flows import Flow

# Load your token from a Prefect Secret block
token = Secret.load("github-pat")

storage = GitRepository(
    url="<https://github.com/Marut-AI/nil_etl>",
    branch="main",
    credentials=GitCredentials(access_token=token),
)

# Replace with your entrypoint: "path/to/file.py:flow_function_name"
flow = Flow.from_source(storage, entrypoint="flows/my_flow.py:my_flow")

flow.deploy(
    name="prod",
    work_pool_name="YOUR_WORK_POOL",
)
Docs: Deploy via Python and Store your flow’s code 2) Using prefect.yaml with pull steps If you’re using 
prefect deploy
, set a git_clone pull step and reference the Secret:
Copy code
deployments:
  - name: prod
    work_pool:
      name: YOUR_WORK_POOL
    entrypoint: flows/my_flow.py:my_flow
    pull:
      - prefect.deployments.steps.pull.git_clone:
          repository: "<https://github.com/Marut-AI/nil_etl>"
          branch: "main"
          access_token: "{{ prefect.blocks.system.Secret.load('github-pat') }}"
Docs: Pull steps API and prefect.yaml reference Why the first run worked once - Often the first run can succeed because the code was present locally at deploy time or cached, then later runs try to pull fresh and fail without credentials. Quick debugging checklist - From the worker environment, try: - 
git clone <https://github.com/Marut-AI/nil_etl> --branch main --depth 1
(you should get an auth error if it’s private without creds). - Verify the token is valid, has access to the repo, and is SSO-authorized if your org requires it. - Confirm your deployment actually references the Secret/token (either via 
GitRepository(..., credentials=...)
or the 
git_clone
pull step). - Ensure the worker has outbound access to github.com. If you can share: - Whether the repo is private, - How you created the deployment (Python vs prefect.yaml) and your deployment snippet, - The worker type and where it runs, I can spot-check the config and point to the exact fix. Relevant docs for reference: - Store your flow’s code (GitRepository, credentials) - Store secrets in Prefect (Secret block)
r
@Marvin The repo is a private repo, which I have deployed from in the past without issue. I am deploying via yaml, which is here: name: nil-etl pull: - prefect.deployments.steps.utility.run_shell_script: script: rm -rf /opt/prefect/nil_etl-main || true - prefect.deployments.steps.pull.agit_clone: repository: https://github.com/Marut-AI/nil_etl branch: main access_token: "{{ prefect.blocks.system.Secret.load('github-access-nil-etl')}}" deployments: - name: Sportradar Sync entrypoint: src/flows/sportradar.py:sportradar work_pool: name: ph_nil_development I still get the exit code 128 error.
m
✅ thought for 52.3 seconds
Thanks — the stack trace actually contains the smoking gun:
Copy code
['git', 'clone', '<https://github.com/Marut-AI/nil_etl>', '--branch', 'main', ...]
The repository URL includes literal angle brackets. Git will treat 
'<https://...>'
as an invalid URL and return exit code 128. I also see a second issue in your yaml: the Secret block value isn’t being dereferenced. Two fixes: 1) Remove angle brackets around the repo URL 2) Call 
.get()
on the Secret so the step receives the token string Try this updated prefect.yaml:
Copy code
name: nil-etl

pull:
  - prefect.deployments.steps.utility.run_shell_script:
      script: rm -rf /opt/prefect/nil_etl-main || true
  - prefect.deployments.steps.pull.git_clone:
      repository: "<https://github.com/Marut-AI/nil_etl>"
      branch: "main"
      access_token: "{{ prefect.blocks.system.Secret.load('github-access-nil-etl').get() }}"

deployments:
  - name: Sportradar Sync
    entrypoint: src/flows/sportradar.py:sportradar
    work_pool:
      name: ph_nil_development
Notes: - I switched to 
git_clone
since that’s the documented step name, though 
agit_clone
is the async variant used internally. Using 
git_clone
in YAML is clearer and equivalent here. - Verify the Secret block 
github-access-nil-etl
exists in your Prefect workspace and contains a valid GitHub PAT with repo read access. If your org enforces SSO, ensure the token is SSO-authorized for the org. If it still fails: - From the worker environment, try cloning manually to confirm the token works:
Copy code
git clone https://<YOUR_TOKEN>@github.com/Marut-AI/nil_etl --branch main --depth 1
If this fails, it’s a token/permissions/SSO issue. - Double-check the exact repo path and branch name. - Ensure the worker has outbound network access to github.com. Docs for reference: - Pull steps (git_clone) - Store secrets in Prefect - prefect.yaml reference If you share a redacted version of your Secret’s scopes and where the worker is running, I can help double-check the remaining pieces.
r
@Marvin using the token to clone the git repo manually works, however when deploying the code, it still fails to clone. I have deployed from git before, and did not need to configure a PAT, as the deployment would use whatever was stored by prefect-cloud github token
m
✅ thought for 311.6 seconds
Got it — thanks for the extra context. Two important clarifications up front: - The Prefect Cloud GitHub “setup”/App does not provide runtime credentials to workers. In Prefect 3.x you must pass credentials in your deployment (pull steps or storage). This is by design. - Your earlier logs showed the repo URL with angle brackets. If those are present in your YAML, git will always return exit code 128. Given that you can clone manually with the token, the remaining common causes are: - The YAML still has angle brackets around the URL (Slack formatting creeping into the file). - The Secret block is not being dereferenced to a raw string (Jinja didn’t render, or the Secret value is not just the token). - Org SSO not authorized for the token (works on your laptop if you’ve authorized, but not in the worker’s context). - Permission to write to /opt/prefect (less likely but worth confirming). Please try one of these two known-good configurations: Option A: Use access_token directly (ensure no angle brackets and call .get())
Copy code
name: nil-etl

pull:
  - prefect.deployments.steps.utility.run_shell_script:
      script: rm -rf /opt/prefect/nil_etl-main || true
  - prefect.deployments.steps.pull.git_clone:
      repository: "<https://github.com/Marut-AI/nil_etl>"
      branch: "main"
      access_token: "{{ prefect.blocks.system.Secret.load('github-access-nil-etl').get() | trim }}"

deployments:
  - name: Sportradar Sync
    entrypoint: src/flows/sportradar.py:sportradar
    work_pool:
      name: ph_nil_development
Option B: Use the GitHubCredentials block (prefect-github) - Install/register once, then reference the block:
Copy code
# (once) pip install prefect-github && prefect block register -m prefect_github
# Create a GitHubCredentials block in the UI named "github-cred-nil-etl" with your PAT

name: nil-etl

pull:
  - prefect.deployments.steps.utility.run_shell_script:
      script: rm -rf /opt/prefect/nil_etl-main || true
  - prefect.deployments.steps.pull.git_clone:
      repository: "<https://github.com/Marut-AI/nil_etl>"
      branch: "main"
      credentials: "{{ prefect_github.GitHubCredentials.load('github-cred-nil-etl') }}"

deployments:
  - name: Sportradar Sync
    entrypoint: src/flows/sportradar.py:sportradar
    work_pool:
      name: ph_nil_development
Things to double-check: - The repo URL must not have angle brackets. It should be exactly: - 
repository: "<https://github.com/Marut-AI/nil_etl>"
- The Secret block value must be the raw PAT string. If you stored JSON like {"token":"..."} you’ll need to index it: - 
access_token: "{{ prefect.blocks.system.Secret.load('github-access-nil-etl').get()['token'] | trim }}"
- If your org enforces SSO, make sure the PAT is SSO-authorized for the org. - Ensure the worker can write to /opt/prefect. If you see “Permission denied” in stderr, adjust permissions or run the worker with the right user. If it still fails, could you share the git stderr line from the worker logs? That will tell us if it’s “Repository not found”, “Authentication failed”, or a filesystem error. You can also temporarily set: - In your work pool or deployment job variables: 
PREFECT_LOGGING_LEVEL=DEBUG
to get more detail. Relevant docs: - Pull steps (git_clone): Pull steps API - prefect.yaml reference: prefect.yaml - Storing secrets: Store secrets in Prefect - GitHub integration (blocks): prefect-github integration If you paste the exact stderr from the failing git clone, I can pinpoint the root cause quickly.
29 Views
Open in Slack
PreviousNext
Powered by