<@ULVA73B9P> I've tried making `git clone` work fo...
# ask-marvin
m
@Marvin I've tried making
git clone
work for my setup (private repository, self-deployed prefect on a k8s cluster), but it keeps failing Here is the error message:
Copy code
RuntimeError: Failed to clone repository '<https://github.com/xxx/xxx-api.git>'

 with exit code 128.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/runner/runner.py", line 1548, in _run_on_crashed_hooks
    flow = await load_flow_from_flow_run(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run
    output = await run_steps(
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 231, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone
10:15:18 AM
prefect.flow_runs.runner
I followed documentation for the
pull
step from here: https://reference.prefect.io/prefect/deployments/steps/pull/, neither approach has worked for me:
Copy code
pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/org/repo.git>
        credentials: "{{ prefect.blocks.github-credentials.my-github-credentials-block }}"
nor
Copy code
pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/org/repo.git>
        access_token: "{{ prefect.blocks.secret.github-access-token }}" # Requires creation of a Secret block
I am certain I have the secret defined as a
Block
, I tried both secret/github credentials, neither work. I created new tokens with different permissions, but I am sure that's not the problem because all of them say 'never used', meaning the issue is that the gitclone step does not pick-up the token from the block. What to do?
m
thought for 975.1 seconds
Thanks for sharing the details (and the exact error). Exit code 128 from git usually means “authentication failed” or “repo not reachable”. Given your tokens show “never used”, the clone is almost certainly happening without credentials. Here’s a focused checklist to get this working on Kubernetes with Prefect 3.x, plus a minimal working example. What to check first - Git is installed in the flow-run container image - If git is missing, clone will fail. Use a custom image for your Kubernetes pool/runner. - Example Dockerfile snippet:
Copy code
FROM prefecthq/prefect:3-python3.11
    RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var/lib/apt/lists/*
    # If you plan to use a GitHubCredentials block:
    RUN pip install "prefect-github>=0.5.0"
- If you use a GitHubCredentials block, the runtime image must have the integration installed - Without
prefect-github
in the container, Prefect can’t deserialize/use the
GitHubCredentials
block at runtime, and the step will fall back to unauthenticated clone. - Docs: Prefect GitHub integration - Block slugs and references are exact - For a Secret block named “github-access-token”: reference it as
{{ prefect.blocks.secret.github-access-token }}
. - For a GitHubCredentials block named “my-github-credentials-block”:
{{ prefect.blocks.github-credentials.my-github-credentials-block }}
. - Confirm the blocks exist in the same workspace the run is using (UI -> Blocks). - The runner can reach Prefect API to resolve blocks and can reach GitHub - If the pod can’t reach the Prefect API, the block reference can’t be resolved. - If the pod can’t reach github.com (egress policy/proxy/DNS), clone fails and tokens won’t be used. - Prefer one credential method at a time - Use either
access_token
(via Secret) or
credentials
(via integration block), not both. Minimal working example (no integration package required) 1) Create a Secret block with just the raw PAT string (no quotes, no user prefix): - UI -> Blocks -> Secret -> name it “github-access-token” 2) Use this prefect.yaml:
Copy code
pull:
     - id: repo
       prefect.deployments.steps.git_clone:
         repository: <https://github.com/org/repo.git>
         branch: main
         access_token: "{{ prefect.blocks.secret.github-access-token }}"
     - prefect.deployments.steps.set_working_directory:
         directory: "{{ repo.directory }}"

   deployments:
     - name: test
       entrypoint: path/to/your_flow.py:flow
3) Make sure your Kubernetes work pool/runner uses the custom image that includes git. 4) Run the deployment and check logs before the StepExecutionError for the actual git error line. If you prefer a GitHubCredentials block (requires prefect-github in the image) - Create the block (one-time; can be done in a scratch Python shell):
Copy code
from prefect_github import GitHubCredentials
  GitHubCredentials(token="ghp_...").save("my-github-credentials-block", overwrite=True)
- Then in prefect.yaml:
Copy code
pull:
    - id: repo
      prefect.deployments.steps.git_clone:
        repository: <https://github.com/org/repo.git>
        branch: main
        credentials: "{{ prefect.blocks.github-credentials.my-github-credentials-block }}"
Notes and common pitfalls - The git_clone step parameters in Prefect 3.x are: -
repository
(required),
branch
,
commit_sha
,
include_submodules
,
access_token
,
credentials
,
directories
(sparse-checkout) - No
ssh_key
, no
depth
parameter - API ref: git_clone step docs - Tokens showing “never used” almost always means the clone is not sending credentials at all: - Missing integration package when using credentials block - Block reference typo/slug mismatch - Runner cannot fetch blocks (API connectivity) - Clone attempted anonymously and failed
- Increase verbosity to see the underlying git error: - Set env on your Kubernetes pool/runner:
PREFECT_LOGGING_LEVEL=DEBUG
- You’ll see the git stderr line (e.g., “fatal: Authentication failed” or “Could not resolve host”). Alternative approach: define the source at deploy time - You can avoid a pull step entirely by deploying from source: - In code:
Copy code
from prefect_github import GitHubCredentials
    from my_flows import my_flow

    creds = GitHubCredentials.load("my-github-credentials-block")
    my_flow.from_source(
        source="<https://github.com/org/repo>",
        entrypoint="path/to/your_flow.py:my_flow",
        credentials=creds,
    ).deploy(name="from-source")
- This still requires the runtime image to have the integration package if you use GitHubCredentials. Helpful docs - git_clone step: prefect.deployments.steps.pull - prefect.yaml and templating: prefect.yaml guide - Storing secrets: Store secrets - Blocks overview: Blocks concept If you can share: - The image your Kubernetes pool/runner is using - The exact names/slugs of the blocks referenced - The one or two git stderr lines before the StepExecutionError …I can pinpoint which of the above is the culprit and get you to green quickly.
m
I just skipped the git clone step and baked the code into the docker image:
Copy code
pull:
  - prefect.deployments.steps.git_clone: null