https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> with prefect 3.X i use python sdk to ...
# ask-marvin
e
@Marvin with prefect 3.X i use python sdk to deploy() the fit clone fail at the auth while is use gitrepository() with 
credential = Secret.load("mysecret")
why ?
m
✅ thought for 276.8 seconds
Short answer: passing a Secret block directly to GitRepository won’t format auth for git. You need to either: - wrap the Secret in a credentials dict with the right key, or - use a provider-specific credentials block (recommended) Why it fails - 
Secret.load("mysecret")
returns a Secret Block object. 
GitRepository
only knows how to turn credentials into a git URL if: - you pass a dict with known keys like 
access_token
, 
token
, or 
password
, or - you pass a provider-specific credentials block (e.g., 
GitHubCredentials
, 
GitLabCredentials
) that implements git URL formatting. - Passing a Secret block by itself doesn’t tell Prefect what key it is (token vs password), so the clone happens without auth and fails. What to do instead Option A: Keep using Secret, but wrap it in a dict
Copy code
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

repo = GitRepository(
    url="<https://github.com/org/repo.git>",
    credentials={"access_token": Secret.load("mysecret")},
    branch="main",
)
flow.from_source(source=repo, entrypoint="flows.py:my_flow").deploy(
    name="my-deployment",
    work_pool_name="my-pool",
)
Option B (preferred): Use provider-specific credentials blocks - GitHub:
Copy code
from prefect.runner.storage import GitRepository
from prefect_github import GitHubCredentials

repo = GitRepository(
    url="<https://github.com/org/repo.git>",
    credentials=GitHubCredentials.load("my-github-creds"),
)
- GitLab:
Copy code
from prefect.runner.storage import GitRepository
from prefect_gitlab import GitLabCredentials

repo = GitRepository(
    url="<https://gitlab.com/org/repo.git>",
    credentials=GitLabCredentials.load("my-gitlab-creds"),
)
- Bitbucket:
Copy code
from prefect.runner.storage import GitRepository
from prefect_bitbucket import BitBucketCredentials

repo = GitRepository(
    url="<https://bitbucket.org/org/repo.git>",
    credentials=BitBucketCredentials.load("my-bitbucket-creds"),
)
Notes - Make sure you’re using HTTPS URLs (token-in-URL auth), not SSH. - If you include a 
username
in the credentials dict, you must also include `access_token`/`token`/`password`. - Verify your Secret block stores only the token string (no “Bearer ” prefix). Docs - Store flow code (GitRepository examples): Store flow code - Deploy via Python: Deploy via Python - Blocks (Secret): Blocks concept - GitHub integration: prefect-github If you share: - which git provider you’re using (GitHub/GitLab/Bitbucket), - your 
GitRepository(...)
snippet, - and the exact auth error from the logs, I can tailor the example and confirm the right keys/format for your case.
e
Copy code
flow.from_source(
  source=GitRepository(
    url="<https://github.com/org/repo.git>",
    credentials={
      "username": Secret.load("myuser"),
      "access_token": Secret.load("mysecret"),
    },
    branch="main",
  ),
 entrypoint="flows.py:my_flow"
).deploy(
    name="my-deployment",
    work_pool_name="my-pool",
)
@Marvin i'm using this
m
✅ thought for 41.9 seconds
Thanks for sharing the snippet — a couple of common gotchas can cause auth failures here. Likely issues in your example - The URL includes angle brackets: 
"<https://github.com/org/repo.git>"
. If that’s literal in your code, git will fail to parse/authenticate. Use 
<https://github.com/org/repo.git>
(no 
< >
). - GitHub username handling: with a PAT, GitHub expects your actual GitHub username + the token as the password. If your 
myuser
Secret isn’t your GitHub username (or is empty/incorrect), auth will fail. In many cases you can omit 
username
entirely and just provide the token. Two solid ways to fix Option A (quick): keep a Secret, pass it as 
access_token
, and remove username
Copy code
flow.from_source(
  source=GitRepository(
    url="<https://github.com/org/repo.git>",
    credentials={
      "access_token": Secret.load("mysecret"),
    },
    branch="main",
  ),
  entrypoint="flows.py:my_flow",
).deploy(
  name="my-deployment",
  work_pool_name="my-pool",
)
Option B (recommended): use the GitHub credentials block - Install and register the integration once:
Copy code
pip install -U prefect-github
prefect block register -m prefect_github
- Create and use the block:
Copy code
from prefect_github import GitHubCredentials
from prefect.runner.storage import GitRepository
from prefect import flow

# One-time setup (can be done in a separate script)
# GitHubCredentials(token="ghp_xxx").save("my-github-creds", overwrite=True)

flow.from_source(
  source=GitRepository(
    url="<https://github.com/org/repo.git>",
    credentials=GitHubCredentials.load("my-github-creds"),
    branch="main",
  ),
  entrypoint="flows.py:my_flow",
).deploy(
  name="my-deployment",
  work_pool_name="my-pool",
)
Checklist if it still fails - Ensure the PAT has the right scopes (typically “repo” for private repos). - Make sure your Secret block contains only the token string (no quotes/newlines). - Confirm your worker can resolve blocks from your Prefect workspace (the block name must exist in that workspace, and the worker must be authenticated to it). - Ensure 
git
is available in the worker environment/image. - Use HTTPS, not SSH, with GitRepository. Docs for reference - Store flow code (GitRepository): Store flow code - Deploy via Python: Deploy via Python - GitHub integration: prefect-github If you remove the angle brackets and switch to Option A or B and still see failures, could you share the exact git error from the worker logs? I can pinpoint whether it’s a token/username mismatch vs. block resolution vs. network.
e
@Marvin sorry i miss type i use gitlab.com not github
m
✅ thought for 47.1 seconds
Got it — for GitLab specifically, here’s how to make auth reliable. Best approach (recommended): use the GitLab credentials block - Install and register once:
Copy code
pip install -U prefect-gitlab
prefect block register -m prefect_gitlab
- Create the block (one-time; can be run in a separate script or notebook):
Copy code
from prefect_gitlab import GitLabCredentials
GitLabCredentials(token="glpat_your_token_here").save("my-gitlab-creds", overwrite=True)
- Use it in your deployment:
Copy code
from prefect.runner.storage import GitRepository
from prefect_gitlab import GitLabCredentials

flow.from_source(
  source=GitRepository(
    url="<https://gitlab.com/org/repo.git>",
    credentials=GitLabCredentials.load("my-gitlab-creds"),
    branch="main",
  ),
  entrypoint="flows.py:my_flow",
).deploy(
  name="my-deployment",
  work_pool_name="my-pool",
)
Why this helps: the GitLabCredentials block formats the URL the way GitLab expects (e.g., 
oauth2:TOKEN@...
) and handles edge cases like deploy tokens. If you prefer to stick with Secret blocks - For a GitLab Personal Access Token (PAT), use username 
oauth2
with the token as the access token. Don’t store the username in a Secret block unless you need to; set it directly.
Copy code
flow.from_source(
  source=GitRepository(
    url="<https://gitlab.com/org/repo.git>",
    credentials={
      "username": "oauth2",
      "access_token": Secret.load("mysecret"),  # contains your glpat-xxxx token
    },
    branch="main",
  ),
  entrypoint="flows.py:my_flow",
).deploy(
  name="my-deployment",
  work_pool_name="my-pool",
)
Deploy tokens (GitLab “Deploy Token” with username + password)
Copy code
flow.from_source(
  source=GitRepository(
    url="<https://gitlab.com/org/repo.git>",
    credentials={
      "username": Secret.load("deploy_token_username"),
      "password": Secret.load("deploy_token_password"),
    },
    branch="main",
  ),
  entrypoint="flows.py:my_flow",
).deploy(
  name="my-deployment",
  work_pool_name="my-pool",
)
Common pitfalls to check - The repo URL must be HTTPS and have no angle brackets: 
<https://gitlab.com/org/repo.git>
- Token scopes: for cloning private repos, the PAT needs at least the “read_repository” scope. - Make sure your Secret values contain only the raw token/username (no quotes or extra whitespace/newlines). - If you pass a 
username
, ensure it’s either your actual GitLab username or 
oauth2
(for PATs). Using the wrong username will fail auth. - Ensure 
git
is installed in the worker environment and workers can reach gitlab.com. - If you’re self-hosted GitLab, use your correct hostname (e.g., `https://gitlab.mycompany.com/...`). Docs - Store flow code (Git, GitRepository): Store flow code - Deploy via Python: Deploy via Python - GitLab integration: prefect-gitlab If it still fails, can you share the exact error from the worker logs (e.g., auth failed, 401/403, prompt for username/password)? I can pinpoint whether it’s a token scope/format vs. block resolution vs. network issue.
e
@Marvin Failed to clone repository 'https://gitlab.com/g/f/d.git' with exit code 128.
m
✅ thought for 41.0 seconds
Exit code 128 is a generic git clone failure. It’s usually one of: - Auth failed (wrong username/token format, missing scopes) - Repo not found or wrong URL - Network/SSL issues in the worker environment Quickest fix for GitLab: use the GitLab credentials block so Prefect formats auth correctly for you.
Copy code
pip install -U prefect-gitlab
prefect block register -m prefect_gitlab
Then:
Copy code
from prefect.runner.storage import GitRepository
from prefect_gitlab import GitLabCredentials

flow.from_source(
  source=GitRepository(
    url="<https://gitlab.com/g/f/d.git>",
    credentials=GitLabCredentials.load("my-gitlab-creds"),
    branch="main",
  ),
  entrypoint="flows.py:my_flow",
).deploy(
  name="my-deployment",
  work_pool_name="my-pool",
)
If you haven’t created the block yet:
Copy code
from prefect_gitlab import GitLabCredentials
GitLabCredentials(token="glpat_xxx").save("my-gitlab-creds", overwrite=True)
If you want to keep Secrets instead of the provider block - For a GitLab Personal Access Token (PAT), use username 
oauth2
and the token as the access token:
Copy code
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

flow.from_source(
  source=GitRepository(
    url="<https://gitlab.com/g/f/d.git>",
    credentials={
      "username": "oauth2",
      "access_token": Secret.load("mysecret"),  # the raw glpat_xxx string
    },
    branch="main",
  ),
  entrypoint="flows.py:my_flow",
).deploy(
  name="my-deployment",
  work_pool_name="my-pool",
)
- For a GitLab Deploy Token, use its username/password:
Copy code
credentials={
  "username": Secret.load("deploy_token_username"),
  "password": Secret.load("deploy_token_password"),
}
Sanity checks - URL must be HTTPS with no angle brackets in code: 
<https://gitlab.com/g/f/d.git>
- PAT should have at least “read_repository” scope. - Secret values must be the raw string (no quotes/newlines). - git must be installed in the worker’s environment and the worker must have network access to gitlab.com. Quick local test from the worker environment - PAT:
Copy code
git ls-remote <https://oauth2:<YOUR_TOKEN>@gitlab.com/g/f/d.git>
- Deploy token:
Copy code
git ls-remote https://<USERNAME>:<PASSWORD>@gitlab.com/g/f/d.git
If this fails, you’ll get the real git error (auth vs. not found vs. SSL). If it still fails in Prefect - Turn on debug logs to capture git stderr: - Set env var on the worker: 
PREFECT_LOGGING_LEVEL=DEBUG
- Share the exact git error line (e.g., “Authentication failed”, “Repository not found”, “SSL certificate problem”, etc.) and whether you’re using a PAT or a Deploy Token. I can then pinpoint the cause. Docs - Store flow code (Git): Store flow code - Deploy via Python: Deploy via Python - GitLab integration: prefect-gitlab
e
@Marvin ls-remote work correctly , is there any way to check the credential has been populated correctly in worker ?
m
✅ thought for 546.9 seconds
Got it. Assuming you’re on Prefect 3.x: yes—there are a few concrete ways to verify that Git credentials are available to the worker that will clone your repo. Here are the quickest checks that usually catch issues. 1) Inspect the deployment’s pull steps This confirms the deployment actually includes a credential reference that the worker will resolve at runtime.
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
Look for a 
pull_steps
section—e.g., a 
prefect.deployments.steps.pull.git_clone
step with either: - 
credentials: "{{ prefect.blocks.github-credentials.my-block }}"
, or - 
access_token: "{{ prefect.blocks.secret.my-github-token }}"
Docs: - How to retrieve code from storage - Pull steps API - GitRepository storage 2) Start the worker with debug logging and watch for pull-step execution This shows the worker resolving blocks and running the 
git_clone
step (tokens will be redacted in logs).
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool <your-pool>
You should see logs like “Running pull step prefect.deployments.steps.pull.git_clone…”. Note: The env var is 
PREFECT_LOGGING_LEVEL
in 3.x. - Settings and profiles - Settings reference 3) Load the credential block from the worker runtime If you’re using a Secret block (or a credentials block), try loading it from the same environment/image your worker uses. - Secret block:
Copy code
python -c "from prefect.blocks.system import Secret; print('Loaded:', bool(Secret.load('github-token').get()))"
- GitHubCredentials block (if you’re using the integration and have 
prefect-github
installed):
Copy code
python -c "from prefect_github import GitHubCredentials; c=GitHubCredentials.load('my-gh-creds'); print('Has token:', bool(c.token.get_secret_value()))"
4) Manually test git inside the worker’s runtime “git ls-remote works” on your laptop is good, but the worker may be in a different container/VM. Test in the same environment the worker uses: - Docker worker: - 
docker exec -it <worker-container> sh
- Then:
Copy code
GIT_TRACE=1 git ls-remote https://<token>@github.com/org/repo.git
or if using SSH:
Copy code
GIT_SSH_COMMAND="ssh -v" git ls-remote git@github.com:org/repo.git
- Kubernetes worker: - 
kubectl exec -it <worker-pod> -- sh
- Run the same 
git
commands above 5) Trigger a run and watch logs Kick off a flow run and watch worker logs. A bad/expired token usually shows as 
fatal: Authentication failed
or 
exit code 128
from the 
git clone
step. Submodule issues will show separate auth errors if you use submodules without credentials configured for them. Common gotchas - Token scopes: For GitHub, ensure the token has at least “Contents: Read” on the repo. - Different runtime: Your worker image/pod may not have your local git config/SSH agent—always test inside the worker runtime. - Submodules: If used, include credentials and enable submodule fetching or ensure the token is valid for submodule repos. - Logs are redacted: You won’t see tokens in logs by design—use block-load tests to confirm presence. If you can share: - Worker type (Process, Docker, Kubernetes) - How you configured credentials (Secret block, GitHubCredentials block, SSH, env var) - The 
pull_steps
from 
prefect deployment inspect
…I can give a more targeted set of checks for your setup.
e
@Marvin 'credentials': '{{ prefect.blocks.gitlab-credentials.gitlab-lh-access-token }}'}
m
✅ thought for 198.1 seconds
Thanks for sharing the templated value. That syntax looks correct for a GitLab credentials block in Prefect 3.x:
Copy code
'credentials': '{{ prefect.blocks.gitlab-credentials.gitlab-lh-access-token }}'
This assumes you have a GitLabCredentials block named 
gitlab-lh-access-token
and the 
prefect-gitlab
integration is installed/registered. Quick ways to verify it’s available to the worker 1) Confirm the block exists and is the right type - List and inspect blocks:
Copy code
prefect block ls
prefect block inspect gitlab-credentials/gitlab-lh-access-token
If you see “unknown block type,” ensure the integration is installed in the worker environment/image and the block type is registered:
Copy code
pip install "prefect[gitlab]"   # or: pip install prefect-gitlab
prefect block register -m prefect_gitlab
2) Load the block from the worker runtime Run this in the same environment/image your worker uses (Docker/K8s shell or the host if it’s a process worker):
Copy code
python -c "from prefect_gitlab import GitLabCredentials; c=GitLabCredentials.load('gitlab-lh-access-token'); print('Loaded:', bool(c.token.get_secret_value()))"
Seeing 
Loaded: True
confirms the worker can fetch and decrypt the credential (token value is not printed for security). 3) Inspect the deployment’s pull steps and run with debug logs - Check the deployment includes that templated credential:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
- Start the worker with debug logging and watch for the 
git_clone
step:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool <your-pool>
Docs and references - How to retrieve code from storage: Git-based storage guide - Pull steps API (git_clone): Pull steps API - GitLab integration: prefect-gitlab integration If you tell me which worker type you’re using (Process/Docker/Kubernetes) I can tailor the exact check steps for that environment.
e
@Marvin k8s
m
✅ thought for 88.7 seconds
Great — for Kubernetes work pools, an important nuance: the Git credential is resolved and used inside the flow run pod (the Kubernetes Job container), not in the long‑running worker process. So to verify it’s “populated,” check from the job pod where the pull step runs. Here’s a quick, K8s-focused checklist: 1) Verify the deployment has the templated credential
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
Confirm you see something like:
Copy code
prefect.deployments.steps.pull.git_clone:
  repository: ...
  credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-lh-access-token }}"
2) Trigger a run and watch the flow pod logs - Start a run:
Copy code
prefect deployment run "<flow-name>/<deployment-name>"
- Find the new pod (adjust namespace):
Copy code
kubectl get pods -n <namespace> --sort-by=.metadata.creationTimestamp | tail -n 5
- Stream logs from the pod’s main container:
Copy code
kubectl logs -f <pod-name> -n <namespace>
With debug logs on, you should see the git pull step run:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool <your-pool>
Docs: - How to retrieve code from storage - Pull steps API - prefect-gitlab integration 3) Exec into the flow pod and load the block This proves the pod can fetch and decrypt the block (token is redacted in logs, so this is the definitive check):
Copy code
kubectl exec -it <pod-name> -n <namespace> -- python - <<'PY'
from prefect_gitlab import GitLabCredentials
c = GitLabCredentials.load("gitlab-lh-access-token")
print("Loaded block and token present:", bool(c.token.get_secret_value()))
PY
Expect: 
Loaded block and token present: True
4) Optional: test the clone step from inside the pod This confirms the credential works for Git operations in that container:
Copy code
kubectl exec -it <pod-name> -n <namespace> -- python - <<'PY'
from prefect.deployments.steps.pull import git_clone
from prefect_gitlab import GitLabCredentials
c = GitLabCredentials.load("gitlab-lh-access-token")
out = git_clone(
    repository="<https://gitlab.com/<org>/<repo>.git>",
    credentials=c,
    directories=["."]
)
print(out)
PY
5) Ensure the pod image has the integration available If you see import errors for `prefect_gitlab`: - Build it into your image:
Copy code
pip install "prefect[gitlab]"
# or
pip install prefect-gitlab
prefect block register -m prefect_gitlab
- Or add a 
requires
to your pull step so Prefect installs it before running the step:
Copy code
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://gitlab.com/<org>/<repo>.git>
      credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-lh-access-token }}"
      requires:
        - prefect-gitlab
Common K8s gotchas - The worker’s environment is not the same as the flow pod. Always test inside the flow pod. - If the pod cannot reach the Prefect API, block loading will fail. Make sure network/DNS and API auth are set in the pod. - If you’re using submodules, ensure the token has access to them and enable submodule pulling. If you can share your namespace and the image used by the Kubernetes job pods, I can suggest the exact 
kubectl
commands and the best place to bake in 
prefect-gitlab
.
e
i can run 
git clone  https://$(python -c 'from prefect.blocks.system import Secret; print(Secret.load("gitlab-username").get())'):$(python -c 'from prefect.blocks.system import Secret; print(Secret.load("gitlab-lh-access-token").get())')@gitlab.com/f/h.git
without prblem in the pod of the jobs @Marvin any idea ?
m
✅ thought for 197.0 seconds
Thanks — that’s a helpful datapoint. Since you can clone manually inside the job pod with 
https://<username>:<token>@gitlab.com/...
, the most common culprits with Prefect’s 
git_clone
on Kubernetes are: 1) The block type or token format doesn’t match what 
git_clone
expects - Your deployment shows:
Copy code
credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-lh-access-token }}"
This requires a GitLabCredentials block named 
gitlab-lh-access-token
, not a system Secret. If you only created system Secret blocks, the templating above won’t work as intended. - For GitLabCredentials: - Personal Access Token: set token to either 
oauth2:<PAT>
or just 
<PAT>
(Prefect will add 
oauth2:
). - Deploy Token: set token to 
<deploy-username>:<deploy-token>
(no 
oauth2:
). This matches your manual 
username:token
format. - Quick check (inside the flow job pod):
Copy code
python - <<'PY'
from prefect_gitlab import GitLabCredentials
c = GitLabCredentials.load("gitlab-lh-access-token")
print("Loaded:", bool(c.token.get_secret_value()))
print("Token value preview format contains colon?:", ":" in c.token.get_secret_value())
PY
If you’re using a deploy token, the last line should print True (because it’s username:token). 2) The 
prefect-gitlab
integration is not installed in the job image - If the job pod can’t import 
prefect_gitlab
, Prefect can’t use the GitLabCredentials formatter and will not embed credentials correctly. - Fix by either: - Baking it into the image:
Copy code
pip install "prefect[gitlab]"
    prefect block register -m prefect_gitlab
- Or add an auto-install to your pull step:
Copy code
pull:
      - prefect.deployments.steps.git_clone:
          repository: <https://gitlab.com/f/h.git>
          credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-lh-access-token }}"
          requires:
            - prefect-gitlab
3) Using the wrong auth parameter for a deploy token - If you set 
access_token:
instead of 
credentials:
for a deploy token, Prefect will default to 
oauth2:<token>@...
, which fails for deploy tokens that need 
<username>:<token>
. - For deploy tokens, use a GitLabCredentials block with the token set to 
<username>:<token>
. Alternatively, build your deployment in Python with a credentials dict:
Copy code
GitRepository(
    url="<https://gitlab.com/f/h.git>",
    credentials={"username": "<deploy-username>", "access_token": "<deploy-token>"},
  )
4) Clone worked but Prefect failed on entrypoint path or sparse checkout - Prefect clones to 
./<repo-name>
(or 
./<repo-name>-<branch>
). If your 
entrypoint
assumes a subdirectory, add a working directory step:
Copy code
pull:
    - prefect.deployments.steps.pull.set_working_directory:
        directory: repo/subdir
    - prefect.deployments.steps.git_clone:
        repository: <https://gitlab.com/f/h.git>
        credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-lh-access-token }}"
Or ensure the entrypoint path is correct relative to the clone root. - If you set 
directories:
for sparse checkout and they don’t match real paths, git won’t error but your checkout will be empty. Remove 
directories
to test. 5) You accidentally provided both 
access_token
and 
credentials
- 
git_clone
raises: “Please provide either an access token or credentials but not both.” What I’d do next in your pod - Verify the block type and that the integration is present:
Copy code
python - <<'PY'
import sys
try:
    import prefect_gitlab
    print("prefect_gitlab: OK")
except Exception as e:
    print("prefect_gitlab import failed:", e, file=sys.stderr)
from prefect.blocks.system import Secret
from prefect.blocks.core import Block
print("Secret(gitlab-username):", bool(Secret.load("gitlab-username").get()))
print("Secret(gitlab-lh-access-token):", bool(Secret.load("gitlab-lh-access-token").get()))
PY
- If you’re using a deploy token, ensure the GitLabCredentials block token equals 
<username>:<token>
.
If you can paste the exact error lines around the pull step from the job pod logs (e.g., auth failed vs. file not found), I can pinpoint it. In many cases on K8s it’s either the missing 
prefect-gitlab
package or using 
access_token
for a deploy token rather than a GitLabCredentials block. Useful refs: - Git-based storage: How to retrieve code from storage - Pull steps API: Pull steps API - GitLab integration: prefect-gitlab integration
e
@Marvin i used a credential dict
m
✅ thought for 98.8 seconds
e
Copy code
flow.from_source(
   source=GitRepository(
   url="<https://gitlab.com/gs/g/g.git>",
   credentials={
    "username": Secret.load("gitlab-username"),
    "access_token": Secret.load("gitlab-lh-access-token"),
    },
     branch="prefect",
   ),
    entrypoint="flows/b2r.py:b2r",
    ).deploy(...
m
Perfect, using a credentials dict is supported. For GitLab on Prefect 3.x, here’s exactly how Prefect formats that dict and what to check. What a credentials dict should look like - For a GitLab deploy token: - Use username + token:
Copy code
credentials={"username": "<deploy-username>", "access_token": "<deploy-token>"}
This becomes:
Copy code
https://<deploy-username>:<deploy-token>@gitlab.com/<group>/<repo>.git
- For a GitLab PAT (personal access token): - If you’re using a dict, Prefect will not automatically add the GitLab-specific oauth2: prefix unless you use the GitLabCredentials block. With a dict, you can either: - Use username “oauth2” and the PAT as the token:
Copy code
credentials={"username": "oauth2", "access_token": "<pat>"}
→ https://oauth2:&lt;pat&gt;@gitlab.com/... - Or if you are not providing a username, GitLab expects oauth2:<pat> embedded as the token; with a dict there isn’t a built-in way to add the oauth2: prefix unless you include it in the value yourself, e.g.:
Copy code
credentials={"access_token": "oauth2:<pat>"}
→ https://oauth2:&lt;pat&gt;@gitlab.com/... - If you want Prefect to add oauth2: for you automatically, switch to a GitLabCredentials block. How Prefect uses your dict under the hood - Keys recognized: 
username
, 
access_token
(preferred), 
token
, 
password
- Priority if multiple are present: 
access_token
> 
token
> 
password
- If 
username
is present, Prefect formats 
username:<token>
into the URL netloc - If 
username
is not present, it uses just 
<token>
in the URL netloc - Credentials are only embedded for HTTPS URLs, not SSH Why manual git clone works but Prefect clone fails - Your manual clone uses 
username:token@...
which matches a deploy token format. - If your dict didn’t include 
username
or used a PAT without the oauth2: format, GitLab auth would fail via Prefect. - Or, if you used both 
access_token
and a credentials block elsewhere, Prefect’s git_clone would error out. Concrete next steps to try in your deployment - If this is a deploy token, make sure your deployment uses:
Copy code
pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://gitlab.com/f/h.git>
        credentials:
          username: "<deploy-username>"
          access_token: "{{ prefect.blocks.secret.gitlab-lh-access-token }}"  # or the raw value
Or in Python:
Copy code
GitRepository(
    url="<https://gitlab.com/f/h.git>",
    credentials={"username": "<deploy-username>", "access_token": "<deploy-token>"},
  )
- If this is a PAT and you wish to keep using a dict: - Use username oauth2:
Copy code
credentials={"username": "oauth2", "access_token": "<pat>"}
- Or embed the oauth2: prefix:
Copy code
credentials={"access_token": "oauth2:<pat>"}
- Or switch to a GitLabCredentials block to get automatic formatting for both PATs and deploy tokens:
Copy code
pip install "prefect[gitlab]"
  prefect block register -m prefect_gitlab
  prefect block create gitlab-credentials
  # Set token to either:
  #   - "<deploy-username>:<deploy-token>"   (deploy token)
  #   - "<pat>" or "oauth2:<pat>"            (PAT; 'oauth2:' will be added if missing)
K8s-specific verification - Exec into a flow job pod and import check:
Copy code
python - <<'PY'
from prefect.runner.storage import GitRepository
# Try the exact dict you used:
creds = {"username": "<deploy-username>", "access_token": "<token>"}
repo = GitRepository(url="<https://gitlab.com/f/h.git>", credentials=creds)
print("URL with creds would be embedded as:", repo._repository_url_with_credentials)
PY
- If this prints a netloc that matches your working manual command, the auth shape is correct and any remaining issues are likely: - Wrong entrypoint path after clone - Sparse checkout 
directories
filtering too aggressively (can result in empty working dir without errors) - Missing submodule auth if you use submodules
If you can share your exact credentials dict structure (redact sensitive values) and whether it’s a PAT or a deploy token, I can confirm the precise formatting you need.
e
@Marvin
Copy code
flow.from_source(
   source=GitRepository(
   url="<https://gitlab.com/gs/g/g.git>",
   credentials={
    "username": Secret.load("gitlab-username"),
    "access_token": Secret.load("gitlab-lh-access-token"),
    },
     branch="prefect",
   ),
    entrypoint="flows/b2r.py:b2r",
m
✅ thought for 162.4 seconds
Thanks for the snippet — that dict looks fine. For GitLab, Prefect will embed it as 
https://<username>:<access_token>@gitlab.com/...
. Since you can git clone manually inside the pod with the same username/token, the remaining issues are usually one of: 1) Token type vs formatting mismatch - Deploy token: your dict is perfect: - 
{"username": "<deploy-username>", "access_token": "<deploy-token>"}
- PAT: GitLab expects oauth2:<pat> when using HTTPS - With a dict you must either set 
username: "oauth2"
and 
access_token: "<pat>"
, or set 
access_token: "oauth2:<pat>"
. - If you’re actually using a PAT but kept your real GitLab username in 
username
, auth will fail. Please confirm if this is a deploy token or a PAT. 2) Entrypoint path vs working directory after clone - In Prefect 3.x, after running pull steps, the engine will automatically chdir into the directory returned by the step output before importing the entrypoint. - 
git_clone
returns a relative directory (e.g., 
g
from your repo URL). Then Prefect changes CWD to that directory and imports the entrypoint. - Given your entrypoint is 
flows/b2r.py:b2r
, verify that in the cloned repo the file exists at 
./flows/b2r.py
. If your flow actually lives under a subfolder, either: - Update 
entrypoint
accordingly, or - Add a 
set_working_directory
pull step to descend further before import. 3) Sparse checkout or branch path - If you used 
directories=[...]
in your GitRepository, an incorrect directory will silently yield an empty checkout. Try removing 
directories
to test. - Confirm the branch 
prefect
exists on the remote (and your flow code is on that branch). The destination dir will be 
g-prefect
(branch slashes sanitized in directory name only), but Prefect still changes CWD to that path for import. 4) Submodules - If your repo uses private submodules, ensure you set 
include_submodules=True
and that the same credentials work for submodules. Otherwise, the clone may succeed but code pulls may be incomplete. Targeted checks inside the flow job pod - Show how Prefect embeds your dict:
Copy code
python - <<'PY'
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret
creds = {
  "username": Secret.load("gitlab-username").get(),
  "access_token": Secret.load("gitlab-lh-access-token").get(),
}
repo = GitRepository(url="<https://gitlab.com/gs/g/g.git>", credentials=creds, branch="prefect")
print("Resolved URL netloc:", repo._repository_url_with_credentials.split("://",1)[1].split("@",1)[0])
print("Destination path:", repo.destination)
PY
- Confirm your entrypoint exists relative to the repo root:
Copy code
python - <<'PY'
import os, glob
print("CWD:", os.getcwd())
print("Expect to change into:", "g-prefect")  # typical destination dir
print("Does file exist after chdir?:", os.path.exists("g-prefect/flows/b2r.py"))
PY
If it’s a PAT, try this change - Change credentials to:
Copy code
credentials={
  "username": "oauth2",
  "access_token": Secret.load("gitlab-lh-access-token").get(),
}
Or set:
Copy code
credentials={
  "access_token": f"oauth2:{Secret.load('gitlab-lh-access-token').get()}",
}
If you share the exact error message from the job pod logs (auth failed vs. module not found vs. file not found), I can pinpoint which of the above it is.
e
Copy code
Resolved URL netloc: oauth2:glpat-...
Destination path: /opt/prefect/g-prefect
@Marvin
m
✅ thought for 51.6 seconds
Thanks — that output is exactly what we want to see. - Resolved URL netloc: 
oauth2:glpat-...
→ confirms you’re using a GitLab PAT and Prefect is embedding it correctly for HTTPS. - Destination path: 
/opt/prefect/g-prefect
→ Prefect will clone into that directory, then chdir there before importing your entrypoint. Given credentials look good, the remaining failures are usually path/import issues after clone. Quick checks: 1) Verify the entrypoint path exists after clone - It should be relative to the repo root (g-prefect):
Copy code
kubectl exec -it <pod> -n <ns> -- ls -la /opt/prefect/g-prefect/flows/b2r.py
- If your flow lives elsewhere (e.g., src/flows/b2r.py), update the entrypoint or add a set_working_directory step. 2) Confirm the pull step runs and chdir happens - Inspect the deployment to ensure a git_clone pull step is present:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
You should see a 
prefect.deployments.steps.pull.git_clone
step. - Watch flow pod logs for: - “Running X deployment pull step(s)” - “Changing working directory to 'g-prefect'” - “Importing flow code from 'flows/b2r.py:b2r'” Run the worker with debug logs if needed:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool <your-pool>
3) Sanity-check clone inside the pod - List the repo root:
Copy code
kubectl exec -it <pod> -n <ns> -- sh -lc "ls -la /opt/prefect/g-prefect && git -C /opt/prefect/g-prefect status"
- If you use submodules, make sure they’re fetched and accessible; if not, set 
include_submodules=True
. 4) If this is a PAT, your dict is correct now Your dict should look like:
Copy code
credentials={
  "username": Secret.load("gitlab-username"),       # should be 'oauth2' for PAT
  "access_token": Secret.load("gitlab-lh-access-token"),
}
Given your resolved netloc shows 
oauth2:glpat-...
, that’s good. If 
gitlab-username
is not actually 
oauth2
, swap it to a Secret with value 
oauth2
for PAT usage. If you’re still seeing errors, can you paste the specific log lines from the job pod around: - the pull step execution - the “Importing flow code…” line - the traceback (ModuleNotFoundError, Missing file, etc.) Docs for reference: - Git-based storage: How to retrieve code from storage - Pull steps API: Pull steps API - GitLab integration (if you later switch to the block): prefect-gitlab
e
@Marvin
Copy code
18:04:03.934 | DEBUG   | Flow run 'spectral-jellyfish' - Running 1 deployment pull step(s)
  18:04:03.942 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f323d9c9a10> enqueuing item {'name': 'prefect.flow_runs', 'level': 20, 'message': ' > Running git_clone step...', 'timestamp': '2025-12-11T18:04:03.942303Z', 'flow_run_id': '406bfff3-d524-4350-a92d-ad4433502c5
  18:04:03.942 | INFO    | Flow run 'spectral-jellyfish' -  > Running git_clone step...
  18:04:03.943 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server:4200/api/>
  18:04:03.943 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=1.9999807011336088) in thread 'APILogWorkerThread'
  18:04:03.944 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
  18:04:03.944 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' COMPLETED, runtime=0.00> exited
  18:04:03.944 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Finished call get(<dropped>)
  18:04:03.945 | DEBUG   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f323d9c9a10> added item {'name': 'prefect.flow_runs', 'level': 10, 'message': 'Running 1 deployment pull step(s)', 'timestamp': '2025-12-11T18:04:03.934891Z', 'flow_run_id': '406bfff3-d524-4350-a92d-ad4433502c5
  18:04:03.945 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=1.998583922162652) in thread 'APILogWorkerThread'
  18:04:03.945 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
  18:04:03.945 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' COMPLETED, runtime=0.00> exited
  18:04:03.945 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Finished call get(<dropped>)
  18:04:03.945 | DEBUG   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f323d9c9a10> added item {'name': 'prefect.flow_runs', 'level': 20, 'message': ' > Running git_clone step...', 'timestamp': '2025-12-11T18:04:03.942303Z', 'flow_run_id': '406bfff3-d524-4350-a92d-ad4433502c5a', '
  18:04:03.945 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=1.9978082180023193) in thread 'APILogWorkerThread'
  18:04:03.946 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
  18:04:03.950 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server:4200/api/>
  18:04:04.002 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server:4200/api/>
  18:04:04.004 | DEBUG   | WorkerThread-0 | prefect._internal.concurrency - Running call git_clone(branch='prefect', repository='<https://gitlab.com/g/g/g.git>', credentials={'u...) in thread 'WorkerThread-0'
  18:04:04.004 | DEBUG   | WorkerThread-0 | prefect._internal.concurrency - <WatcherThreadCancelScope, name='git_clone' RUNNING, runtime=0.00> entered
  18:04:04.005 | DEBUG   | WorkerThread-1 | prefect._internal.concurrency - Running call coroutine_wrapper() in thread 'WorkerThread-1'
  18:04:04.005 | DEBUG   | WorkerThread-1 | prefect._internal.concurrency - <WatcherThreadCancelScope, name='coroutine_wrapper' RUNNING, runtime=0.00> entered
  18:04:04.005 | DEBUG   | WorkerThread-1 | prefect._internal.concurrency - <WatcherThreadCancelScope, name='coroutine_wrapper' COMPLETED, runtime=0.00> exited
  18:04:04.005 | DEBUG   | WorkerThread-1 | prefect._internal.concurrency - Executing coroutine for call coroutine_wrapper(<dropped>) in new loop
  18:04:04.005 | DEBUG   | WorkerThread-1 | prefect._internal.concurrency - <AsyncCancelScope, name='coroutine_wrapper' RUNNING, runtime=0.00> entered
  18:04:04.005 | DEBUG   | prefect.runner.storage.git-repository.g-prefect - Pulling contents from repository 'g-prefect' to '/opt/prefect/g-prefect'...
  18:04:04.006 | DEBUG   | prefect.runner.storage.git-repository.g-prefect - Cloning repository <https://gitlab.com/g/g/g.git>
  18:04:04.066 | WARNING | WorkerThread-1 | prefect._internal - Attempt 1 of function 'git_clone' failed with RuntimeError: Failed to clone repository '<https://gitlab.com/g/g/g.git>' with exit code 128.. Retrying in 0.81 seconds...
  18:04:04.548 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' COMPLETED, runtime=2.00> exited
  18:04:04.548 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Encountered exception in call get(<dropped>)
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
      result = self.fn(*self.args, **self.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/queue.py", line 179, in get
      raise Empty
  _queue.Empty
  18:04:04.550 | DEBUG   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f98b4aa61d0> processing batch of size 202
  18:04:04.575 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=1.9999585002660751) in thread 'APILogWorkerThread'
  18:04:04.575 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
m
Your message was too long, here's your message at the allowed limit: ```18 04 03 934 | DEBUG | Flow run 'spectral-jellyfish' - Running 1 deployment pull step(s) 18 04 03 942 | DEBUG | RunSyncEventLoopThread | prefect _internal concurrency - Service <prefect logging handlers APILogWorker object at 0x7f323d9c9a10> enqueuing item {'name' 'prefect flow_runs' 'level' 20 'message' ' > Running git_clone step ' 'timestamp' '2025-12-11T18 04 03 942303Z' 'flow_run_id' '406bfff3-d524-4350-a92d-ad4433502c5 18 04 03 942 | INFO | Flow run 'spectral-jellyfish' - > Running git_clone step 18 04 03 943 | DEBUG | prefect client - Connecting to API at <http //prefect-server 4200/api/> 18 04 03 943 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Running call get(timeout=1 9999807011336088) in thread 'APILogWorkerThread' 18 04 03 944 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get' RUNNING runtime=0 00> entered 18 04 03 944 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get' COMPLETED runtime=0 00> exited 18 04 03 944 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Finished call get(<dropped>) 18 04 03 945 | DEBUG | GlobalEventLoopThread | prefect _internal concurrency - Service <prefect logging handlers APILogWorker object at 0x7f323d9c9a10> added item {'name' 'prefect flow_runs' 'level' 10 'message' 'Running 1 deployment pull step(s)' 'timestamp' '2025-12-11T18 04 03 934891Z' 'flow_run_id' '406bfff3-d524-4350-a92d-ad4433502c5 18 04 03 945 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Running call get(timeout=1 998583922162652) in thread 'APILogWorkerThread' 18 04 03 945 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get' RUNNING runtime=0 00> entered 18 04 03 945 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get' COMPLETED runtime=0 00> exited 18 04 03 945 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Finished call get(<dropped>) 18 04 03 945 | DEBUG | GlobalEventLoopThread | prefect _internal concurrency - Service <prefect logging handlers APILogWorker object at 0x7f323d9c9a10> added item {'name' 'prefect flow_runs' 'level' 20 'message' ' > Running git_clone step ' 'timestamp' '2025-12-11T18 04 03 942303Z' 'flow_run_id' '406bfff3-d524-4350-a92d-ad4433502c5a' ' 18 04 03 945 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Running call get(timeout=1 9978082180023193) in thread 'APILogWorkerThread' 18 04 03 946 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get' RUNNING runtime=0 00> entered 18 04 03 950 | DEBUG | prefect client - Connecting to API at <http //prefect-server 4200/api/> 18 04 04 002 | DEBUG | prefect client - Connecting to API at <http //prefect-server 4200/api/> 18 04 04 004 | DEBUG | WorkerThread-0 | prefect _internal concurrency - Running call git_clone(branch='prefect' repository='<https //gitlab com/g/g/g git>' credentials={'u ) in thread 'WorkerThread-0' 18 04 04 004 | DEBUG | WorkerThread-0 | prefect _internal concurrency - <WatcherThreadCancelScope name='git_clone' RUNNING runtime=0 00> entered 18 04 04 005 | DEBUG | WorkerThread-1 | prefect _internal concurrency - Running call coroutine_wrapper() in thread 'WorkerThread-1' 18 04 04 005 | DEBUG | WorkerThread-1 | prefect _internal concurrency - <WatcherThreadCancelScope name='coroutine_wrapper' RUNNING runtime=0 00> entered 18 04 04 005 | DEBUG | WorkerThread-1 | prefect _internal concurrency - <WatcherThreadCancelScope name='coroutine_wrapper' COMPLETED runtime=0 00> exited 18 04 04 005
e
@Marvin better ?
Copy code
18:04:03.934 | DEBUG   | Flow run 'spectral-jellyfish' - Running 1 deployment pull step(s)
  18:04:03.942 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f323d9c9a10> enqueuing item {'name': 'prefect.flow_runs', 'level': 20, 'message': ' > Running git_clone step...', 'timestamp': '2025-12-11T18:04:03.942303Z', 'flow_run_id': '406bfff3-d524-4350-a92d-ad4433502c5
  18:04:03.942 | INFO    | Flow run 'spectral-jellyfish' -  > Running git_clone step...
  18:04:03.943 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server:4200/api/>
  18:04:03.945 | DEBUG   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f323d9c9a10> added item {'name': 'prefect.flow_runs', 'level': 10, 'message': 'Running 1 deployment pull step(s)', 'timestamp': '2025-12-11T18:04:03.934891Z', 'flow_run_id': '406bfff3-d524-4350-a92d-ad4433502c5
  18:04:03.945 | DEBUG   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f323d9c9a10> added item {'name': 'prefect.flow_runs', 'level': 20, 'message': ' > Running git_clone step...', 'timestamp': '2025-12-11T18:04:03.942303Z', 'flow_run_id': '406bfff3-d524-4350-a92d-ad4433502c5a', '
  18:04:03.945 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=1.9978082180023193) in thread 'APILogWorkerThread'
  18:04:03.946 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
  18:04:03.950 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server:4200/api/>
  18:04:04.002 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server:4200/api/>
  18:04:04.004 | DEBUG   | WorkerThread-0 | prefect._internal.concurrency - Running call git_clone(branch='prefect', repository='<https://gitlab.com/g/g/g.git>', credentials={'u...) in thread 'WorkerThread-0'
  18:04:04.004 | DEBUG   | WorkerThread-0 | prefect._internal.concurrency - <WatcherThreadCancelScope, name='git_clone' RUNNING, runtime=0.00> entered
  18:04:04.005 | DEBUG   | prefect.runner.storage.git-repository.g-prefect - Pulling contents from repository 'g-prefect' to '/opt/prefect/g-prefect'...
  18:04:04.006 | DEBUG   | prefect.runner.storage.git-repository.g-prefect - Cloning repository <https://gitlab.com/g/g/g.git>
  18:04:04.066 | WARNING | WorkerThread-1 | prefect._internal - Attempt 1 of function 'git_clone' failed with RuntimeError: Failed to clone repository '<https://gitlab.com/g/g/g.git>' with exit code 128.. Retrying in 0.81 seconds...
  18:04:04.548 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' COMPLETED, runtime=2.00> exited
  18:04:04.548 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Encountered exception in call get(<dropped>)
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
      result = self.fn(*self.args, **self.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/queue.py", line 179, in get
      raise Empty
  _queue.Empty
  18:04:04.550 | DEBUG   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at 0x7f98b4aa61d0> processing batch of size 202
  18:04:04.575 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=1.9999585002660751) in thread 'APILogWorkerThread'
  18:04:04.575 | DEBUG   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
m
Your message was too long, here's your message at the allowed limit: better ? ```18 04 03 934 | DEBUG | Flow run 'spectral-jellyfish' - Running 1 deployment pull step(s) 18 04 03 942 | DEBUG | RunSyncEventLoopThread | prefect _internal concurrency - Service <prefect logging handlers APILogWorker object at 0x7f323d9c9a10> enqueuing item {'name' 'prefect flow_runs' 'level' 20 'message' ' > Running git_clone step ' 'timestamp' '2025-12-11T18 04 03 942303Z' 'flow_run_id' '406bfff3-d524-4350-a92d-ad4433502c5 18 04 03 942 | INFO | Flow run 'spectral-jellyfish' - > Running git_clone step 18 04 03 943 | DEBUG | prefect client - Connecting to API at <http //prefect-server 4200/api/> 18 04 03 945 | DEBUG | GlobalEventLoopThread | prefect _internal concurrency - Service <prefect logging handlers APILogWorker object at 0x7f323d9c9a10> added item {'name' 'prefect flow_runs' 'level' 10 'message' 'Running 1 deployment pull step(s)' 'timestamp' '2025-12-11T18 04 03 934891Z' 'flow_run_id' '406bfff3-d524-4350-a92d-ad4433502c5 18 04 03 945 | DEBUG | GlobalEventLoopThread | prefect _internal concurrency - Service <prefect logging handlers APILogWorker object at 0x7f323d9c9a10> added item {'name' 'prefect flow_runs' 'level' 20 'message' ' > Running git_clone step ' 'timestamp' '2025-12-11T18 04 03 942303Z' 'flow_run_id' '406bfff3-d524-4350-a92d-ad4433502c5a' ' 18 04 03 945 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Running call get(timeout=1 9978082180023193) in thread 'APILogWorkerThread' 18 04 03 946 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get' RUNNING runtime=0 00> entered 18 04 03 950 | DEBUG | prefect client - Connecting to API at <http //prefect-server 4200/api/> 18 04 04 002 | DEBUG | prefect client - Connecting to API at <http //prefect-server 4200/api/> 18 04 04 004 | DEBUG | WorkerThread-0 | prefect _internal concurrency - Running call git_clone(branch='prefect' repository='<https //gitlab com/g/g/g git>' credentials={'u ) in thread 'WorkerThread-0' 18 04 04 004 | DEBUG | WorkerThread-0 | prefect _internal concurrency - <WatcherThreadCancelScope name='git_clone' RUNNING runtime=0 00> entered 18 04 04 005 | DEBUG | prefect runner storage git-repository g-prefect - Pulling contents from repository 'g-prefect' to '/opt/prefect/g-prefect' 18 04 04 006 | DEBUG | prefect runner storage git-repository g-prefect - Cloning repository <https //gitlab com/g/g/g git> 18 04 04 066 | WARNING | WorkerThread-1 | prefect _internal - Attempt 1 of function 'git_clone' failed with RuntimeError Failed to clone repository '<https //gitlab com/g/g/g git>' with exit code 128 Retrying in 0 81 seconds 18 04 04 548 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get' COMPLETED runtime=2 00> exited 18 04 04 548 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Encountered exception in call get(<dropped>) Traceback (most recent call last) File /usr/local/lib/python3 11/site-packages/prefect/_internal/concurrency/calls py line 399 in _run_sync result = self fn(*self args **self kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 11/queue py line 179 in get raise Empty _queue Empty 18 04 04 550 | DEBUG | GlobalEventLoopThread | prefect _internal concurrency - Service <prefect logging handlers APILogWorker object at 0x7f98b4aa61d0> processing batch of size 202 18 04 04 575 | DEBUG | APILogWorkerThread | prefect _internal concurrency - Running call get(timeout=1 9999585002660751) in thread 'APILogWorkerThread' 18 04 04 575 | DEBUG | APILogWorkerThread | prefect _internal concurrency - <WatcherThreadCancelScope name='get'
e
@Marvin
Copy code
| D   | Flow run 'spectral-jellyfish' - Running 1 deployment pull step(s)
 | D   | RunSyncEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at ...> enqueuing item {'name': 'prefect.flow_runs', 'level': 20, 'message': ' > Running git_clone step...', 'timestamp': '2..., 'flow_run_id': '.
 | I    | Flow run 'spectral-jellyfish' -  > Running git_clone step...
 | D   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at ...> added item {'name': 'prefect.flow_runs', 'level': 10, 'message': 'Running 1 deployment pull step(s)', 'timestamp': '...', 'flow_run_id': '.| D   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at ...> added item {'name': 'prefect.flow_runs', 'level': 20, 'message': ' > Running git_clone step...', 'timestamp': '..., 'flow_run_id': '... '
| D   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=1.9978082180023193) in thread 'APILogWorkerThread'
| D   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
| D   | WorkerThread-0 | prefect._internal.concurrency - Running call git_clone(branch='prefect', repository='<https://gitlab.com/g/g/g.git>', credentials={'u...) in thread 'WorkerThread-0'
 D   | WorkerThread-0 | prefect._internal.concurrency - <WatcherThreadCancelScope, name='git_clone' RUNNING, runtime=0.00> entered
 | D   | prefect.runner.storage.git-repository.g-prefect - Pulling contents from repository 'g-prefect' to '/opt/prefect/g-prefect'...
| D   | prefect.runner.storage.git-repository.g-prefect - Cloning repository <https://gitlab.com/g/g/g.git>
| W | WorkerThread-1 | prefect._internal - Attempt 1 of function 'git_clone' failed with RuntimeError: Failed to clone repository '<https://gitlab.com/g/g/g.git>' with exit code 128.. Retrying in 0.81 seconds...
| D   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' COMPLETED, runtime=2.00> exited
 | D   | APILogWorkerThread | prefect._internal.concurrency - Encountered exception in call get(<dropped>)
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
      result = self.fn(*self.args, **self.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/queue.py", line 179, in get
      raise Empty
  _queue.Empty
 | D   | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.logging.handlers.APILogWorker object at ...> processing batch of size 202
 | D   | APILogWorkerThread | prefect._internal.concurrency - Running call get(timeout=...) in thread 'APILogWorkerThread'
| D   | APILogWorkerThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='get' RUNNING, runtime=0.00> entered
m
✅ thought for 147.1 seconds
Thanks for the logs. Prefect is definitely executing the git_clone pull step, but Git exits with 128 during clone. Since your resolved netloc is 
oauth2:glpat-...
and destination is 
/opt/prefect/g-prefect
, creds are being embedded as a GitLab PAT. That narrows this down to one of a few common causes. Most likely causes for exit 128 here - PAT scopes or access: The PAT user must have access to the repo and the token must include the read_repository scope. A 401/403/404 from GitLab will surface as exit 128 in git. - Token content (newline/whitespace): A trailing newline in the Secret value will break auth. - URL formatting: Ensure there are no angle brackets or stray whitespace in the URL string (e.g., not "<https://...>"). - Corporate proxy/SSL: Less likely since you can clone in the pod, but worth checking if the PAT path specifically is blocked. - Self‑hosted GitLab with custom CA: The container may not have the CA; test to confirm. Action plan to pinpoint the error 1) Get the exact HTTP error from git in the same flow pod - Add git verbose env to your deployment/job so clone logs show HTTP status: - Add env to job variables or work pool default env: - GIT_TRACE=1 - GIT_CURL_VERBOSE=1 - Re-run the flow and capture the lines around the git clone failure. You’ll typically see something like “HTTP/2 401 Unauthorized” or “The project you were looking for could not be found”. 2) Manually test the PAT auth format in that pod - You verified deploy-token format works. Now test the PAT oauth2 format explicitly:
Copy code
kubectl exec -it <pod> -n <ns> -- sh -lc '
GIT_TRACE=1 GIT_CURL_VERBOSE=1 \
git ls-remote <https://oauth2:$(python> - <<PY
from prefect.blocks.system import Secret
print(Secret.load("gitlab-lh-access-token").get())
PY
)@gitlab.com/g/g/g.git
'
- If this fails with 401/403/404, it’s a PAT scope/access issue. Grant read_repository and ensure the PAT user has project access (and SSO if enforced). 3) Check for trailing newline/whitespace in the token
Copy code
kubectl exec -it <pod> -n <ns> -- python - <<'PY'
from prefect.blocks.system import Secret
t = Secret.load("gitlab-lh-access-token").get()
print("endswith newline:", t.endswith("\n"))
print("repr head:", repr(t[:10]), "repr tail:", repr(t[-10:]))
PY
If there’s a newline, save a new Secret without it. 4) Double-check the repo URL in your code - Make sure it is exactly:
Copy code
url="<https://gitlab.com/g/g/g.git>"
- No angle brackets or trailing spaces. In Slack you showed url="<https://...>", which in actual code would break parsing and could lead to failures. 5) If you prefer deploy token auth (since it worked) - Switch back to deploy token in the dict:
Copy code
credentials={"username": "<deploy-username>", "access_token": "<deploy-token>"}
- Or use a GitLabCredentials block with token set to "<deploy-username>:<deploy-token>" and reference it in your pull step. If you share the HTTP status/error from step 1 or the result of step 2 (manual PAT test inside the pod), I can give a precise fix. In many cases it’s either missing read_repository on the PAT or a newline in the Secret. References: - Git-based storage: How to retrieve code from storage - Pull steps API: Pull steps API - GitLab integration: prefect-gitlab
e
@Marvin
Copy code
git clone  https://$(python -c 'from prefect.blocks.system import Secret; print(Secret.load("gitlab-username").get())'):$(python -c 'from prefect.blocks.system import Secret; print(Secret.load("gitlab-lh-access-token").get())')@gitlab.com/g/g.git
this clone the repo without problem inside the job created by the kubernetes worker.
m
✅ thought for 35.8 seconds
I think we found it. Your manual command clones from:
Copy code
<https://gitlab.com/g/g.git>
but Prefect’s logs show it’s trying to clone:
Copy code
<https://gitlab.com/g/g/g.git>
That extra path segment would cause git to return exit code 128 (“repository not found”/auth error), even though your credentials are correct. What to do - Update the deployment to use the exact URL that works manually:
Copy code
flow.from_source(
  source=GitRepository(
    url="<https://gitlab.com/g/g.git>",   # <- match your manual command
    credentials={
      "username": Secret.load("gitlab-username"),
      "access_token": Secret.load("gitlab-lh-access-token"),
    },
    branch="prefect",
  ),
  entrypoint="flows/b2r.py:b2r",
).deploy(...)
- Double-check you didn’t paste Slack’s angle-bracket formatting into code; use a plain string, not “<https://…>”. - Verify the deployment pull step points at the right URL:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
Look for:
Copy code
prefect.deployments.steps.pull.git_clone:
  repository: <https://gitlab.com/g/g.git>
Everything else (your credentials dict, the resolved netloc 
oauth2:glpat-...
, destination 
/opt/prefect/g-prefect
) looks good. If you still see exit 128 after correcting the URL, grab the HTTP status with verbose git logs in the pod:
Copy code
kubectl exec -it <pod> -n <ns> -- sh -lc '
GIT_TRACE=1 GIT_CURL_VERBOSE=1 git ls-remote <https://oauth2:$(python> - <<PY
from prefect.blocks.system import Secret
print(Secret.load("gitlab-lh-access-token").get())
PY
)@gitlab.com/g/g.git
'
But I expect fixing the repository URL in the deployment will resolve this. Docs for reference: - Git-based storage: How to retrieve code from storage - Pull steps API: Pull steps API
e
it look like the issues is that /opt/prefect (default working directory) is read-only, do you know a way to change the destination for the source ? @Marvin using deploy(...) pyhton
m
✅ thought for 324.4 seconds
You’re right that many Prefect Docker images default to using /opt/prefect as the working directory. If that path is read-only in your runtime, you’ll want to point Prefect’s “source checkout/execution” to a writable location. The exact fix depends on your work pool type. Can you confirm which worker you’re using (Process, Docker, or Kubernetes)? In the meantime, here are the options: - Process work pool (local execution) - Set a writable working directory via job variables when you deploy:
Copy code
from prefect import flow

  @flow
  def my_flow():
      ...

  my_flow.from_source(
      source="<https://github.com/org/repo.git>",
      entrypoint="flows.py:my_flow",
  ).deploy(
      name="my-deployment",
      work_pool_name="my-process-pool",
      job_variables={
          "working_dir": "/tmp/prefect"  # any writable path
      },
  )
This ensures the source is pulled and executed under /tmp/prefect instead of /opt/prefect. - Docker work pool - Set the container working directory to a writable path in the work pool’s base job template (or job variables if your pool exposes it). For example, in the base job template, set:
Copy code
{
    "command": "{{ command }}",
    "image": "{{ image }}",
    "working_dir": "/data/app",        // ensure this exists and is writable
    "volumes": ["/data:/data"]         // mount a writable volume if needed
  }
This makes the runtime working directory /data/app, so any source pulled by the deployment will land there instead of /opt/prefect. - Kubernetes work pool - Set the container’s workingDir in the base job template and mount a writable volume:
Copy code
spec:
    template:
      spec:
        containers:
          - name: prefect-job
            image: "{{ image }}"
            args: "{{ command }}"
            workingDir: /data/app
            volumeMounts:
              - name: data
                mountPath: /data
        volumes:
          - name: data
            persistentVolumeClaim:
              claimName: my-writable-pvc
Additional tips - If you’re using flow.from_source(...), Prefect will generate “pull steps” to fetch your code before execution. Those steps clone/extract into the current working directory of the flow run. So making sure the container’s working directory is writable solves the issue globally. - You can control the subfolder name (not the base directory) by using a Git source with a custom name:
Copy code
from prefect import flow
  from prefect.runner.storage import GitRepository

  repo = GitRepository(
      url="<https://github.com/org/repo.git>",
      name="my-repo-folder"  # controls the subdirectory name
  )

  flow.from_source(
      source=repo,
      entrypoint="flows.py:my_flow",
  ).deploy(
      name="my-deployment",
      work_pool_name="your-pool",
  )
- If you want to go further and never pull at runtime, you can bake the code into the image you deploy with and avoid runtime cloning entirely. If you can share your work pool type (process/docker/k8s) and how the worker is launched, I can give you the exact snippet to apply.
e
i use k8s but i do want to pull git at runtime
@Marvin adding the workingDir does change it but the clone is still failling
m
✅ thought for 190.6 seconds
Got it — if changing the container’s workingDir helped but the clone still fails, you’re likely hitting where Prefect actually clones when using flow.from_source(...).deploy(...). Important detail: - With from_source(...), Prefect clones into a temporary directory derived from tempfile.gettempdir() — i.e., $TMPDIR/runner_storage/<uuid>/... — not necessarily the container workingDir. - If $TMPDIR points to a read-only location (or a path your UID can’t write to), git clone will fail with exit code 128 (often “could not create work tree dir …: Permission denied”). What to try 1) Set TMPDIR to a writable mount - Kubernetes work pool: add a writable volume and set TMPDIR to that mount in the base job template. - Example:
Copy code
spec:
    template:
      spec:
        containers:
          - name: prefect-job
            image: "{{ image }}"
            args: "{{ command }}"
            env:
              - name: TMPDIR
                value: /scratch
            volumeMounts:
              - name: scratch
                mountPath: /scratch
            # Optional (OpenShift/non-root): runAsUser: 1000
        volumes:
          - name: scratch
            emptyDir: {}
- Docker work pool: mount a writable dir and set TMPDIR in env. - Example base job template fragment:
Copy code
{
  "image": "{{ image }}",
  "env": {
    "TMPDIR": "/scratch"
  },
  "volumes": [
    "/host/scratch:/scratch"
  ]
}
2) If you are using explicit pull steps (prefect.deployments.steps.git_clone) - Ensure you set the working directory before cloning so the repo is cloned into a writable path:
Copy code
pull:
    - prefect.deployments.steps.set_working_directory:
        directory: /scratch/flows
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/org/repo.git>
- If /scratch/flows may not exist, add a prep step:
Copy code
- prefect.deployments.steps.utility.run_shell_script:
      script: "mkdir -p /scratch/flows && chmod -R u+rwX /scratch/flows"
3) Double-check the failure mode - If it’s permissions, you’ll typically see exit code 128 with a “Permission denied” message from git. - If it’s auth (token/SSH), use a credentials block or access token in the git_clone step. Quick sanity checklist - Mount a writable path and set TMPDIR to it. - If using pull steps, run set_working_directory before git_clone, and ensure the directory exists and is writable. - If running non-root, confirm the UID can write to the mounted path (securityContext/runAsUser on K8s). If you can share: - The exact error message from the clone - Your work pool type (Docker/K8s/Process) - How you set workingDir and any env/volume config …I can give you a precise patch for your deployment/work pool config.
e
@Marvin TMPDIR is set and working but error still appear Failed to clone repository 'https://gitlab.com/g/g.git' with exit code 128.
m
✅ thought for 151.2 seconds
Thanks for the extra detail. Since TMPDIR is writable but the clone still fails with exit code 128, the most common remaining cause is authentication (private repo or GitLab token formatting). Prefect redacts git stderr for safety, so you’ll see only the generic error. Quick fixes to try - If your repo is private, pass credentials explicitly. With flow.from_source(...), use a GitRepository with credentials:
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

repo = GitRepository(
    url="<https://gitlab.com/g/g.git>",
    # For GitLab PATs, either of these work:
    credentials={"access_token": "glpat-<your-token>"}  # simplest
    # or, explicitly use the oauth2 username convention for GitLab:
    # credentials={"username": "oauth2", "access_token": "glpat-<your-token>"}
)

flow.from_source(
    source=repo,
    entrypoint="path/to/flow.py:my_flow",
).deploy(
    name="my-deployment",
    work_pool_name="<your-pool>",
)
- If you’re using pull steps in prefect.yaml, set credentials there:
Copy code
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /scratch/flows
  - prefect.deployments.steps.git_clone:
      repository: <https://gitlab.com/g/g.git>
      # Option A: use a Secret block
      credentials:
        access_token: "{{ prefect.blocks.secret.gitlab_token }}"
      # Option B: inline (for testing only)
      # credentials:
      #   username: oauth2
      #   access_token: "glpat-<your-token>"
Other things to rule out - Missing git credentials for submodules: - If your repo uses submodules, add include_submodules=True to the GitRepository or git_clone step and ensure those submodules are accessible with the same credentials. - TLS/Proxy issues (often also show up as exit 128 but with hidden stderr): - If you’re behind a corporate proxy/SSL interception, set env vars on the worker: - https_proxy / HTTP_PROXY / NO_PROXY - GIT_SSL_CAINFO pointing to your corporate CA bundle - As a last resort for testing only: GIT_SSL_NO_VERIFY=true - In a Kubernetes work pool, add them to the container env in the base job template. In Docker, add them to env in the base job template. - Sanity checks inside the running worker container/pod: - Exec in and run:
Copy code
echo $TMPDIR && ls -ld $TMPDIR && whoami
    git --version
    git clone <https://gitlab.com/g/g.git> /tmp/test-clone
If this fails, the stderr will reveal whether it’s auth, SSL, or network. If you can share: - Is the repo private? - Your deploy code or prefect.yaml (redact tokens) - Whether you’re using from_source with just a URL string vs a GitRepository with credentials - Worker type (Docker/K8s/Process) …I can tailor the exact snippet for your setup.
e
git clone with credential (pull from prefect block) work on the container, @Marvin i'm using k8s worker from the helm chart ,
m
✅ thought for 44.8 seconds
Thanks — that helps. Since you’re on a K8s worker (Helm) and 
git clone
works inside the container, the remaining gap is usually that the deployment created with flow.from_source(...) isn’t getting credentials. With a plain URL string, Prefect won’t inject auth for private repos. Fix: pass credentials via GitRepository in from_source - Use a GitRepository with credentials (GitLab PAT). For GitLab, “oauth2” is the username convention for PATs.
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

@flow
def my_flow():
    ...

repo = GitRepository(
    url="<https://gitlab.com/g/g.git>",
    # Option 1: inline for a quick test
    credentials={"username": "oauth2", "access_token": "glpat-<your-token>"}
    # Option 2: use a Secret block (recommended)
    # credentials={"access_token": "{{ prefect.blocks.secret.gitlab_pat }}"}
)

flow.from_source(
    source=repo,
    entrypoint="path/to/flow.py:my_flow",
).deploy(
    name="my-deployment",
    work_pool_name="<your-k8s-pool>",
)
If you prefer prefect.yaml pull steps, ensure you set a writable working directory before cloning (even though TMPDIR is set):
Copy code
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /scratch/flows
  - prefect.deployments.steps.git_clone:
      repository: <https://gitlab.com/g/g.git>
      credentials:
        username: oauth2
        access_token: "{{ prefect.blocks.secret.gitlab_pat }}"
Extra K8s worker notes - You already set TMPDIR; keep it in the K8s work pool base job template and mount a writable volume:
Copy code
spec:
  template:
    spec:
      containers:
        - name: prefect-job
          env:
            - name: TMPDIR
              value: /scratch
          volumeMounts:
            - name: scratch
              mountPath: /scratch
      volumes:
        - name: scratch
          emptyDir: {}
- If you use submodules, set include_submodules=True on GitRepository or the git_clone step. If this still fails, please share: - How you’re creating the deployment now (your deploy code or prefect.yaml) - Whether the repo is private and if it has submodules - A snippet of your K8s work pool base job template (env/volumes) - DEBUG logs (set 
PREFECT_LOGGING_LEVEL=DEBUG
in the job env). This won’t print secrets but can confirm where Prefect is attempting to clone.
e
i'm using the python sdk deploy() No submodule, credential works without issues i'm using python -c 'from prefect.blocks.system import Secret; from prefect.runner.storage import GitRepository; g =GitRepository( url="https://gitlab.com/g/g.git", credentials={ "username": Secret.load("gitlab-username"), "access_token": Secret.load("gitlab-lh-access-token"), }, branch="prefect",) ; import asyncio; asyncio.run(g.pull_code())'
@Marvin ^
m
✅ thought for 36.8 seconds
Thanks for the bump. Since you’re on a K8s worker and can 
git clone
with credentials inside the container, the remaining issue is almost certainly that the deployment created with 
flow.from_source(...)
is not getting credentials. A plain URL string won’t include auth, so Prefect’s internal clone will fail with exit code 128. Two solid ways to fix it: A) Pass credentials via GitRepository in from_source (recommended) - Explicitly provide GitLab PAT credentials to the storage object you pass to 
from_source
. For GitLab, using username 
oauth2
with a PAT is reliable.
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

@flow
def my_flow():
    ...

# Option 1: inline for a quick test (not for prod)
repo = GitRepository(
    url="<https://gitlab.com/g/g.git>",
    credentials={"username": "oauth2", "access_token": "glpat-<your-token>"},
    # include_submodules=True,  # if your repo needs it
)

# Option 2: use a Secret block (preferred)
# token_block = Secret.load("gitlab_pat")  # pre-created Prefect Secret block
# repo = GitRepository(
#     url="<https://gitlab.com/g/g.git>",
#     credentials={"username": "oauth2", "access_token": token_block},
# )

my_flow.from_source(
    source=repo,
    entrypoint="path/to/flow.py:my_flow",
).deploy(
    name="my-deployment",
    work_pool_name="<your-k8s-pool>",
)
B) Use pull steps in prefect.yaml and pass credentials there - Make sure you set a writable working directory first (even though 
TMPDIR
is set), then supply credentials to 
git_clone
.
Copy code
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /scratch/flows
  - prefect.deployments.steps.git_clone:
      repository: <https://gitlab.com/g/g.git>
      credentials:
        username: oauth2
        access_token: "{{ prefect.blocks.secret.gitlab_pat }}"
      # include_submodules: true  # if needed
Why this matters - 
flow.from_source("https://...")
by itself constructs a GitRepository without credentials, so private GitLab repos will fail at clone time (128). - Providing credentials via GitRepository (in Python) or via 
git_clone.credentials
(in YAML pull steps) fixes it. Optional sanity checks - If your repo has submodules, add 
include_submodules=True
(Python) or 
include_submodules: true
(YAML). - If you’re behind a proxy/SSL intercept, ensure the worker has the proxy envs and CA trust configured, but since 
git clone
works in the container, auth is the likely gap. - Turn on debug logs to confirm where Prefect is attempting to clone: - Set 
PREFECT_LOGGING_LEVEL=DEBUG
in the K8s work pool base job template env for the job container. If you can share your current deploy snippet (Python) or 
prefect.yaml
(redact secrets), I’ll tailor the exact patch.
3 Views
Open in Slack
PreviousNext
Powered by