<@ULVA73B9P>, I am doing Gitlab migration with my ...
# ask-marvin
k
@Marvin, I am doing Gitlab migration with my current private repo with Prefect Cloud. However I am struggling with the pull step using the PAT (Personal or Project Access Token). As I understand based on the code in
prefect/runner/storage.py
I have to setup the token as
oauth2:<token>@gitlab.corp-name/group/project.git
However this does not work and i keep getting error 128. Any idea how to resolve this one? Or maybe some workaround that does not depends on pulling perhaps ?
m
thought for 565.6 seconds
Thanks for the details — a few things to clarify and some concrete ways to fix this. First, that
oauth2:<token>@gitlab.corp-name/group/project.git
form is only correct when it’s part of a full HTTPS URL (and the token must be URL-safe). The correct format for GitLab over HTTPS is:
Copy code
<https://oauth2:<YOUR_TOKEN>@gitlab.corp-name/group/project.git>
Common pitfalls that produce git error 128: - Missing
https://
scheme - Token not URL-encoded (if it contains special characters) - Insufficient scopes (GitLab PAT must include at least `read_repository`;
api
also works) - Wrong host/URL or network/SSL issues on the worker - Corporate instances sometimes disable PATs for Git over HTTPS; use a deploy token or SSH in that case That said, in Prefect 3.x you do not need to embed the token in the URL. Prefer one of these supported approaches. Option A (recommended): pass the token via the git_clone pull step - prefect.yaml
Copy code
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://gitlab.corp-name/group/project.git>
      branch: main
      access_token: "{{ env.GITLAB_PAT }}"
Then provide
GITLAB_PAT
in your worker environment. This keeps the URL clean and avoids encoding problems. Docs: git_clone step Option B: use a GitLabCredentials block - Create and save a credentials block:
Copy code
from prefect_gitlab import GitLabCredentials
GitLabCredentials(access_token="glpat-...").save("my-gitlab-creds")
- Use that credentials block in your deployment/pull configuration (either in prefect.yaml via a block reference or programmatically). Docs: - Prefect GitLab integration - Runner storage &amp; GitRepository API Option C: programmatically with Flow.from_source (no URL-embedded token)
Copy code
from prefect.flows import flow
from prefect.runner.storage import GitRepository
from prefect_gitlab import GitLabCredentials

creds = GitLabCredentials(access_token="glpat-...")
repo = GitRepository(
    url="<https://gitlab.corp-name/group/project.git>",
    branch="main",
    credentials=creds,  # no URL token needed
)

f = flow.from_source(repo, entrypoint="path/to/flow.py:my_flow")
# e.g. f.deploy(name="my-deploy", work_pool_name="my-pool")
Workarounds that avoid git pulling entirely - Bake your code into an image used by the worker (Docker/K8s) and point the deployment at a local path inside the image:
Copy code
from pathlib import Path
from prefect.flows import flow

f = flow.from_source(Path("/opt/prefect/flows"), entrypoint="path/to/flow.py:my_flow")
Now the worker runs with code already present; no git pull required. - Use remote object storage instead of git (e.g., S3/GS/Azure) with the
pull_from_remote_storage
step to download your project bundle at runtime. Docs: Store your flow code Quick checklist to resolve git error 128 - Confirm the worker can resolve and reach
gitlab.corp-name
(DNS/SSL/proxy) - Ensure the PAT has
read_repository
scope (or
api
) - If embedding in URL, use the full scheme
<https://oauth2:<token>@>...
and URL-encode special chars - Prefer passing the token via
access_token
or a GitLabCredentials block instead of embedding - If PATs are restricted by your org, try a GitLab Deploy Token (
username:deploy_token
) or SSH - Turn on debug logs to see the actual git command: - Set
PREFECT_LOGGING_LEVEL=DEBUG
on the worker References - Prefect git_clone step - GitRepository / RunnerStorage API - Prefect GitLab integration
- GitLab docs: Use a PAT for Git over HTTPS If you can share: - How you’re configuring the pull (prefect.yaml, Python with GitRepository, or URL embedding) - The exact error 128 message from the worker logs (with DEBUG if possible) - Whether you’re using a PAT, Project Access Token, or Deploy Token and its scopes - Whether your GitLab is self-hosted and if SSH or deploy tokens are allowed …I can tailor the fix to your setup.
k
- Confirm the worker can resolve and reach
gitlab.corp-name
(DNS/SSL/proxy) - Ensure the PAT has
read_repository
scope (or
api
) - If embedding in URL, use the full scheme
<https://oauth2>:<token>@...
and URL-encode special chars - Prefer passing the token via
access_token
or a GitLabCredentials block instead of embedding - If PATs are restricted by your org, try a GitLab Deploy Token (
username:deploy_token
) or SSH - Turn on debug logs to see the actual git command: - Set
PREFECT_LOGGING_LEVEL=DEBUG
on the worker i checked and has confirmed I have done properly the aformentioned things. I made a simple flow in prefect and run locally, it can clone the repo. Not sure what went wrong here
@Marvin, can you answer my question above
m
thought for 20.1 seconds
Short answer: you don’t need to put
oauth2:<token>@
in the URL. In Prefect 3.x, the most reliable fix is to pass the token via the
git_clone
step’s
access_token
or a
GitLabCredentials
block instead of embedding it in the URL. What to do: - Use the
git_clone
step with
access_token
Copy code
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://gitlab.corp-name/group/project.git>
      branch: main
      access_token: "{{ env.GITLAB_PAT }}"
Set
GITLAB_PAT
on your worker environment. Link: git_clone step - Or use a GitLabCredentials block
Copy code
from prefect_gitlab import GitLabCredentials
GitLabCredentials(access_token="glpat-...").save("my-gitlab-creds")
Then reference the block in your pull config or
GitRepository
credentials. Link: Prefect GitLab integration If you must embed in the URL, use the full HTTPS form and URL-encode any special characters:
Copy code
<https://oauth2:<YOUR_TOKEN>@gitlab.corp-name/group/project.git>
Common causes of git error 128: - Missing
https://
scheme - Token not URL-encoded - Token lacks
read_repository
(or
api
) scope - Wrong URL/host, DNS/SSL/proxy restrictions on the worker - Org policy blocks PATs for Git over HTTPS (use a Deploy Token or SSH instead) Workarounds that avoid pulling: - Bake the flow code into your worker image and load it via
Flow.from_source
with a local path - Use remote storage (S3/GS/Azure) and
pull_from_remote_storage
Link: Store your flow code If you share how you’re configuring the pull (prefect.yaml vs Python), which token type (Personal vs Project vs Deploy Token) and the exact 128 message from worker logs, I can pinpoint the fix quickly.
k
Token not URL-encoded ?
@Marvin Token not URL-encoded ? How to do that?
m
thought for 37.3 seconds
Happy to help! Can you share where you’re seeing “Token not URL-encoded”? For example: a database connection URL, storage URL (S3/GCS), a webhook, or a deployment source in Prefect? The fix depends on where the token is being used. General guidance: - Only URL-encode when embedding a token inside a URL (e.g., username/password in a DB URI). If a field accepts the token directly (like a header, a Prefect Secret, or a dedicated “access_token” field), do not encode it. - Avoid double-encoding. If you already see %xx sequences (like %2F), it might already be encoded. Quick ways to URL-encode: - Python
Copy code
from urllib.parse import quote, quote_plus

token = "p@ss/word?:=+#&"

# Strict encoding (good for usernames/passwords in URLs)
encoded = quote(token, safe="")
print(encoded)  # p%40ss%2Fword%3F%3A%3D%2B%23%26

# If you specifically want spaces as + (query strings), use quote_plus:
encoded_qs = quote_plus(token, safe="")
print(encoded_qs)
- Build a full DB URL safely
Copy code
from urllib.parse import quote

user = quote("my_user", safe="")
pwd = quote("p@ss/word?:=+#&", safe="")
host = "<http://db.example.com|db.example.com>"
db = "analytics"

sqlalchemy_url = f"postgresql+psycopg://{user}:{pwd}@{host}/{db}"
print(sqlalchemy_url)
Notes for common Prefect cases: - Database/warehouse blocks or connection strings: URL-encode username/password if they contain special characters. - Storage or webhooks: If you paste a full webhook URL (e.g., Slack), do not re-encode it. Use the URL as provided. - Prefect 3.x `flow.from_source(..., access_token=...)`: pass the token as the
access_token
argument; no encoding needed. If you can share the exact string you’re building (with secrets redacted) and where you’re using it in Prefect, I can suggest the precise encoding you need.