<@ULVA73B9P> The flows deployed in the managed poo...
# ask-marvinm
Medwin David
10/26/2025, 2:15 AM@Marvin The flows deployed in the managed pool started crashing with error. There was no changes made recently
Error:
Failed to clone repository with exit code 128.
m
Marvin
10/26/2025, 2:15 AM✅ thought for 236.6 seconds
Marvin
10/26/2025, 2:19 AMThanks for flagging this — sorry your runs started crashing.
Exit code 128 from “git clone” almost always means auth or reachability. In Prefect 3.x managed work pools (Prefect-hosted), the runtime environment does not have your SSH keys, so cloning private repos must use HTTPS with a token provided via deployment pull steps.
A few quick confirmations that often explain “no code changes” but sudden failures:
- Personal Access Token (PAT) expired, got revoked, or lost org access (e.g., SSO/2FA enforcement or repo permission change)
- Repo renamed/transferred/archived, branch deleted, or a pinned commit SHA no longer exists
- Fine-grained PAT lost repository permission
- Private submodules were added or updated but token doesn’t have access to them
- Temporary GitHub outage or rate limits (unauthenticated limit is low)
Could you share:
- Whether the repo is private and which provider (GitHub/GitLab/Bitbucket)?
- Your deployment’s
prefect.yamlpullbranchcommit_sha Copy code
curl -H "Authorization: Bearer <TOKEN>" <https://api.github.com/user> Copy code
from prefect.blocks.system import Secret
Secret(value="ghp_...").save("github-token", overwrite=True) Copy code
pull:
- prefect.deployments.steps.pull.git_clone:
repository: "<https://github.com/ORG/REPO>"
branch: "main" # or remove if you use commit_sha
include_submodules: false # set true if you use them
access_token: "{{ prefect.blocks.secret.github-token }}"access_tokeninclude_submodules: trueprefect.yamlcommit_sha Copy code
pull:
- prefect.deployments.steps.pull.pull_from_remote_storage:
- path: "<s3://my-bucket/path/to/project.tar.gz>"prefect.yamlm
Medwin David
10/26/2025, 2:28 AM@Marvin
yaml code:
Copy code
definitions:
work_pools:
managed-pool: &managed-pool
name: managed-pool
work_queue_name: nexus
pull:
- prefect.deployments.steps.git_clone:
id: git-clone
repository: "<https://gitlab.example.com/example/data-analytics/nexa.git>"
branch: "dev"
credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-integration }}"
- prefect.deployments.steps.run_shell_script:
directory: "{{ git-clone.directory }}"
stream_output: true
script: bash prefect-build-project.sh Copy code
#!/usr/bin/env sh
echo ">>> Installing prefect-gitlab to fetch GitLab PAT"
pip install prefect-gitlab
# enable installing of private repos
echo "================= Building project using UV"
GITLAB_PAT=$(
python - <<EOF
from prefect_gitlab import GitLabCredentials
# Load the secret block by its name
gitlab_creds = GitLabCredentials.load("gitlab-integration")
gitlab = gitlab_creds.get_client()
# Retrieve and print the secret value
print(gitlab_creds.token.get_secret_value())
EOF
) || {
printf '%s\n' "Failed to get the secret value" >&2
exit 1
}
echo "================= Setting gitconfig"
git config --global url."<https://oauth2>:${GITLAB_PAT}@gitlab.example.com".insteadOf <ssh://git@gitlab.example.com>
echo "================= Running using UV to build"
uv pip install --system . || {
printf '%s\n' "UV sync failed in directory: $(pwd) --> gitlab pat: ${GITLAB_PAT}" >&2
exit 1
}
# add a random path to /app/ to error out and print logs to screenm
Marvin
10/26/2025, 2:28 AM✅ thought for 57.3 seconds
Marvin
10/26/2025, 2:29 AMThanks for sharing the YAML and script — that helps a lot.
I see two likely culprits for “Failed to clone repository with exit code 128” in a Managed Work Pool:
1) The credentials block type isn’t available at clone time in the managed image
- Your pull step passes a GitLabCredentials block:
Copy code
credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-integration }}"access_token Copy code
pull:
- prefect.deployments.steps.pull.git_clone:
id: git-clone
repository: "<https://gitlab.example.com/example/data-analytics/nexa.git>"
branch: "dev"
# If you use submodules, set this true and ensure the PAT can access them
include_submodules: false
access_token: "{{ prefect.blocks.secret.gitlab-pat }}"
- prefect.deployments.steps.utility.run_shell_script:
directory: "{{ git-clone.directory }}"
stream_output: true
script: bash prefect-build-project.sh Copy code
from prefect.blocks.system import Secret
Secret(value="YOUR_GITLAB_PAT").save("gitlab-pat", overwrite=True) Copy code
curl --header "PRIVATE-TOKEN: <PAT>" <https://gitlab.example.com/api/v4/user>include_submodules: trueprefect.deployments.steps.pull.git_cloneprefect.deployments.steps.utility.run_shell_scriptm
Medwin David
10/26/2025, 2:39 AM@Marvin repo dont have submodules, the gitlab.example.com is not publicly reachable. The PAt status its new with all the required permissions.
Why would it start crashing if there was nothing changed recently , as it was running till friday
m
Marvin
10/26/2025, 2:39 AM✅ thought for 249.0 seconds
Marvin
10/26/2025, 2:43 AMThanks for the added detail — this explains the behavior.
Key point: your GitLab is not publicly reachable. Prefect’s Managed Work Pool runs your code in Prefect-hosted infrastructure on the public Internet. Those runs cannot reach private/internal endpoints (no VPN, no private routes). If your repo is only accessible on your internal network, a managed pool won’t be able to clone it — token or not.
Why it “suddenly” started failing if nothing changed
- Most common: earlier successful runs were executed by a self-hosted worker inside your network (e.g., a Process/Docker/K8s pool), then that worker stopped or became unavailable, and the Managed pool picked up runs. Managed runs can’t reach your internal GitLab and fail with exit code 128.
- Other possibilities outside your code:
- Your firewall/DNS changed on Friday (blocking public egress to gitlab.example.com).
- The work queue routing changed so a different pool/worker started executing your runs.
How to confirm
- Open a successful pre-Friday run and a failing run in the Prefect UI and compare the “Worker”/“Infrastructure” info. If the successful ones show a self-hosted worker and the failing ones show “Managed”, that’s the change.
- You can also temporarily replace the first pull step with a quick connectivity probe to see what the managed environment can reach:
Copy code
pull:
- prefect.deployments.steps.utility.run_shell_script:
stream_output: true
script: |
set -x
curl -I <https://gitlab.example.com> || true
getent hosts <http://gitlab.example.com|gitlab.example.com> || trueprefect.deployments.steps.pull.git_cloneprefect.deployments.steps.utility.run_shell_script Copy code
prefect worker start -p your-internal-pool -q nexus Copy code
pull:
- prefect.deployments.steps.pull.pull_from_remote_storage:
path: "<s3://my-bucket/path/to/project.tar.gz>" Copy code
pull:
- prefect.deployments.steps.pull.git_clone:
repository: "<https://gitlab.example.com/example/data-analytics/nexa.git>"
branch: "dev"
access_token: "{{ prefect.blocks.secret.gitlab-pat }}" Copy code
from prefect.blocks.system import Secret
Secret(value="YOUR_GITLAB_PAT").save("gitlab-pat", overwrite=True)Marvin
10/26/2025, 2:43 AMIf you can check a pre-Friday successful run’s “Worker” and share what pool/worker ran it, I can give you exact steps to migrate the deployment to that pool or set up a new self-hosted one that will restore connectivity.
m
Medwin David
10/26/2025, 3:05 AM@Marvin my mistake the gitlab repo is publicly accessible
m
Marvin
10/26/2025, 3:05 AM✅ thought for 35.9 seconds
Marvin
10/26/2025, 3:05 AMGot it — if the repo is public, managed should be able to clone it without credentials. A few things jump out that can cause exit code 128 here:
Most likely issues
- Angle brackets in your YAML/script: Your snippets show repository and git config values inside <...>. Those angle brackets are Slack’s link formatting; if they made it into your actual files, git treats them as part of the URL and fails.
- Wrong step path: Use prefect.deployments.steps.pull.git_clone (not prefect.deployments.steps.git_clone).
- Branch mismatch: If the dev branch was renamed/deleted or protected recently, git clone -b dev will fail with exit 128.
Immediate fixes
- Use the fully-qualified pull step and remove credentials for a public repo. Also remove any < > around URLs.
- Keep your script for installing deps as is, but fix the git config line to not include < >.
Working example
Copy code
pull:
- prefect.deployments.steps.utility.run_shell_script:
stream_output: true
script: |
set -ex
git --version
# Check that the repo and branch resolve from the managed environment
git ls-remote --heads <https://gitlab.example.com/example/data-analytics/nexa.git> dev
- prefect.deployments.steps.pull.git_clone:
id: git-clone
repository: "<https://gitlab.example.com/example/data-analytics/nexa.git>"
branch: "dev"
# no credentials needed for public repo
- prefect.deployments.steps.utility.run_shell_script:
directory: "{{ git-clone.directory }}"
stream_output: true
script: bash prefect-build-project.sh Copy code
#!/usr/bin/env sh
set -e
echo ">>> Installing prefect-gitlab to fetch GitLab PAT"
pip install prefect-gitlab
# If you still need PAT for private pip installs later, keep this:
GITLAB_PAT=$(
python - <<EOF
from prefect_gitlab import GitLabCredentials
gitlab_creds = GitLabCredentials.load("gitlab-integration")
print(gitlab_creds.token.get_secret_value())
EOF
) || {
printf '%s\n' "Failed to get the secret value" >&2
exit 1
}
# Only needed if you install private repos via git+ssh later; fix URL patterns and drop <>
echo "================= Setting gitconfig"
git config --global url."<https://oauth2:${GITLAB_PAT}@gitlab.example.com>".insteadOf <ssh://git@gitlab.example.com>
echo "================= Running using UV to build"
uv pip install --system . || {
printf '%s\n' "UV sync failed in directory: $(pwd)" >&2
exit 1
}