<@ULVA73B9P> I'm getting a `StepExecutionError: En...
# ask-marvin
a
@Marvin I'm getting a
StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone
when deploying to Prefect Cloud managed infrastructure. My setup: python
Copy code
source = GitRepository(
    url="<https://gitlab.com/myorg/myrepo.git>",
    branch="main",
    credentials=GitLabCredentials.load("gitlab-creds")
)

deployment = flow.from_source(
    source=source,
    entrypoint="flows/my_flow.py:my_flow"
).deploy(
    name="my-deployment",
    work_pool_name="prefect-managed",  # Prefect managed infrastructure
    job_variables={
        "pip_packages": ["pandas>=1.3.0", "boto3>=1.34.162", ...]  # Required packages
    }
)
The issue: • Other deployments work fine • This deployment ONLY works if I include
pip_packages
in
job_variables
• But when I include
pip_packages
, the git_clone step fails • Using Prefect Cloud with managed work pools (not self-hosted) Question: Whats is going up ? Yesterday works just fine Environment: • Prefect Cloud (managed work pools) • GitLab private repository • Prefect version: 2.20
m
thought for 308.7 seconds
Thanks for the details — a few things jump out that we should clarify and test. First, a quick version check - The code you posted (
flow.from_source(...)
with
GitRepository
) is a Prefect 3.x API. If you are truly on Prefect 2.20, this code shouldn’t exist. Could you paste the output of:
Copy code
prefect version
from the exact environment where you run the deploy? What I suspect is happening - Managed work pools run your “pull steps” (including
prefect.deployments.steps.git_clone
) inside a Prefect-managed container. - When you set
job_variables.pip_packages
, the managed runner takes a slightly different path to prep the runtime. It looks like that path is currently using an image that does not have the
git
binary available, causing the
git_clone
step to fail. Without
pip_packages
, it uses an image where
git
is present and the clone succeeds. - If the error log includes something like “git: not found” or an exit code 127, that’s the smoking gun. Can you share the exact error lines from the pull-step logs? - Open the failed flow run in the UI → Logs → look for the “pull step” message that includes
prefect.deployments.steps.git_clone
, and paste the last ~20 lines around the error. This will tell us if it’s: - “git: not found” (missing binary) - “fatal: Authentication failed” (token/permissions) - TLS/SSL error (certs) - Exit code 128/other Quick things to double-check - Ensure your actual code uses
>=
and not the HTML-escaped
>=
that Slack shows. It should be:
Copy code
job_variables={
  "pip_packages": ["pandas>=1.3.0", "boto3>=1.34.162"]
}
- Confirm your
GitLabCredentials
block is valid and hasn’t expired or changed permissions (though this seems less likely since clone works when
pip_packages
is omitted). Workarounds to unblock you 1) Pin the runtime image to one that includes git - Add an image to your job variables to override the base image used by the managed pool:
Copy code
deployment = flow.from_source(
    source=source,
    entrypoint="flows/my_flow.py:my_flow"
).deploy(
    name="my-deployment",
    work_pool_name="prefect-managed",
    job_variables={
        "image": "prefecthq/prefect:3-latest",
        "pip_packages": [
            "pandas>=1.3.0",
            "boto3>=1.34.162"
        ]
    }
)
- If this makes the
git_clone
step succeed, it confirms the base image used when
pip_packages
is set was missing
git
. 2) Alternatively, bake a custom image - Create a small image FROM
prefecthq/prefect:3-latest
(which includes git) with your dependencies pre-installed, push to your registry, then set:
Copy code
job_variables={"image": "<your-registry>/<your-image>:<tag>"}
- This avoids runtime pip entirely. References - Customizing managed work pool job variables: Customize job variables - Managed infrastructure overview: Managed infrastructure Next steps - Please share: - The exact
prefect version
output - The specific error text from the
git_clone
step logs - Whether pinning
image: prefecthq/prefect:3-latest
resolves the failure If it’s indeed “git not found” when
pip_packages
is present, we’ll flag this as a regression on the managed image path and get it fixed. In the meantime, pinning the image should unblock you.
a
The version I use to test is the version 2.20. Which image I could use ? @Marvin
m
thought for 41.8 seconds
Thanks — that helps. There are two important points here: 1) You’re testing with Prefect 2.20, but your code and work pool type are 3.x -
flow.from_source(...).deploy(...)
and the
prefect-managed
work pool are Prefect 3.x features. They aren’t available in 2.20. - If you want to use Prefect Cloud Managed Work Pools, you’ll need to run a Prefect 3.x client. Migration guide: Upgrade 2.x to 3.x 2) Which image to use - If you’re on 3.x with a Managed work pool, set an image in
job_variables
to avoid any base-image drift and ensure
git
is present. The recommended image is: -
prefecthq/prefect:3-latest
- Example:
Copy code
deployment = flow.from_source(
    source=source,
    entrypoint="flows/my_flow.py:my_flow"
).deploy(
    name="my-deployment",
    work_pool_name="prefect-managed",
    job_variables={
        "image": "prefecthq/prefect:3-latest",
        "pip_packages": [
            "pandas>=1.3.0",
            "boto3>=1.34.162"
        ]
    }
)
- This also pins the image so “yesterday worked, today broke” doesn’t happen due to a base image change. If you must stay on 2.20 and run your own workers (Docker/K8s/etc.), you can use: -
prefecthq/prefect:2-latest
or a specific tag like
prefecthq/prefect:2.20-python3.11
- Ensure
git
is installed in the image. If it’s not, bake a small custom image:
Copy code
FROM prefecthq/prefect:2.20-python3.11
RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*
Two quick gotchas: - Make sure your
pip_packages
use real
>=
(not Slack’s
>=
), e.g.
pandas>=1.3.0
. - If the
git_clone
step still fails, please paste the last ~20 lines of the pull-step logs. If it says “git: not found”, the image didn’t include git; if it’s an auth error, we’ll focus on the
GitLabCredentials
block. Docs for reference: - Managed infrastructure overview: Managed infrastructure - Customizing job variables: Customize job variables If you confirm which path you’re taking (upgrading to 3.x managed, or staying on 2.20 with your own workers), I can give you exact, copy/paste-ready config.
a
@Marvin I'm currently using version 3 of prefect, this is my deploy and I've already configured the image and the error still doesn't matter that it can't clone the code.
Copy code
deployment = flow.from_source(
                source=source, entrypoint=flow_config["entrypoint"]
            ).deploy(
                name=flow_config["name"],
                work_pool_name=env_config["work_pool"],
                tags=flow_tags,
                description=flow_config["description"],
                version=os.getenv("CI_COMMIT_SHORT_SHA", "latest"),
                schedule=flow_config.get("schedule"),
                job_variables={
                    "image": "prefecthq/prefect:3-latest",
                    "pip_packages": [
                        "requests>=2.32.5",
                        "boto3>=1.34.162",
                        "pyarrow>=17.0.0",
                        "prefect-aws",
                        "pandas>=1.3.0",
                        "sqlalchemy>=1.4.0",
                        "psycopg2-binary>=2.9.0",
                        "numpy>=1.22.0",
                        "scipy>=1.9.0"
                    ]
                },
            )
m
thought for 271.8 seconds
Thanks for confirming you’re on 3.x and for sharing the deploy snippet. A few focused checks and a reliable workaround: Key things to verify - Check the actual error text from the pull-step logs. In the failed flow run, open Logs and find the lines around
prefect.deployments.steps.git_clone
. Is it: - “git: not found” or “[Errno 2] No such file or directory: 'git'” → image doesn’t have git - “fatal: Authentication failed” → token/permissions on the GitLabCredentials block - TLS/SSL or network errors Please paste the last ~20 lines of that section. - Make sure your
pip_packages
use
>=
not Slack’s
>=
. In your message it shows:
Copy code
"requests>=2.32.5"
It must be:
Copy code
"requests>=2.32.5"
If the literal
>=
gets into the container, pip will error and the runtime prep path may break before pull steps. - Token scope: your GitLab PAT should have at least the read_repository scope. Double-check the
GitLabCredentials
block is up to date. Why this likely happens - Managed work pools prep the container (including installing
pip_packages
) before running pull steps. If that prep path uses an image without
git
or the prep fails,
git_clone
will fail. You’ve set
image: prefecthq/prefect:3-latest
, which should include git, but if the prep is still failing or a different image path is being used, the result looks the same. A robust way to unblock - Bake a small image that includes git and all your deps, then do not use
pip_packages
(so there’s no runtime package install step): Dockerfile:
Copy code
FROM prefecthq/prefect:3-latest
RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var/lib/apt/lists/*
# Optional: preinstall your deps to avoid runtime installs
RUN pip install --no-cache-dir \
    requests>=2.32.5 \
    boto3>=1.34.162 \
    pyarrow>=17.0.0 \
    prefect-aws \
    pandas>=1.3.0 \
    sqlalchemy>=1.4.0 \
    psycopg2-binary>=2.9.0 \
    numpy>=1.22.0 \
    scipy>=1.9.0
Then set your deployment to use only the custom image:
Copy code
job_variables={
  "image": "<your-registry>/<your-image>:<tag>"
  # no pip_packages
}
This removes the runtime path that seems to be causing the clone failure and guarantees
git
is present. If you want to keep pip_packages - First fix any
>=
to
>=
. - Try pinning to a specific Prefect 3 image tag instead of
3-latest
(to avoid any base-image drift), e.g. the exact tag you know worked previously. - If it still fails, please share: - The exact error lines from
git_clone
- The flow run ID and work pool name - Confirmation of the GitLab PAT scopes Docs - Managed work pools: Managed infrastructure - Job variables: Customize job variables If you share the error text, I can pinpoint whether it’s missing
git
, an auth issue, or something else and advise a narrower fix. If it’s a regression on the managed path, I’ll escalate with our team using your run ID.
a
@Marvin the error is Failed to clone repository 'https://gitlab.com/parcoapp/data/etl/verified_users_etl.git' with exit code 128.
m
thought for 140.4 seconds
Thanks for the snippet and the error. Exit code 128 is a generic git failure, but the message you pasted is a big clue: Failed to clone repository 'https://gitlab.com/parcoapp/data/etl/verified_users_etl.git' with exit code 128. Two important issues to fix/check: 1) Your repository URL must not include angle brackets - In your code, make sure the URL is a plain string without Slack’s <> markup:
Copy code
source = GitRepository(
    url="<https://gitlab.com/parcoapp/data/etl/verified_users_etl.git>",
    branch="main",
    credentials=GitLabCredentials.load("gitlab-creds")
)
If the literal
<
and
>
make it into the clone command, git will fail (128) because the URL is malformed. The duplicated-looking URL in the error strongly suggests markup is leaking into the string. 2) Fix pip_packages operators (avoid Slack’s >) - Ensure your
pip_packages
use real >= operators, not HTML-escaped
>=
. For example:
Copy code
job_variables={
  "image": "prefecthq/prefect:3-latest",
  "pip_packages": [
    "requests>=2.32.5",
    "boto3>=1.34.162",
    "pyarrow>=17.0.0",
    "prefect-aws",
    "pandas>=1.3.0",
    "sqlalchemy>=1.4.0",
    "psycopg2-binary>=2.9.0",
    "numpy>=1.22.0",
    "scipy>=1.9.0"
  ],
  "env": {
    "PREFECT_LOGGING_LEVEL": "DEBUG",
    "GIT_TRACE": "1",
    "GIT_CURL_VERBOSE": "1"
  }
}
Those env vars will make the git error much more explicit in your run logs if something else is wrong. If you still see exit code 128 after the above - Auth is the next most common cause (repo not found / authentication failed). Please confirm: - Your GitLab PAT in the
GitLabCredentials
block has read_repository scope - The repo/branch exist and are accessible to that token - For a quick A/B test, embed the token in the URL temporarily to confirm auth (replace once verified):
Copy code
# Be careful with secrets; use this only to test quickly
token = os.getenv("GITLAB_TOKEN")
source = GitRepository(
    url=f"<https://oauth2:{token}@gitlab.com/parcoapp/data/etl/verified_users_etl.git>",
    branch="main",
)
If this works, the issue is with how the credentials block is being resolved; if it still fails, it’s the URL/permissions. Optional hardening - Pin to a specific image tag to avoid base-image drift, e.g.:
Copy code
"image": "prefecthq/prefect:3.0.3-python3.11"
- Or bake a small image with git and your deps pre-installed and remove pip_packages:
Copy code
FROM prefecthq/prefect:3-latest
RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir requests>=2.32.5 boto3>=1.34.162 pyarrow>=17.0.0 prefect-aws pandas>=1.3.0 sqlalchemy>=1.4.0 psycopg2-binary>=2.9.0 numpy>=1.22.0 scipy>=1.9.0
Then:
Copy code
job_variables={"image": "<your-registry>/<your-image>:<tag>"}
If the issue persists after removing <> from the URL and fixing >=, please paste the git stderr lines from the logs (with GIT_TRACE/GIT_CURL_VERBOSE enabled) and the flow run ID. I can pinpoint whether it’s malformed URL vs. auth and escalate if needed.