<@ULVA73B9P>: In the prefect.deployments.steps.git...
# ask-marvin
m
@Marvin: In the prefect.deployments.steps.git_clone step in the prefect.yaml file I would like to clone a branch from an azure devops repo. How would I go about doing this with a PAT? I have tried pasting the PAT and base64 encoded pat in the access_token field, I have also tried pasting them into the repository URL. Nothing is working and is giving exit code 128
m
thought for 153.7 seconds
Thanks for the details — a couple of gotchas here. First, in Prefect 3.x the step name is: -
prefect.deployments.steps.pull.git_clone
(not
prefect.deployments.steps.git_clone
) About Azure DevOps + PAT: - Do NOT base64 encode the PAT. Use the raw PAT. - You can authenticate either by: 1) Supplying
access_token:
and a clean HTTPS repo URL, or 2) Embedding a username and PAT in the URL (username can be anything non-empty) Here are two working patterns you can try in your prefect.yaml. Option A — use access_token (recommended; keep URL clean):
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      repository: "<https://dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"
      access_token: "${AZDO_PAT}"
- Set
AZDO_PAT
in your worker/work-pool environment. The PAT should have at least “Code: Read” scope. Option B — embed username + PAT in the URL (ensure URL-encoding if needed):
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      repository: "<https://pat:${AZDO_PAT}@dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"
- Any non-empty username is fine;
pat
is a common placeholder. - If your PAT contains special characters, URL-encode it before embedding. Common causes of exit code 128: - Auth failed: wrong/missing PAT, missing scope, or malformed URL - PAT accidentally base64-encoded - The branch name doesn’t exist (set
branch:
correctly) - The PAT is not available in the worker environment (e.g., env var not set) - Typos in the Azure DevOps path (verify with a manual
git ls-remote
on the worker) Quick verification tips: - On the same machine/environment your worker runs, try: - Using Option B’s URL directly with
git ls-remote <https://pat:<PAT>@dev.azure.com/<org>/<project>/_git/<repo>>
- Confirm the branch exists and the PAT has “Code (Read)” scope and org access - If it still fails, capture the full stderr from the step (not just exit code) — that often points to the exact auth or URL parsing issue. Docs: - Prefect step reference: prefect.deployments.steps.pull.git_clone - Prefect YAML guide: prefect.yaml how-to - Azure DevOps PAT usage: Use personal access tokens - Azure DevOps If you can share (sanitized) your current
prefect.yaml
step and the exact error output, I can pinpoint the issue quickly.
m
@Marvin: The issue with cloning was due to trying to clone a local branch - stupid user error. After I pushed my local branch I was able to clone the repo. The next step in my pull section is to install the requirements in my pyproject.toml file, with this step:
Copy code
- prefect.deployments.steps.run_shell_script:
    script: uv sync
    directory: "{{ git-clone.directory }}"
This fails with the error:
run_shell_script
failed with error code 2: error: No
pyproject.toml
found in current directory or any parent directory, even though the pyproject.toml file is in the root of the cloned repo. How can I fix this?
m
thought for 158.2 seconds
Nice catch on the branch! The “No pyproject.toml found” error almost always means the shell step didn’t run in the repo’s root. In prefect.yaml, you need to reference the output of the git clone step via its step id. If the clone step doesn’t have an id,
{{ git-clone.directory }}
won’t resolve and the command runs in the default working dir. Try this: - Give the clone step an id - Either set the working directory once, or pass it to run_shell_script Example 1 — set working directory once:
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      id: git_clone
      repository: "<https://dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"

  - prefect.deployments.steps.pull.set_working_directory:
      directory: "{{ git_clone.directory }}"

  - prefect.deployments.steps.utility.run_shell_script:
      script: uv sync
Example 2 — pass directory to the shell step:
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      id: git_clone
      repository: "<https://dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"

  - prefect.deployments.steps.utility.run_shell_script:
      script: uv sync
      directory: "{{ git_clone.directory }}"
Notes: - Use the canonical step path for the shell step:
prefect.deployments.steps.utility.run_shell_script
- If your project lives in a subdirectory of the repo, point to it: -
directory: "{{ git_clone.directory }}/path/to/project"
- To debug quickly, add a step:
Copy code
- prefect.deployments.steps.utility.run_shell_script:
      script: "pwd && ls -la"
      directory: "{{ git_clone.directory }}"
Docs: - git_clone: prefect.deployments.steps.pull.git_clone - set_working_directory: prefect.deployments.steps.pull.set_working_directory - run_shell_script: prefect.deployments.steps.utility.run_shell_script If it still fails, share your full pull section and I’ll spot-check the templating and paths.
m
@Marvin I've added the id to the git_clone step and I also added the step:
Copy code
- prefect.deployments.steps.pull.set_working_directory:
    directory: "{{ git_clone.directory }}/prefect"
This points to the subdirectory where my flow code are stored, and also the prefect.yaml file, assuming that the
{{ git_clone.directory }}
is the root of my cloned repo. Inside the prefect-folder I have a new folder called flows, so that the entrypoint in my deployment is:
flows/my_flow.py:my_flow
. However, the deployment is failing with the error: [Errno 2] No such file or directory: '/prefect' Indicating that the set-directory-step is failing
m
thought for 90.8 seconds
Thanks for the details — the error “[Errno 2] No such file or directory: ‘/prefect’” tells us Jinja rendered
{{ git_clone.directory }}
to an empty string, so your template became just
/prefect
. That means the git_clone step’s output isn’t being found by the template. Most common causes: - The step id isn’t attached correctly to the git_clone step (indentation/YAML structure) - The referenced id in the template doesn’t exactly match the step id - set_working_directory runs before git_clone - The step path is slightly off Use this minimal working example as a template:
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      id: git_clone
      repository: "<https://dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"

  # Optional: Debug what the template resolves to
  - prefect.deployments.steps.utility.run_shell_script:
      script: |
        echo "Repo dir: {{ git_clone.directory }}"
        ls -la "{{ git_clone.directory }}"
        ls -la "{{ git_clone.directory }}/prefect"

  - prefect.deployments.steps.pull.set_working_directory:
      directory: "{{ git_clone.directory }}/prefect"

  - prefect.deployments.steps.utility.run_shell_script:
      script: uv sync
Checklist: - Ensure the id is nested under the git_clone step (as shown). This is the correct syntax: - prefect.deployments.steps.pull.git_clone: id: git_clone ... - The template must match the id exactly:
{{ git_clone.directory }}
- set_working_directory must come after the git_clone step - The canonical path for the shell step is
prefect.deployments.steps.utility.run_shell_script
(not
prefect.deployments.steps.run_shell_script
) If the debug step prints an empty line for “Repo dir:”, the id reference isn’t resolving. In that case, please paste your full pull section and I’ll spot the mismatch. Docs for reference: - git_clone: prefect.deployments.steps.pull.git_clone - set_working_directory: prefect.deployments.steps.pull.set_working_directory - run_shell_script: prefect.deployments.steps.utility.run_shell_script
m
@Marvin: There is no output in the prefect UI for the run_shell_scripts. I am using a prefect managed worker. There is even this comment in the prefect docs:
run_shell_script
allows for the execution of one or more shell commands in a subprocess, and returns the standard output and standard error of the script. This step is useful for scripts that require execution in a specific environment, or those which have specific input and output requirements. Note that setting
stream_output: true
for
run_shell_script
writes the output and error to stdout in the execution environment, which will not be sent to the Prefect API.
m
thought for 59.3 seconds
You’re right about the logging note — pull steps run on the worker and, with stream_output: true (the default), their stdout/stderr won’t be sent to the API/UI. To see those logs: - Check your Managed Worker logs in the UI (Work Pools -> your managed pool -> Worker logs) - Or temporarily set stream_output: false and write the output to a file in the repo if you need to retrieve it later; the result isn’t automatically shown in the UI either. That said, the “[Errno 2] No such file or directory: ‘/prefect’” means the Jinja template resolved to an empty string for the repo path, so the final path became “/prefect”. In other words, {{ git_clone.directory }} did not resolve. This almost always comes down to YAML structure or an id mismatch. Use this known-good structure:
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      id: git_clone
      repository: "<https://dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"

  # Quick debug: verify the resolved directory on the worker
  - prefect.deployments.steps.utility.run_shell_script:
      script: |
        echo "Repo dir: {{ git_clone.directory }}"
        ls -la "{{ git_clone.directory }}"
        ls -la "{{ git_clone.directory }}/prefect"
      stream_output: true

  # Option A: set CWD for subsequent steps
  - prefect.deployments.steps.pull.set_working_directory:
      directory: "{{ git_clone.directory }}/prefect"

  # Option B (simpler/less stateful): pass directory directly to each script
  - prefect.deployments.steps.utility.run_shell_script:
      script: uv sync
      directory: "{{ git_clone.directory }}/prefect"
      stream_output: true
Checklist: - The id must be nested under the git_clone step itself (indentation matters) - The reference must match exactly: {{ git_clone.directory }} - set_working_directory must come after git_clone - Use the canonical paths: - prefect.deployments.steps.pull.git_clone - prefect.deployments.steps.pull.set_working_directory - prefect.deployments.steps.utility.run_shell_script If the debug step prints “Repo dir:” as empty or errors on the ls command, the template isn’t resolving. In that case, please paste your full pull section and I’ll spot any indentation/id/path issues. Docs for reference: - git_clone: prefect.deployments.steps.pull.git_clone - set_working_directory: prefect.deployments.steps.pull.set_working_directory - run_shell_script: prefect.deployments.steps.utility.run_shell_script
m
@Marvin There was indeed a typo in the id reference, after that was fixed I now get the error
FileNotFoundError: [Errno 2] No such file or directory: 'my_repo-my-branch/prefect'
even though I know that I have a folder named prefect in the root of my repo
m
thought for 278.3 seconds
Got it — that FileNotFoundError is happening because the path you’re giving set_working_directory is relative. The value from {{ git_clone.directory }} is typically the repo folder name (e.g. my_repo-my-branch), not an absolute path. On a managed worker, the initial CWD may not be the parent of that folder, so chdir to "my_repo-my-branch/prefect" fails. Two solid fixes: Option A — set a known base CWD before cloning - This ensures git_clone creates the repo under a directory you control (e.g. /tmp), so the relative path works.
Copy code
pull:
  - prefect.deployments.steps.pull.set_working_directory:
      directory: "/tmp"

  - prefect.deployments.steps.pull.git_clone:
      id: git_clone
      repository: "<https://dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"

  - prefect.deployments.steps.pull.set_working_directory:
      directory: "{{ git_clone.directory }}/prefect"

  - prefect.deployments.steps.utility.run_shell_script:
      script: uv sync
Option B — don’t change global CWD; point run_shell_script at the repo folder - Less stateful and easier to debug.
Copy code
pull:
  - prefect.deployments.steps.pull.set_working_directory:
      directory: "/tmp"

  - prefect.deployments.steps.pull.git_clone:
      id: git_clone
      repository: "<https://dev.azure.com/<org>/<project>/_git/<repo>>"
      branch: "my-branch"

  - prefect.deployments.steps.utility.run_shell_script:
      script: uv sync
      directory: "{{ git_clone.directory }}/prefect"
Tip: Add a quick sanity check step to confirm the path on the worker (logs show up in Managed Worker logs, not the flow run UI):
Copy code
- prefect.deployments.steps.utility.run_shell_script:
    script: |
      echo "PWD: $(pwd)"
      echo "Repo dir: {{ git_clone.directory }}"
      ls -la "{{ git_clone.directory }}"
      ls -la "{{ git_clone.directory }}/prefect"
    stream_output: true
Relevant refs: - set_working_directory: prefect.deployments.steps.pull.set_working_directory - git_clone: prefect.deployments.steps.pull.git_clone - run_shell_script: prefect.deployments.steps.utility.run_shell_script If you prefer a different base than /tmp, just set that in the first set_working_directory step and everything else will line up.