<@ULVA73B9P> <@ULVA73B9P> I'm trying to install a python package from a private github repository, ...
s
@Marvin @Marvin I'm trying to install a python package from a private github repository, using managed workpools. I am using a prefect.yaml file for deployment, and adding steps to install via uv instead of pip, from a pyproject.toml file. Everything works, except for the installation of the python lib on the private github repository. Note: I know the token is valid because the git clone step works fine. here is my prefect.yaml file:
Copy code
# Prefect deployment configuration for ingest-mongotoy workflow
# Generated using Python

build: null
push: null
pull: null
deployments:
- name: deployment-ingest-mongotoy-prod-daily
  flow_name: ingest-mongotoy
  entrypoint: workflows/ingestion/ingest-mongotoy/src/ingest_mongotoy/_flows.py:flow_main
  pull:
  - prefect.deployments.steps.git_clone:
      id: clone-step
      repository: <https://github.com/myorg/my-repo.git>
      branch: main
      credentials: '{{ prefect.blocks.github-credentials.my-repo }}'
  - prefect.deployments.steps.run_shell_script:
      id: install-uv
      script: pip install uv
  - prefect.deployments.steps.run_shell_script:
      id: seed-git-creds
      # This avoids prompting and makes any git+https fetch use the token
      script: |
        git config --global url."<https://x-access-token>:${GITHUB_TOKEN}@github.com/".insteadOf "<https://github.com/>"
        git config --global url."<https://x-access-token>:${GITHUB_TOKEN}@github.com/".insteadOf "<ssh://git@github.com/>"
        git config --global url."<https://x-access-token>:${GITHUB_TOKEN}@github.com/".insteadOf "git@github.com:"
      env:
        GITHUB_TOKEN: "{{ prefect.blocks.github-credentials.my-repo.token }}"
      expand_env_vars: true
  - prefect.deployments.steps.run_shell_script:
      id: uv-sync
      script: 'uv export --format requirements.txt --locked --no-dev  --output-file
        requirements.txt

        uv pip install --system -r requirements.txt

        uv pip install --system .'
      directory: '{{ clone-step.directory }}/workflows/ingestion/ingest-mongotoy'
  work_pool:
    name: managed-workpool
    work_queue_name: null
    job_variables:
      image: prefecthq/prefect:3-python3.11
It seems the seed-git-creds is not working. I get the following error:
fatal: could not read Username for '<https://github.com>': terminal
m
thought for 197.7 seconds
Thanks for the detail — that error means Git is still trying to prompt for credentials during the
uv pip install
of your private dependency, so the URL rewrite isn’t being applied. A couple of likely issues and fixes: 1) Make sure the seeding script uses the correct format (no angle brackets, correct username:password placement) Slack auto-adds <…> around URLs, but in your YAML those must not be present. Also, the username/password separator needs to be a colon after the username. This is the most robust form:
Copy code
git config --global url."<https://x-access-token:${GITHUB_TOKEN}@github.com/>".insteadOf "<https://github.com/>"
git config --global url."<https://x-access-token:${GITHUB_TOKEN}@github.com/>".insteadOf "<ssh://git@github.com/>"
git config --global url."<https://x-access-token:${GITHUB_TOKEN}@github.com/>".insteadOf "git@github.com:"
Notes: - No angle brackets anywhere - The base ends with a trailing slash - Username is
x-access-token
, followed by a colon, then the token 2) Consider using a .netrc file instead (very reliable) This avoids the URL rewrite entirely and is often simpler for VCS installs:
Copy code
printf "machine <http://github.com|github.com>\n  login x-access-token\n  password %s\n" "$GITHUB_TOKEN" > ~/.netrc
chmod 600 ~/.netrc
3) Prevent interactive prompts during install Add this to your
uv
step environment to fail fast if credentials aren’t picked up:
Copy code
GIT_TERMINAL_PROMPT: "0"
Putting it together, here’s a cleaned-up version of your steps:
Copy code
- prefect.deployments.steps.git_clone:
    id: clone-step
    repository: <https://github.com/myorg/my-repo.git>
    branch: main
    credentials: '{{ prefect.blocks.github-credentials.my-repo }}'

- prefect.deployments.steps.run_shell_script:
    id: install-uv
    script: pip install uv

# Option A: URL rewrite
- prefect.deployments.steps.run_shell_script:
    id: seed-git-creds
    script: |
      set -eu
      git config --global url."<https://x-access-token:${GITHUB_TOKEN}@github.com/>".insteadOf "<https://github.com/>"
      git config --global url."<https://x-access-token:${GITHUB_TOKEN}@github.com/>".insteadOf "<ssh://git@github.com/>"
      git config --global url."<https://x-access-token:${GITHUB_TOKEN}@github.com/>".insteadOf "git@github.com:"
    env:
      GITHUB_TOKEN: "{{ prefect.blocks.github-credentials.my-repo.token }}"
    expand_env_vars: true

# Option B: .netrc (use this instead of the URL rewrite block above)
# - prefect.deployments.steps.run_shell_script:
#     id: seed-netrc
#     script: |
#       set -eu
#       printf "machine <http://github.com|github.com>\n  login x-access-token\n  password %s\n" "$GITHUB_TOKEN" > ~/.netrc
#       chmod 600 ~/.netrc
#     env:
#       GITHUB_TOKEN: "{{ prefect.blocks.github-credentials.my-repo.token }}"
#     expand_env_vars: true

- prefect.deployments.steps.run_shell_script:
    id: uv-sync
    script: |
      uv export --format requirements.txt --locked --no-dev --output-file requirements.txt
      uv pip install --system -r requirements.txt
      uv pip install --system .
    directory: '{{ clone-step.directory }}/workflows/ingestion/ingest-mongotoy'
    env:
      GIT_TERMINAL_PROMPT: "0"
A couple of additional checks: - What is the exact VCS URL in your pyproject/requirements (e.g.,
git+<https://github.com/org/private.git@tag>
vs `git+ssh://git@github.com/org/private.git@tag`)? If it’s using SSH, the rewrite above should still cover it, but using
git+https
is simpler with tokens. - Avoid echoing
git config -l
in logs — it will leak the token. If you apply the rewrite exactly as shown (or the .netrc method) and set
GIT_TERMINAL_PROMPT=0
,
uv pip install
should stop prompting and pull your private dependency non-interactively. References: - Git url.insteadOf docs: https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadof