<@ULVA73B9P> when pulling codes from github, we us...
# ask-marvin
f
@Marvin when pulling codes from github, we use github credential block with PAT. Is there any other way to connect to github rather than using pat?
m
thought for 307.7 seconds
Short answer: yes. You don’t have to use a GitHub PAT. In Prefect 3.x you can pull from private GitHub repos using a few approaches: - SSH keys (deploy keys or user keys) - Use an SSH clone URL like
git@github.com:org/repo.git
- Ensure the worker’s environment has SSH set up (private key available, optional ssh-agent, and
known_hosts
for github.com) - In
prefect.yaml
, no token is needed if SSH is already configured on the worker node/container:
Copy code
deployments:
      - name: my-deployment
        entrypoint: flows/flow.py:flow
        source:
          repo: git@github.com:org/repo.git
          branch: main
- For containers/Kubernetes, mount the private key and
known_hosts
, or set
GIT_SSH_COMMAND='ssh -i /path/to/key -o StrictHostKeyChecking=yes'
. - GitHub App installation tokens (non-PAT) - Treat these just like a token in Prefect. Store it in a Secret block and reference it in `source.credentials.access_token`:
Copy code
variables:
      GITHUB_TOKEN: "{{ prefect.blocks.secret.github_app_token }}"

    deployments:
      - name: my-deployment
        entrypoint: flows/flow.py:flow
        source:
          repo: <https://github.com/org/repo>
          branch: main
          credentials:
            access_token: "{{ variables.GITHUB_TOKEN }}"
- This avoids using a personal token and supports rotation/least privilege. - System Git credentials (GIT_ASKPASS/credential helpers) - If your workers already have a Git credential helper set up (e.g.,
GIT_ASKPASS
,
gh auth
, or enterprise SSO), Prefect can just call
git clone
and it will use those credentials automatically. In that case you can omit
source.credentials
. Related docs: - prefect.yaml reference (see the source/credentials section) A couple of notes: - Prefect doesn’t manage ssh-agent or
known_hosts
for you; that’s provided by your worker environment (local, Docker, or Kubernetes). - If you’re currently on Prefect 2.x using the GitHub credentials block, we recommend moving to Prefect 3.x and the
source
section in
prefect.yaml
. In 3.x we no longer use GitHub storage blocks; instead, use
.from_source('<https://github.com/owner/repo>')
or
prefect deploy
with
source
. Would you like guidance for your specific runtime (local, Docker, or Kubernetes), and can you confirm if you’re on Prefect 3.x? That will help me give exact steps for SSH vs GitHub App token setup.
f
@Marvin tell me more about configuring ssh on the worker node. And also about GitHub App installation tokens (non-PAT)
m
thought for 52.7 seconds
Great question — here’s a deeper dive on both topics. I’ll assume Prefect 3.x. SSH on the worker node You can pull private repos over SSH by configuring the worker environment to perform
git clone
with an SSH key. Prefect just calls Git; it doesn’t manage ssh-agent or known_hosts for you. High-level steps - Generate an SSH key (prefer ed25519) dedicated to Prefect -
ssh-keygen -t ed25519 -C "prefect-worker" -f ~/.ssh/id_ed25519
- Add the public key as a GitHub Deploy Key on the repo (least privilege) or to a GitHub user account - Deploy key: repo Settings → Deploy keys → Add key - User key: Add an SSH key to your GitHub account - Lock down permissions -
chmod 700 ~/.ssh
-
chmod 600 ~/.ssh/id_ed25519
- Trust GitHub host key -
ssh-keyscan <http://github.com|github.com> >> ~/.ssh/known_hosts
- Optionally enforce strict host checking:
-o StrictHostKeyChecking=yes
- GitHub host keys: GitHub’s SSH host keys - Test from the worker environment -
ssh -T <mailto:git@github.com|git@github.com>
should say “Hi <user/org>!” Using SSH with Prefect deployments - Use an SSH URL in
prefect.yaml
Copy code
deployments:
    - name: my-deployment
      entrypoint: flows/flow.py:flow
      source:
        repo: git@github.com:org/repo.git
        branch: main
- If your key isn’t the default path, set `GIT_SSH_COMMAND`:
Copy code
GIT_SSH_COMMAND="ssh -i /path/to/key -o StrictHostKeyChecking=yes"
Local worker (process) - Put the key in
~/.ssh
for the user that runs the worker - Ensure
known_hosts
contains GitHub’s keys - Optional: run an ssh-agent and
ssh-add ~/.ssh/id_ed25519
or just rely on
GIT_SSH_COMMAND
Docker workers - Best: mount the host’s
~/.ssh
read-only
Copy code
docker run \
    -v $HOME/.ssh:/home/prefect/.ssh:ro \
    -e GIT_SSH_COMMAND="ssh -i /home/prefect/.ssh/id_ed25519 -o StrictHostKeyChecking=yes" \
    prefecthq/prefect:3
- If you must bake keys into an image, use BuildKit secrets so they don’t persist in layers
Copy code
# syntax=docker/dockerfile:1.4
  FROM prefecthq/prefect:3
  RUN mkdir -p /home/prefect/.ssh && chmod 700 /home/prefect/.ssh
  RUN ssh-keyscan <http://github.com|github.com> >> /home/prefect/.ssh/known_hosts
  RUN --mount=type=secret,id=sshkey,dst=/tmp/sshkey \
      install -m 600 /tmp/sshkey /home/prefect/.ssh/id_ed25519
  ENV GIT_SSH_COMMAND="ssh -i /home/prefect/.ssh/id_ed25519 -o StrictHostKeyChecking=yes"
Build with:
docker build --secret id=sshkey,src=./id_ed25519 .
Kubernetes workers - Create a Secret with the private key and a ConfigMap or Secret for known_hosts
Copy code
apiVersion: v1
  kind: Secret
  metadata:
    name: github-ssh
  type: Opaque
  data:
    id_ed25519: <base64 of private key>

  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: ssh-known-hosts
  data:
    known_hosts: |
      <http://github.com|github.com> ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...
- Mount them into the Prefect worker pod and set perms (via initContainer or fsGroup) - In your Kubernetes work pool’s base job template, add: - volumes, volumeMounts for
/home/prefect/.ssh
- an initContainer to
chmod 700
the directory and
chmod 600
the key - env:
GIT_SSH_COMMAND=ssh -i /home/prefect/.ssh/id_ed25519 -o StrictHostKeyChecking=yes
Notes - Prefer Deploy Keys for specific repos to minimize blast radius - Avoid embedding secrets in images or environment if you can mount them at runtime - Keep
StrictHostKeyChecking
on where possible GitHub App installation tokens (non-PAT) You can use GitHub App installation tokens instead of PATs. Key points: - Installation tokens are short-lived (typically 1 hour). You’ll need a way to mint them just before the clone happens. - Prefect can consume a token like any other HTTPS token: - via
source.credentials.access_token
in
prefect.yaml
, or
- via Git credential helper/`GIT_ASKPASS`/`.netrc` (managed by your worker environment) Simplest (manual/short-lived) approach - Generate an installation token out-of-band and store it in a Prefect Secret block - Reference it in your deployment
Copy code
variables:
    GITHUB_TOKEN: "{{ prefect.blocks.secret.github_installation_token }}"

  deployments:
    - name: my-deployment
      entrypoint: flows/flow.py:flow
      source:
        repo: <https://github.com/org/repo>
        branch: main
        credentials:
          access_token: "{{ variables.GITHUB_TOKEN }}"
- Works for testing, but you must refresh the secret before it expires Automated token minting approaches - Sidecar/init step mints a fresh token and exposes it to Git - Use a small script or
gh
CLI to: 1) Create a JWT signed with your GitHub App private key 2) Exchange it for an installation token via GitHub API 3) Write creds for Git to use (e.g.,
.netrc
or
GIT_ASKPASS
script) - Mount the App’s private key as a secret; never hardcode it - Kubernetes: put this in an initContainer that writes
.netrc
into a shared volume mounted at
/home/prefect
-
.netrc
example:
Copy code
machine <http://github.com|github.com>
      login x-access-token
      password <INSTALLATION_TOKEN>
- Then
git clone <https://github.com/org/repo>
will work without putting the token in the URL - Docker/local: run a pre-start script that writes
.netrc
or sets
GIT_ASKPASS
to supply the token References - Prefect deployments/prefect.yaml: prefect.yaml reference - GitHub App auth: Authenticating with a GitHub App - GitHub SSH keys: Connecting to GitHub with SSH Recommendations - For hands-off automation, SSH deploy keys are the least fragile option (no rotation headaches). - If you must use GitHub App tokens, plan a short-lived token minting step that runs right before the clone (initContainer/sidecar or a wrapper script). - Never put tokens in git URLs; prefer
.netrc
, credential helpers, or Prefect’s
source.credentials.access_token
. If you can share where your workers run (local process, Docker, Kubernetes), I can give you a concrete, copy-paste config for that environment.