Hui Wu
03/31/2026, 8:26 AMMarvin
03/31/2026, 8:26 AMMarvin
03/31/2026, 8:32 AMflow.from_source(...). Use GitRepository(credentials=...) for HTTPS tokens or rely on your worker’s SSH setup for SSH URLs.
Quick overview:
- HTTPS + token: pass credentials (dict or block) to GitRepository
- SSH: use git@... URL; ensure your worker has the SSH key configured
- Submodules: set `include_submodules=True`; Prefect will auto-propagate creds
- Environment vars: Prefect does not auto-read `GITHUB_TOKEN`/`GIT_TOKEN`; read them yourself (or use a Secret block) and pass to credentials
Links
- Docs: Store flow code (Git, from_source, git_clone)
- Source: GitRepository (auth and submodules)
- Integration blocks (GitHub, GitLab, Bitbucket)
Supported parameters (Python SDK)
- GitRepository(url, credentials=None, branch=None, commit_sha=None, include_submodules=False, directories=None, pull_interval=60)
- credentials can be:
- A simple dict such as {"access_token": "<token>"} (optionally with username for Bitbucket Server)
- A Block that formats git credentials (e.g., GitHub/GitLab/Bitbucket credentials from integrations)
- A Secret block used within the dict, e.g., {"access_token": Secret.load("...")}
Examples
1) HTTPS with GitHub PAT (recommended: Secret block)
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret
# Save once (or create via UI)
# Secret(value="ghp_xxx").save("github-token", overwrite=True)
repo = GitRepository(
url="<https://github.com/owner/repo.git>",
credentials={"access_token": Secret.load("github-token")}
)
f = flow.from_source(source=repo, entrypoint="flows/my_flow.py:my_flow")
f.deploy(name="github-pat-deployment", work_pool_name="my-pool")
2) HTTPS with GitLab PAT or Deploy Token
- GitLab PATs are automatically formatted as oauth2:<token>.
- Deploy tokens in username:token form are used as-is (no oauth2: prefix).
from prefect import flow
from prefect.runner.storage import GitRepository
# Personal Access Token (PAT)
repo_pat = GitRepository(
url="<https://gitlab.com/group/project.git>",
credentials={"access_token": "glpat_xxx"}
)
# Deploy Token (username:token)
repo_deploy = GitRepository(
url="<https://gitlab.com/group/project.git>",
credentials={"access_token": "deploy-user:deploy-token-xxx"}
)
flow.from_source(source=repo_pat, entrypoint="flows/main.py:my_flow") \
.deploy(name="gitlab-pat", work_pool_name="my-pool")
3) SSH key auth (no SDK parameter; configure on the worker)
- Use SSH URL: git@github.com:owner/repo.git
- Ensure the worker runtime has the SSH private key and known_hosts configured (e.g., in ~/.ssh, with ssh-agent)
from prefect import flow
from prefect.runner.storage import GitRepository
repo = GitRepository(url="git@github.com:owner/private-repo.git")
flow.from_source(source=repo, entrypoint="flows/main.py:my_flow") \
.deploy(name="ssh-deployment", work_pool_name="my-pool")
Notes:
- Prefect does not accept an SSH key parameter. Git/SSH uses the process environment (e.g., SSH_AUTH_SOCK, GIT_SSH_COMMAND) if present.
4) Private submodules
- Prefect will automatically propagate credentials to submodules using a git config rewrite when include_submodules=True.
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret
repo = GitRepository(
url="<https://github.com/owner/parent-with-submodules.git>",
credentials={"access_token": Secret.load("github-token")},
include_submodules=True
)
flow.from_source(source=repo, entrypoint="flows/main.py:my_flow") \
.deploy(name="with-submodules", work_pool_name="my-pool")Marvin
03/31/2026, 8:32 AM# pip install prefect-github
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect_github import GitHubCredentials
# GitHubCredentials(token="ghp_xxx").save("my-github-creds", overwrite=True)
creds = GitHubCredentials.load("my-github-creds")
repo = GitRepository(
url="<https://github.com/owner/repo.git>",
credentials=creds,
)
flow.from_source(source=repo, entrypoint="flows/main.py:my_flow") \
.deploy(name="github-creds-block", work_pool_name="my-pool")
Integration docs: https://docs.prefect.io/integrations/
Using environment variables
- Prefect 3.x does not automatically read GITHUB_TOKEN, GH_TOKEN, or similar for Git auth. If you prefer env vars:
- Read them in Python and pass to credentials
- Or store them in a Prefect Secret block and reference that
- Git itself will use standard env if present (e.g., GIT_ASKPASS, SSH_AUTH_SOCK, GIT_SSH_COMMAND) because the process inherits the environment
import os
from prefect.runner.storage import GitRepository
token = os.getenv("GITHUB_TOKEN")
repo = GitRepository(
url="<https://github.com/owner/repo.git>",
credentials={"access_token": token} if token else None
)
Via deployment (Python SDK)
- This is the core pattern for 3.x (note: Deployment.build_from_flow() and prefect deployment build are removed in 3.x):
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret
repo = GitRepository(
url="<https://github.com/owner/repo.git>",
credentials={"access_token": Secret.load("github-token")},
branch="main",
)
flow.from_source(source=repo, entrypoint="flows/my_flow.py:my_flow") \
.deploy(
name="my-deployment",
work_pool_name="my-pool",
tags=["prod"],
parameters={"x": 1},
)
Notes and tips
- Token formatting is automatic per host:
- GitHub: plain token
- GitLab: oauth2: prefix for PATs; deploy tokens kept as username:token
- Bitbucket Cloud: x-token-auth: prefix
- Bitbucket Server: requires username + password/token
- For SSH, just ensure your worker image/VM has the key and agent configured. Use git@... URLs.
- For private submodules, turn on `include_submodules=True`; Prefect injects a git config rewrite so submodules clone with the same creds.
- Be careful not to commit credentials in code or YAML; use Secret or credentials blocks.
If you can share which host (GitHub/GitLab/Bitbucket), token type (PAT vs deploy token), and whether you use SSH or HTTPS, I can tailor the exact snippet.Hui Wu
03/31/2026, 11:20 AMMarvin
03/31/2026, 11:21 AMMarvin
03/31/2026, 11:23 AMfrom prefect import flow
from prefect.runner.storage import GitRepository
repo = GitRepository(
url="<https://github.com/owner/repo.git>",
branch="main",
# credentials={"access_token": "..."} # if private
)
f = flow.from_source(source=repo, entrypoint="flows/my_flow.py:my_flow")
2) fsspec-based remote filesystems (RemoteStorage)
- Any fsspec-supported protocol with the right extra installed (e.g., s3fs, gcsfs, adlfs).
- Common schemes: s3://, gs://, abfs:// (Azure ADLS/Blob), https://, sftp://, ftp://, hdfs://
- Example:
from prefect import flow
from prefect.runner.storage import RemoteStorage
storage = RemoteStorage(url="<s3://my-bucket/my-folder>")
f = flow.from_source(source=storage, entrypoint="flows/my_flow.py:my_flow")
3) Storage blocks (passed directly or via BlockStorageAdapter)
- Built-in blocks: RemoteFileSystem (generic fsspec), LocalFileSystem, SMB.
- Integration blocks (install first): S3Bucket (prefect-aws), GcsBucket (prefect-gcp), AzureBlobStorageContainer (prefect-azure).
- Example (S3 via RemoteFileSystem block):
from prefect import flow
from prefect.filesystems import RemoteFileSystem
s3_block = RemoteFileSystem.load("my-s3-storage") # basepath = <s3://bucket/prefix>
f = flow.from_source(source=s3_block, entrypoint="flows/my_flow.py:my_flow")
Notes
- You can also pass a string/Path and Prefect will auto-detect: Git URLs → GitRepository, fsspec URLs → RemoteStorage, local paths → LocalStorage.
- For credentials on cloud storage, use the filesystem’s standard fsspec extras or the provider’s block (S3Bucket/GcsBucket/Azure) for a smoother experience.
Helpful links
- How-to: Store flow code (Git, fsspec, blocks)
- API: prefect.runner.storage (GitRepository, RemoteStorage)
- API: prefect.filesystems (RemoteFileSystem, LocalFileSystem, SMB)
- prefect-aws (S3Bucket) | prefect-gcp (GcsBucket) | prefect-azure (Azure Blob)
If you tell me which backend you’re targeting (e.g., S3, GCS, Azure, GitHub), I can share a ready-to-run snippet with auth included.