<@ULVA73B9P> When pulling flow from source via git...
# ask-marvin
h
@Marvin When pulling flow from source via git how to config auth using Python SDK
m
thought for 351.7 seconds
Great question. In Prefect 3.x, when you pull a flow from a Git repo using the Python SDK, you configure auth on the storage object you pass to
flow.from_source(...)
. Use
GitRepository(credentials=...)
for HTTPS tokens or rely on your worker’s SSH setup for SSH URLs. Quick overview: - HTTPS + token: pass credentials (dict or block) to GitRepository - SSH: use
git@...
URL; ensure your worker has the SSH key configured - Submodules: set `include_submodules=True`; Prefect will auto-propagate creds - Environment vars: Prefect does not auto-read `GITHUB_TOKEN`/`GIT_TOKEN`; read them yourself (or use a Secret block) and pass to
credentials
Links - Docs: Store flow code (Git, from_source, git_clone) - Source: GitRepository (auth and submodules) - Integration blocks (GitHub, GitLab, Bitbucket) Supported parameters (Python SDK) -
GitRepository(url, credentials=None, branch=None, commit_sha=None, include_submodules=False, directories=None, pull_interval=60)
-
credentials
can be: - A simple dict such as
{"access_token": "<token>"}
(optionally with
username
for Bitbucket Server) - A Block that formats git credentials (e.g., GitHub/GitLab/Bitbucket credentials from integrations) - A Secret block used within the dict, e.g.,
{"access_token": Secret.load("...")}
Examples 1) HTTPS with GitHub PAT (recommended: Secret block)
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

# Save once (or create via UI)
# Secret(value="ghp_xxx").save("github-token", overwrite=True)

repo = GitRepository(
    url="<https://github.com/owner/repo.git>",
    credentials={"access_token": Secret.load("github-token")}
)

f = flow.from_source(source=repo, entrypoint="flows/my_flow.py:my_flow")
f.deploy(name="github-pat-deployment", work_pool_name="my-pool")
2) HTTPS with GitLab PAT or Deploy Token - GitLab PATs are automatically formatted as
oauth2:<token>
. - Deploy tokens in
username:token
form are used as-is (no
oauth2:
prefix).
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

# Personal Access Token (PAT)
repo_pat = GitRepository(
    url="<https://gitlab.com/group/project.git>",
    credentials={"access_token": "glpat_xxx"}
)

# Deploy Token (username:token)
repo_deploy = GitRepository(
    url="<https://gitlab.com/group/project.git>",
    credentials={"access_token": "deploy-user:deploy-token-xxx"}
)

flow.from_source(source=repo_pat, entrypoint="flows/main.py:my_flow") \
    .deploy(name="gitlab-pat", work_pool_name="my-pool")
3) SSH key auth (no SDK parameter; configure on the worker) - Use SSH URL:
git@github.com:owner/repo.git
- Ensure the worker runtime has the SSH private key and known_hosts configured (e.g., in
~/.ssh
, with ssh-agent)
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

repo = GitRepository(url="git@github.com:owner/private-repo.git")

flow.from_source(source=repo, entrypoint="flows/main.py:my_flow") \
    .deploy(name="ssh-deployment", work_pool_name="my-pool")
Notes: - Prefect does not accept an SSH key parameter. Git/SSH uses the process environment (e.g.,
SSH_AUTH_SOCK
,
GIT_SSH_COMMAND
) if present. 4) Private submodules - Prefect will automatically propagate credentials to submodules using a git config rewrite when
include_submodules=True
.
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

repo = GitRepository(
    url="<https://github.com/owner/parent-with-submodules.git>",
    credentials={"access_token": Secret.load("github-token")},
    include_submodules=True
)

flow.from_source(source=repo, entrypoint="flows/main.py:my_flow") \
    .deploy(name="with-submodules", work_pool_name="my-pool")
5) Using a provider credentials block (optional, nice DX) - GitHub example (requires installing the integration and creating/saving the block)
Copy code
# pip install prefect-github
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect_github import GitHubCredentials

# GitHubCredentials(token="ghp_xxx").save("my-github-creds", overwrite=True)
creds = GitHubCredentials.load("my-github-creds")

repo = GitRepository(
    url="<https://github.com/owner/repo.git>",
    credentials=creds,
)

flow.from_source(source=repo, entrypoint="flows/main.py:my_flow") \
    .deploy(name="github-creds-block", work_pool_name="my-pool")
Integration docs: https://docs.prefect.io/integrations/ Using environment variables - Prefect 3.x does not automatically read
GITHUB_TOKEN
,
GH_TOKEN
, or similar for Git auth. If you prefer env vars: - Read them in Python and pass to
credentials
- Or store them in a Prefect Secret block and reference that - Git itself will use standard env if present (e.g.,
GIT_ASKPASS
,
SSH_AUTH_SOCK
,
GIT_SSH_COMMAND
) because the process inherits the environment
Copy code
import os
from prefect.runner.storage import GitRepository

token = os.getenv("GITHUB_TOKEN")
repo = GitRepository(
    url="<https://github.com/owner/repo.git>",
    credentials={"access_token": token} if token else None
)
Via deployment (Python SDK) - This is the core pattern for 3.x (note:
Deployment.build_from_flow()
and
prefect deployment build
are removed in 3.x):
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

repo = GitRepository(
    url="<https://github.com/owner/repo.git>",
    credentials={"access_token": Secret.load("github-token")},
    branch="main",
)

flow.from_source(source=repo, entrypoint="flows/my_flow.py:my_flow") \
    .deploy(
        name="my-deployment",
        work_pool_name="my-pool",
        tags=["prod"],
        parameters={"x": 1},
    )
Notes and tips - Token formatting is automatic per host: - GitHub: plain token - GitLab:
oauth2:
prefix for PATs; deploy tokens kept as
username:token
- Bitbucket Cloud:
x-token-auth:
prefix - Bitbucket Server: requires username + password/token - For SSH, just ensure your worker image/VM has the key and agent configured. Use
git@...
URLs. - For private submodules, turn on `include_submodules=True`; Prefect injects a git config rewrite so submodules clone with the same creds. - Be careful not to commit credentials in code or YAML; use Secret or credentials blocks. If you can share which host (GitHub/GitLab/Bitbucket), token type (PAT vs deploy token), and whether you use SSH or HTTPS, I can tailor the exact snippet.
h
@Marvin what kinds of remote storeage does prefect support for flow from source?
m
thought for 139.7 seconds
In Prefect 3.x, Flow.from_source supports three main ways to pull your flow code from “remote” storage: 1) Git repositories (GitRepository) - Use HTTPS or SSH Git URLs (GitHub, GitLab, Bitbucket, self-hosted). - Options: branch, commit_sha, include_submodules, directories (sparse checkout), credentials. - Example:
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository

repo = GitRepository(
    url="<https://github.com/owner/repo.git>",
    branch="main",
    # credentials={"access_token": "..."}  # if private
)
f = flow.from_source(source=repo, entrypoint="flows/my_flow.py:my_flow")
2) fsspec-based remote filesystems (RemoteStorage) - Any fsspec-supported protocol with the right extra installed (e.g., s3fs, gcsfs, adlfs). - Common schemes: s3://, gs://, abfs:// (Azure ADLS/Blob), https://, sftp://, ftp://, hdfs:// - Example:
Copy code
from prefect import flow
from prefect.runner.storage import RemoteStorage

storage = RemoteStorage(url="<s3://my-bucket/my-folder>")
f = flow.from_source(source=storage, entrypoint="flows/my_flow.py:my_flow")
3) Storage blocks (passed directly or via BlockStorageAdapter) - Built-in blocks: RemoteFileSystem (generic fsspec), LocalFileSystem, SMB. - Integration blocks (install first): S3Bucket (prefect-aws), GcsBucket (prefect-gcp), AzureBlobStorageContainer (prefect-azure). - Example (S3 via RemoteFileSystem block):
Copy code
from prefect import flow
from prefect.filesystems import RemoteFileSystem

s3_block = RemoteFileSystem.load("my-s3-storage")  # basepath = <s3://bucket/prefix>
f = flow.from_source(source=s3_block, entrypoint="flows/my_flow.py:my_flow")
Notes - You can also pass a string/Path and Prefect will auto-detect: Git URLs → GitRepository, fsspec URLs → RemoteStorage, local paths → LocalStorage. - For credentials on cloud storage, use the filesystem’s standard fsspec extras or the provider’s block (S3Bucket/GcsBucket/Azure) for a smoother experience. Helpful links - How-to: Store flow code (Git, fsspec, blocks) - API: prefect.runner.storage (GitRepository, RemoteStorage) - API: prefect.filesystems (RemoteFileSystem, LocalFileSystem, SMB) - prefect-aws (S3Bucket) | prefect-gcp (GcsBucket) | prefect-azure (Azure Blob) If you tell me which backend you’re targeting (e.g., S3, GCS, Azure, GitHub), I can share a ready-to-run snippet with auth included.