Olve Heitmann
09/13/2025, 11:39 AMMarvin
09/13/2025, 11:39 AMMarvin
09/13/2025, 11:48 AMFlow.from_source(...) is:
- Save your credentials as a block (e.g., GitHubCredentials from the prefect-github integration)
- Load that block in code
- Pass it to Flow.from_source(...) via a GitRepository storage object
Here’s how to do it end-to-end.
Assumptions:
- You’re on Prefect 3.x
- Your server (or workers) can reach your Prefect API and the GitHub repo
- The environment where this runs has the prefect-github extra installed
1) Install the integration and register the block types
- Make sure the runtime environment has the GitHub integration
uv run --with 'prefect[github]' python -c "import prefect_github; print('ok')"
- Register the integration’s block types (so you can create them in the UI or via code)
uv run --with 'prefect[github]' prefect block register -m prefect_github
Links:
- prefect-github docs
- prefect.yaml + steps guide
2) Create the GitHubCredentials block (if you haven’t yet)
You can create it in the UI (via prefect block create github-credentials) or programmatically:
from prefect_github import GitHubCredentials
# Replace with your token; store as a Secret in practice
GitHubCredentials(token="ghp_xxx").save("my-gh-creds", overwrite=True)
3) Retrieve the block on your server and use it with Flow.from_source(...)
Load the credentials block and pass it to a GitRepository, then call `Flow.from_source`:
from prefect import Flow
from prefect.storage import GitRepository
from prefect_github import GitHubCredentials
# Load the saved block from your Prefect server workspace
gh_creds = GitHubCredentials.load("my-gh-creds")
# Point to your repo; use https URL even for private repos
repo = GitRepository(
url="<https://github.com/your-org/your-repo.git>",
credentials=gh_creds, # this uses the stored credential
# optionally pin a branch or commit
# branch="main",
# commit_sha="abcdef1234567890",
)
# Build the flow object from source
flow = Flow.from_source(
source=repo,
entrypoint="flows/my_flow.py:my_flow", # path to file : function name
)
4) Optional: Use your block in prefect.yaml (CLI deploy)
If you deploy with prefect deploy, you can reference the same block in pull steps:
pull:
- prefect.deployments.steps.pull:git_clone:
repository: <https://github.com/your-org/your-repo.git>
credentials: "{{ prefect.blocks.github-credentials.my-gh-creds }}"
- prefect.deployments.steps.pull:set_working_directory:
directory: your-repo
5) Verify the block exists on your server
- List configured blocks
prefect block ls
- Inspect the specific block
prefect block inspect "github-credentials/my-gh-creds"
Notes and gotchas:
- Avoid the old “GitHub storage blocks” pattern from 2.x; in 3.x use Flow.from_source(...) with a GitRepository.
- Ensure the environment running this code has prefect-github installed, otherwise loading the block will fail.
- You can also use prefect.storage.GitCredentials if you prefer a more generic Git credential object, but GitHubCredentials is the easiest for PATs.
If you can share:
- Your block name
- Whether you’re using HTTPS with a PAT or SSH
- How you’re running (local process worker, Docker, K8s)
…I can tailor the snippet exactly to your setup.Olve Heitmann
09/13/2025, 1:08 PMMarvin
09/13/2025, 1:09 PMMarvin
09/13/2025, 1:09 PMnn_etl_get_stocks_prefect), not the flow’s name= value (e.g., nn-etl-get-stocks-prefect)
So if your file contains:
from prefect import flow
@flow(name="nn-etl-get-stocks-prefect")
def nn_etl_get_stocks_prefect(db_type: Literal['sqlite', 'duckdb'] = 'duckdb'):
...
Then your Flow.from_source(...) should look like:
from prefect import Flow
from prefect.storage import GitRepository
# optional, if private repo:
# from prefect_github import GitHubCredentials
# gh = GitHubCredentials.load("my-gh-creds")
repo = GitRepository(
url="<https://github.com/your-org/git-qfin-repo.git>",
# credentials=gh,
)
flow = Flow.from_source(
source=repo,
entrypoint="brrr/data_collection/nordnet/prefect_deploys/nn_get_stocks_prefect.py:nn_etl_get_stocks_prefect"
)
Common pitfalls to double-check:
- The entrypoint after the colon must be nn_etl_get_stocks_prefect (the Python function name), not nn-etl-get-stocks-prefect (the flow’s display name).
- The file path is relative to the repo root and matches the cloned structure.
- The function is defined at module scope (not inside if __name__ == '__main__':).
- Your latest commit with that function is the one being pulled (branch/commit pinning, if any).
- In your decorator, use name=... (no underscores) and normal Python parameter names. I’m assuming the underscores you showed were just for formatting in Slack.
If you’re deploying with prefect.yaml, the same entrypoint format applies there. See: prefect.yaml guide.
If this doesn’t solve it, can you share the exact entrypoint= string you’re using and confirm the relative path to the file in the repo?