<@ULVA73B9P> How can I retrieve the github Block on my server for use in flow.from_source(...) so th...
o
@Marvin How can I retrieve the github Block on my server for use in flow.from_source(...) so that I used the stored credential
m
thought for 579.3 seconds
Great question. In Prefect 3.x, the way to use stored GitHub credentials with
Flow.from_source(...)
is: - Save your credentials as a block (e.g.,
GitHubCredentials
from the
prefect-github
integration) - Load that block in code - Pass it to
Flow.from_source(...)
via a
GitRepository
storage object Here’s how to do it end-to-end. Assumptions: - You’re on Prefect 3.x - Your server (or workers) can reach your Prefect API and the GitHub repo - The environment where this runs has the
prefect-github
extra installed 1) Install the integration and register the block types - Make sure the runtime environment has the GitHub integration
Copy code
uv run --with 'prefect[github]' python -c "import prefect_github; print('ok')"
- Register the integration’s block types (so you can create them in the UI or via code)
Copy code
uv run --with 'prefect[github]' prefect block register -m prefect_github
Links: - prefect-github docs - prefect.yaml + steps guide 2) Create the GitHubCredentials block (if you haven’t yet) You can create it in the UI (via
prefect block create github-credentials
) or programmatically:
Copy code
from prefect_github import GitHubCredentials

# Replace with your token; store as a Secret in practice
GitHubCredentials(token="ghp_xxx").save("my-gh-creds", overwrite=True)
3) Retrieve the block on your server and use it with Flow.from_source(...) Load the credentials block and pass it to a GitRepository, then call `Flow.from_source`:
Copy code
from prefect import Flow
from prefect.storage import GitRepository
from prefect_github import GitHubCredentials

# Load the saved block from your Prefect server workspace
gh_creds = GitHubCredentials.load("my-gh-creds")

# Point to your repo; use https URL even for private repos
repo = GitRepository(
    url="<https://github.com/your-org/your-repo.git>",
    credentials=gh_creds,  # this uses the stored credential
    # optionally pin a branch or commit
    # branch="main",
    # commit_sha="abcdef1234567890",
)

# Build the flow object from source
flow = Flow.from_source(
    source=repo,
    entrypoint="flows/my_flow.py:my_flow",  # path to file : function name
)
4) Optional: Use your block in prefect.yaml (CLI deploy) If you deploy with
prefect deploy
, you can reference the same block in pull steps:
Copy code
pull:
  - prefect.deployments.steps.pull:git_clone:
      repository: <https://github.com/your-org/your-repo.git>
      credentials: "{{ prefect.blocks.github-credentials.my-gh-creds }}"
  - prefect.deployments.steps.pull:set_working_directory:
      directory: your-repo
5) Verify the block exists on your server - List configured blocks
Copy code
prefect block ls
- Inspect the specific block
Copy code
prefect block inspect "github-credentials/my-gh-creds"
Notes and gotchas: - Avoid the old “GitHub storage blocks” pattern from 2.x; in 3.x use
Flow.from_source(...)
with a
GitRepository
. - Ensure the environment running this code has
prefect-github
installed, otherwise loading the block will fail. - You can also use
prefect.storage.GitCredentials
if you prefer a more generic Git credential object, but
GitHubCredentials
is the easiest for PATs. If you can share: - Your block name - Whether you’re using HTTPS with a PAT or SSH - How you’re running (local process worker, Docker, K8s) …I can tailor the snippet exactly to your setup.
o
@Marvin I am trying to deploy from git. My python file has @flow(name='nn-etl-get-stocks-prefect') def nn_etl_get_stocks_prefect(_db_type_: Literal['sqlite', 'duckdb'] = 'duckdb'): when i run flow.from_source() i get the following error prefect.exceptions.MissingFlowError: Flow function with name 'nn-etl-get-stocks-prefect' not found in '/tmp/tmppwzpal62/github-repository-git-qfin-repo/brrr/data_collection/nordnet/prefect_deploys/nn_get_stocks_prefect.py'. It exists in both my local dir and latest commit on git. The "git-qfin-repo" is block directly to the repo I am using. Suggestions?
m
thought for 45.1 seconds
That error usually happens when the entrypoint after the colon is the flow’s “display name” (with hyphens) instead of the Python function name. Entrypoint rules in Prefect 3.x: - The part before the colon is the path to your file relative to the repo root - The part after the colon must be the Python identifier of the function (e.g.,
nn_etl_get_stocks_prefect
), not the flow’s
name=
value (e.g.,
nn-etl-get-stocks-prefect
) So if your file contains:
Copy code
from prefect import flow

@flow(name="nn-etl-get-stocks-prefect")
def nn_etl_get_stocks_prefect(db_type: Literal['sqlite', 'duckdb'] = 'duckdb'):
    ...
Then your
Flow.from_source(...)
should look like:
Copy code
from prefect import Flow
from prefect.storage import GitRepository
# optional, if private repo:
# from prefect_github import GitHubCredentials
# gh = GitHubCredentials.load("my-gh-creds")

repo = GitRepository(
    url="<https://github.com/your-org/git-qfin-repo.git>",
    # credentials=gh,
)

flow = Flow.from_source(
    source=repo,
    entrypoint="brrr/data_collection/nordnet/prefect_deploys/nn_get_stocks_prefect.py:nn_etl_get_stocks_prefect"
)
Common pitfalls to double-check: - The entrypoint after the colon must be
nn_etl_get_stocks_prefect
(the Python function name), not
nn-etl-get-stocks-prefect
(the flow’s display name). - The file path is relative to the repo root and matches the cloned structure. - The function is defined at module scope (not inside
if __name__ == '__main__':
). - Your latest commit with that function is the one being pulled (branch/commit pinning, if any). - In your decorator, use
name=...
(no underscores) and normal Python parameter names. I’m assuming the underscores you showed were just for formatting in Slack. If you’re deploying with prefect.yaml, the same entrypoint format applies there. See: prefect.yaml guide. If this doesn’t solve it, can you share the exact
entrypoint=
string you’re using and confirm the relative path to the file in the repo?