Tushar Mahajan
10/03/2025, 1:00 PMMarvin
10/03/2025, 1:00 PMMarvin
10/03/2025, 1:02 PMpath/to/file.py:flow_func, work pool/work queue, schedule, parameters, and the code “source” info (e.g., Git URL + ref or image tag).
- No flow files are stored in the server DB; workers fetch code when a run starts.
How to “register” your deployments
- Programmatic (Prefect 3.x)
- Point a flow to your code source, then deploy:
from prefect import flow
@flow
def my_flow():
print("hello from k8s")
# Run this once from your laptop/CI to register the deployment
(
my_flow
.from_source(
source="<https://github.com/acme/data-pipelines>",
entrypoint="flows/etl.py:my_flow",
)
.deploy(
name="etl-prod",
work_pool_name="k8s-pool",
cron="0 * * * *",
)
)
- The server records a deployment pointing to your Git repo and entrypoint. When scheduled/triggered, the K8s worker clones the repo, imports my_flow, and runs it.
- CLI (Prefect 3.x)
- Initialize a project, commit prefect.yaml, and deploy from the repo root:
prefect deploy ./flows/etl.py:my_flow --name etl-prod
- The CLI uses prefect.yaml to define how the worker should pull code (e.g., clone a Git repo or use an image). If you don’t pass a pool, you’ll be prompted. See prefect deploy --help for options.
Kubernetes specifics with the Helm worker
- Git-based deployments: the worker’s job clones your repo into the pod at start. Provide credentials via blocks or environment variables as needed for private repos.
- Image-based deployments: set an image when deploying; the K8s job runs that image. This is the most hermetic option.
- Remote storage: configure your deployment’s storage (S3/GCS/Azure). The job will download the code before execution.
Docs and references
- How to store and fetch flow code: Store flow code
- Kubernetes worker/how it runs: Kubernetes deployments
- Workers (vs agents) overview: Upgrade to workers
If you share how you prefer to distribute code (Git vs image vs object storage, and whether your repo is private), I can give you a concrete prefect.yaml or deployment snippet tailored to your setup.Tushar Mahajan
10/03/2025, 1:04 PMMarvin
10/03/2025, 1:04 PMMarvin
10/03/2025, 1:05 PMTushar Mahajan
10/03/2025, 1:14 PMMarvin
10/03/2025, 1:15 PMMarvin
10/03/2025, 1:17 PMfrom prefect import flow
from prefect.storage import GitRepository # GitCredentials optional for private repos
@flow
def my_flow():
...
repo = GitRepository(
url="<https://github.com/acme/data-pipelines.git>",
branch="main", # or commit_sha="abc123"
# credentials=GitCredentials(...) # for private repos
)
(
my_flow
.from_source(source=repo, entrypoint="flows/etl.py:my_flow")
.deploy(
name="etl-prod",
work_pool_name="k8s-pool",
cron="0 * * * *",
)
)
2) CLI from a local clone of the Git repo (auto-detects Git)
- Run the command from inside your repo so Prefect can detect the origin URL and ref:
prefect deploy ./flows/etl.py:my_flow --name etl-prod --pool k8s-pool
- Prefect will record the repo URL and current branch/commit; your worker will clone that repo when the run starts.
Notes
- serve is great for local/ephemeral development. For durable runs on your K8s worker, use .deploy(...) or prefect deploy.
- For private repos, provide credentials via a GitCredentials block, or use standard Git auth in your worker pod (e.g., K8s secret + SSH key or HTTPS PAT). Avoid embedding tokens directly in URLs when possible.
Docs
- Store flow code in Git and run from workers: Store flow code
If you share whether your repo is public/private and which provider (GitHub/GitLab/Bitbucket) you’re using, I can show a concrete auth setup for your K8s worker.