Sơn Lê
11/07/2025, 10:33 AMMarvin
11/07/2025, 10:33 AMMarvin
11/07/2025, 10:38 AMfrom prefect import flow
from prefect.runner.storage import GitRepository
# If the repo is private, also import GitCredentials
etl = flow.from_source(
source=GitRepository(
url="<https://github.com/acme/data-flows.git>",
# credentials=GitCredentials(username="...", access_token="..."),
branch="main",
),
entrypoint="flows/etl.py:etl",
)
etl.deploy(
name="etl-prod",
work_pool_name="k8s-pool",
image="<http://ghcr.io/acme/etl:1.2.3|ghcr.io/acme/etl:1.2.3>",
job_variables={
"namespace": "prefect",
"image_pull_policy": "IfNotPresent",
# add other k8s vars as needed: env, service_account_name, image_pull_secrets, etc.
},
# For prebuilt images: build=False, push=False
)
Docs:
- Flow code storage with from_source
- Job variables (per-deployment K8s settings)
- Kubernetes work pools
3) Have your scheduler only trigger deployments
- Your scheduler receives events and calls `run_deployment`—no need to import flow code or install its libs.
from prefect.deployments import run_deployment
# Trigger an existing deployment by its "flow-name/deployment-name"
run = run_deployment(
name="etl/etl-prod",
parameters={"date": "2025-11-07"},
# Optional: override the image at trigger-time if you must
# job_variables={"image": "<http://ghcr.io/acme/etl:1.2.4|ghcr.io/acme/etl:1.2.4>"}
)
- Alternatively, use the Prefect REST API or client to create a flow run from a deployment (still no imports in your scheduler).
Docs: Run deployments programmatically
API ref: Create flow run from deployment (REST)
Notes and gotchas
- Avoid removed 2.x patterns in 3.x: don’t use Deployment.build_from_flow() or `prefect deployment build`; use flow.from_source(...).deploy(...) or prefect deploy.
- If you must create deployments dynamically on events, you can still do so from your scheduler as long as you:
- Reference flows via from_source(...) (so you don’t import code locally)
- Set image=... (and, if the image is prebuilt, build=False, push=False)
- For private repos, use GitCredentials in GitRepository.
Questions to tailor the best approach for you:
- Do you want to create deployments on-the-fly per event, or just trigger pre-created deployments?
- Where is your flow code hosted (GitHub/GitLab, private repo)?
- Do you already have per-flow Docker images, or should we outline a simple CI step to build/push them?