Dale McGeorge
09/24/2025, 2:07 AMMarvin
09/24/2025, 2:07 AMMarvin
09/24/2025, 2:15 AMprefect profile ls
- prefect profile create dev
- prefect profile use dev
- In Cloud, switch workspace:
- prefect cloud workspace ls
- prefect cloud workspace set
- Create work pools and queues:
- prefect work-pool create "k8s-dev" --type kubernetes
- prefect work-queue create --work-pool "k8s-dev" dev-default
- Set concurrency:
- prefect work-pool set-concurrency-limit "k8s-prod" 5
- prefect work-queue set-concurrency-limit dev-default 3
- Manage variables:
- prefect variable set DB_URL "postgres://..."
Example: three deployments from one repo, one per environment
from prefect.flows import Flow
repo = "<https://github.com/acme/data-pipelines>"
entrypoint = "flows/etl.py:etl" # path in your repo
# Dev
Flow.from_source(repo, entrypoint).deploy(
name="etl-dev",
work_pool_name="k8s-dev",
parameters={"env": "dev"},
tags=["dev"],
cron="*/10 * * * *", # every 10 minutes (example)
)
# Staging
Flow.from_source(repo, entrypoint).deploy(
name="etl-staging",
work_pool_name="k8s-staging",
parameters={"env": "staging"},
tags=["staging"],
paused=True, # optional: require manual unpause for safety
)
# Prod
Flow.from_source(repo, entrypoint).deploy(
name="etl-prod",
work_pool_name="k8s-prod",
parameters={"env": "prod"},
tags=["prod"],
cron="0 * * * *", # hourly
concurrency_limit=1, # ensure only one prod run at a time
)
Where to put configuration/secrets
- Prefect Variables: simple key/value for non-secret values like RUNS_PER_BATCH
or TARGET_SCHEMA
.
- Blocks (e.g., Secret, JSON, storage, notifications): for secrets and structured config. In Cloud, use distinct blocks per workspace; in shared environments, adopt a clear naming pattern, e.g., aws-credentials-prod
, aws-credentials-dev
.
CI/CD promotion pattern
- Build once, deploy many: build your image from main, then deploy to dev/staging/prod with different work_pool_name
and parameters. Tie this to your Git branches/tags.
- Use separate API keys and profiles (or Cloud workspaces) for each environment’s deployment job.Marvin
09/24/2025, 2:15 AM["dev"]
, ["staging"]
, ["prod"]
to help search, filter, and visualize runs by environment.
- Automations/alerts: keep notifications scoped per environment (workspace-scoped in Cloud; use separate blocks or channels otherwise).
A couple of questions to tailor this to you:
- Are you on Prefect Cloud or self-hosting?
- What execution infrastructure are you using (process, Docker, Kubernetes, ECS, etc.)?
- Do you want hard isolation (prod cannot see dev assets), or soft isolation within a shared control plane?
If you share those, I can give you exact commands and a minimal template for your setup.