<@ULVA73B9P> What is the best way to separate envi...
# ask-marvin
d
@Marvin What is the best way to separate environments, i..e dev, staging and prod
m
thought for 482.2 seconds
Great question. In Prefect 3.x, the cleanest way to separate dev/staging/prod is to isolate orchestration, infrastructure, and configuration per environment. Here’s a battle‑tested setup you can use, plus how to operate it day-to-day. High-level recommendations - Prefect Cloud: use a separate Workspace per environment (e.g., acme/dev, acme/staging, acme/prod). Variables, blocks, deployments, automations, and RBAC are all workspace-scoped, so this gives you hard isolation by default. - Self-hosted (Prefect Server): for strict isolation, run separate Server instances (or at least separate databases) per environment. If you must share a single server, use separate work pools, variables, blocks, and strong naming conventions. - One work pool per environment: e.g., k8s-dev, k8s-staging, k8s-prod (or docker-, process-). Run workers only for their pool. - Separate configuration per environment: keep variables and blocks environment-specific. In Cloud this happens naturally with workspace scoping; otherwise, use namespacing conventions like db-url-dev, db-url-prod. - Deploy separately per environment: the same code is deployed multiple times with env-specific work_pool_name, parameters, schedules, and tags. - Concurrency and safety: use per-pool or per-queue concurrency limits; default deployments to paused in staging/prod; use service accounts and least-privilege API keys per environment. Environment primitives to use - Workspaces (Cloud): strict isolation boundary for all objects and RBAC. - Work pools and queues: route runs per environment and set concurrency limits. - Variables and blocks: store env-specific configuration and secrets per environment. - Profiles and Cloud workspace switching: quickly switch CLI context per environment. Day-to-day CLI you’ll use - Switch Prefect profile (local dev, or between server/cloud contexts): -
prefect profile ls
-
prefect profile create dev
-
prefect profile use dev
- In Cloud, switch workspace: -
prefect cloud workspace ls
-
prefect cloud workspace set
- Create work pools and queues: -
prefect work-pool create "k8s-dev" --type kubernetes
-
prefect work-queue create --work-pool "k8s-dev" dev-default
- Set concurrency: -
prefect work-pool set-concurrency-limit "k8s-prod" 5
-
prefect work-queue set-concurrency-limit dev-default 3
- Manage variables: -
prefect variable set DB_URL "postgres://..."
Example: three deployments from one repo, one per environment
Copy code
from prefect.flows import Flow

repo = "<https://github.com/acme/data-pipelines>"
entrypoint = "flows/etl.py:etl"  # path in your repo

# Dev
Flow.from_source(repo, entrypoint).deploy(
    name="etl-dev",
    work_pool_name="k8s-dev",
    parameters={"env": "dev"},
    tags=["dev"],
    cron="*/10 * * * *",  # every 10 minutes (example)
)

# Staging
Flow.from_source(repo, entrypoint).deploy(
    name="etl-staging",
    work_pool_name="k8s-staging",
    parameters={"env": "staging"},
    tags=["staging"],
    paused=True,  # optional: require manual unpause for safety
)

# Prod
Flow.from_source(repo, entrypoint).deploy(
    name="etl-prod",
    work_pool_name="k8s-prod",
    parameters={"env": "prod"},
    tags=["prod"],
    cron="0 * * * *",  # hourly
    concurrency_limit=1,  # ensure only one prod run at a time
)
Where to put configuration/secrets - Prefect Variables: simple key/value for non-secret values like
RUNS_PER_BATCH
or
TARGET_SCHEMA
. - Blocks (e.g., Secret, JSON, storage, notifications): for secrets and structured config. In Cloud, use distinct blocks per workspace; in shared environments, adopt a clear naming pattern, e.g.,
aws-credentials-prod
,
aws-credentials-dev
. CI/CD promotion pattern - Build once, deploy many: build your image from main, then deploy to dev/staging/prod with different
work_pool_name
and parameters. Tie this to your Git branches/tags. - Use separate API keys and profiles (or Cloud workspaces) for each environment’s deployment job.
- Optionally require approvals to unpause or promote prod deployments. Guardrails and observability - Concurrency limits: at the deployment (as shown), work queue, or work pool level. - Tags: add
["dev"]
,
["staging"]
,
["prod"]
to help search, filter, and visualize runs by environment. - Automations/alerts: keep notifications scoped per environment (workspace-scoped in Cloud; use separate blocks or channels otherwise). A couple of questions to tailor this to you: - Are you on Prefect Cloud or self-hosting? - What execution infrastructure are you using (process, Docker, Kubernetes, ECS, etc.)? - Do you want hard isolation (prod cannot see dev assets), or soft isolation within a shared control plane? If you share those, I can give you exact commands and a minimal template for your setup.