<@ULVA73B9P> what is a good convention to create d...
# ask-marvin
v
@Marvin what is a good convention to create dry_runs and/or to separate dev vs prod environment?
m
thought for 201.4 seconds
Great question. Here’s a simple, proven convention for Prefect 3.x that covers both “dev vs prod” separation and supporting dry runs. Core conventions - Separate environments at the control plane - Prefect Cloud: use separate workspaces (e.g., dev, staging, prod). Same block names can exist per workspace with different values. - Open-source: use profiles to point to different API URLs and credentials, and separate work pools per env. - Links: Workspaces, Settings &amp; profiles, Work pools - Separate infrastructure per env - Different work pools (e.g., dev = process/docker, prod = kubernetes) and optionally different work queues. - Same code, multiple deployments - Create one deployment per environment (and optionally per branch). Give them clear names and tags. - Use different schedules and parameters per deployment (e.g., dry_run default true in dev, false in prod). - Config belongs to the control plane - Use Blocks for credentials/resources and Variables for tunables. Keep the block/variable names the same across workspaces so code doesn’t change. - Links: Blocks, Variables Dry-run pattern (there’s no built-in global dry-run) - Add a
dry_run: bool
flow parameter and gate any side-effect operations behind it. - Return a preview of “what would happen” when
dry_run=True
. - Default
dry_run=True
in dev deployments;
False
in prod. Minimal example
Copy code
from prefect import flow

def write_to_db(records):
    # real write here
    ...

def preview_write(records):
    print(f"DRY RUN: would write {len(records)} records; sample: {records[:3]}")

@flow(log_prints=True)
def etl(env: str = "dev", dry_run: bool = True, batch_size: int = 100):
    print(f"env={env} dry_run={dry_run} batch_size={batch_size}")
    records = list(range(batch_size))  # imagine we transformed data here

    if dry_run:
        preview_write(records)
        return {"status": "dry_run", "would_write": len(records)}

    write_to_db(records)
    return {"status": "success", "written": len(records)}
Two deployments of the same flow
Copy code
# Point to code from a branch/commit for prod; local path or feature branch for dev
flow_dev = etl.from_source(
    source=".", 
    entrypoint="flows/etl.py:etl",
)
flow_dev.deploy(
    name="etl-dev",
    work_pool_name="dev-pool",
    parameters={"env": "dev", "dry_run": True, "batch_size": 50},
    tags=["dev"],
    # schedules=[...]  # optional; often manual in dev
)

flow_prod = etl.from_source(
    source="<https://github.com/myorg/myrepo>",
    entrypoint="flows/etl.py:etl",
)
flow_prod.deploy(
    name="etl-prod",
    work_pool_name="prod-k8s",
    parameters={"env": "prod", "dry_run": False, "batch_size": 1000},
    tags=["prod", "critical"],
    # schedules=[Cron(...)]  # prod schedule here
)
Notes - Use
flow.from_source(...).deploy(...)
for 3.x (do not use deprecated 2.x build commands). - Prefer main branch/commit pin for prod; feature branch/local path for dev. - Override at run time when needed: - CLI:
prefect deployment run etl/etl-prod -p dry_run=true
(use sparingly in prod) - Docs: Flow parameters, Deployments, Store flow code, Schedules Recommended defaults - Dev - Workspace: dev - Work pool: dev-process or dev-docker - Tags: dev - Parameters: dry_run=true, smaller batch sizes - Schedules: manual or frequent, but low-impact - Prod - Workspace: prod - Work pool: prod-kubernetes (or your production infra) - Tags: prod, critical
- Parameters: dry_run=false, tuned batch sizes - Schedules: business cadence - Concurrency limits or triggers as needed Would you like me to adapt this pattern to your setup? If you can share: - Cloud or OSS? - Infra (process, Docker, Kubernetes)? - How you store code (GitHub repo/branch/commit)? - Any specific services you want to “preview” in dry runs (e.g., DB writes, API calls)?