Short answer: yes—this pattern works well for a small team on Prefect Server, and it’s a common/solid way to get branch‑isolated testing with clean promotion to prod. A couple of ways to simplify it and some limitations/gotchas to watch for: What looks great - Per-branch dev deployments: Using a deployment per PR branch that points to the branch’s Git ref is a clean way to test in isolation. - Separate work pools for dev/prod: Good isolation, easy to swap environment settings, and avoids resource contention. - Using environment variables to toggle behavior: Works well; just be careful with secrets (see below). - Deleting dev deployments after merge: Keeps the UI tidy. Suggested simplifications - Avoid “diff only the changed flow files.” It’s easy to miss changes in shared modules. For small teams, it’s simpler and safer to either: - Redeploy all flows when a PR opens/updates, or - Create dev deployments on PR open pointing to the branch (not a commit), then you don’t need to redeploy on every push—the worker will pick up the latest commit on that branch at run time. - Prefer prefect.yaml + prefect deploy over ad-hoc scripting. Put your pull steps, work pool, and job variables in prefect.yaml and let

prefect deploy

wire it up. It’s repeatable and easier for the team to maintain. - Disable scheduling on dev deployments. Keep dev deployments manual-only or paused by default to prevent accidental test runs. Common limitations/gotchas - Secrets handling - Don’t put long-lived credentials in work pool env vars. Use Prefect Blocks (e.g., Secret, GitCredentials) and reference them in your deployment or pull steps. See Store secrets. - Git access from workers - If you’re deploying from a private repo, the worker must be able to clone. Use a GitCredentials block or a deploy key/machine user. For Git storage, prefer

Flow.from_source(..., GitRepository(...))

. - Branch name quirks - Deployment names can break on characters like

/

. Slugify branch names in CI and use that in deployment names. - Dependency management - If you’re using Docker/Kubernetes workers, ensure the image includes your Python deps. Decide whether to: - Use a pinned base image for all branches, or - Build a branch-specific image (heavier but reproducible). If you keep it simple, pin a base image and rely on dependency compatibility. - Schedules and clutter - Don’t attach schedules to dev deployments; otherwise you can get surprise runs. Clean them up on PR close with

prefect deployment delete

. - Prefect Server vs Cloud - Your approach works the same on Server. Cloud has a richer UI for deployment versioning/rollback; Server won’t give you the same rollback UX. Concrete patterns (Prefect 3.x) - Use Flow.from_source + GitRepository for precise refs. In CI, set the branch or commit you want to test. Then call deploy. - Key note: In Prefect 3.x, use

Flow.from_source(...).deploy(...)

or

prefect deploy

. Do not use the old 2.x

Deployment.build_from_flow()

or

prefect deployment build

. prefect.yaml skeleton with branch-aware dev/prod

# prefect.yaml
name: my-project
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/your-org/your-repo.git>
      # For dev, pass BRANCH_NAME in CI; for prod, use "main"
      branch: ${BRANCH_NAME}
deployments:
  - name: my-flow-dev-${BRANCH_SLUG}
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: dev
    job_variables:
      env:
        ENV: dev
    schedule: null  # manual-only for dev
  - name: my-flow-prod
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: prod
    job_variables:
      env:
        ENV: prod

GitHub Actions sketch ``` # On PR open or sync: - name: Set vars run: | echo "BRANCH_NAME=${{ github.head_ref }}" >> $GITHUB_ENV

echo "BRANCH_SLUG=$(echo "${{ github.head_ref }}" | tr '/._' '-' | tr -cd '[alnum]-' | tr '[upper]' '[lower]')" >> $GITHUB_ENV - name: Prefect auth run: | prefect config set PREFECT_API_URL="${{ secrets.PREFECT_API_URL }}" prefect auth login --key "${{ secrets.PREFECT_API_KEY }}" - name: Create/Update dev deployment(s) run: | prefect deploy --name "my-flow-dev-${BRANCH_SLUG}" --no-prompt

Cleanup on PR close

- name: Delete dev deployment if: github.event.action == 'closed' run: | prefect deployment delete "my-flow-dev-${{ env.BRANCH_SLUG }}" --yes ``` CLI commands verified -

prefect deploy

exists and is the recommended 3.x command (help:

prefect deploy --help

). -

prefect deployment delete

exists for cleanup (help:

prefect deployment --help

). Links for reference - Deploy from source and CI/CD: CI/CD with Prefect - prefect.yaml structure: prefect.yaml - Customize job variables (env, image, resources): Customize job variables - Runner storage (Git) and from_source: Deployments overview A couple of questions to tailor this further: - Which worker type are you using for dev/prod (process, Docker, Kubernetes)? That affects how we set job variables and where to put dependencies. - Do you need tests to run automatically on every PR push, or is manual triggering in the UI sufficient for dev? If manual is fine, the “deploy on PR open with branch” approach is the simplest.

Great — with manual triggering and process workers, you can simplify a lot. Here’s a detailed “diff” between your plan and a simpler path, plus why and how to implement each change. High-level changes from your proposal 1) Selecting “changed” flows in CI - Your approach: Diff PR vs main, deploy only flows whose files changed. - Simpler option: Skip the diff logic. For small teams, the cognitive load and edge cases (shared modules, utils, config files) can cause under-deploys. - Option A (branch pointer): Create per-PR dev deployments that point to the branch, not a commit. Workers will pull the latest commit at run time; you do not need to redeploy on every push. - Option B (always redeploy all flows on PR events): Simple and safe, usually fine for a small set of flows. Trade-off: - Branch pointer: Easiest, but runs use the latest commit on the branch at execution time (not fully reproducible). - Commit pin (what you proposed): Fully reproducible but requires redeploy for each push. If you keep manual testing, it’s OK to “lose” reproducibility and enjoy the simpler branch-pointer approach. 2) How to define deployments - Your approach: CI scripts build per-branch deployments programmatically. - Simpler option: Use

prefect.yaml

and

prefect deploy

. Centralize deploy config (pull steps, work pool, job variables) and let the CLI manage it. This reduces CI logic and keeps conventions in one place. 3) Scheduling of dev deployments - Your approach: Not explicitly stated. - Simpler option: Keep dev deployments unscheduled (manual-only). This prevents accidental test runs and simplifies cleanup. Add schedules only to prod deployments. 4) Environment configuration and secrets - Your approach: Work pool env vars like `ENV=dev/prod`; flows read these to switch resources. - Keep this, with two tweaks: - Use

job_variables.env

at the deployment level (process worker) for per-deployment env. It’s closer to the code that needs it and easier to reason about than pool-wide env for dev/prod differences. - Use Prefect Blocks for secrets instead of work pool env variables. Store test/prod credentials in Secret blocks and read them in your flows. This avoids leaking secrets in pool env and keeps rotation easy. - Docs: Store secrets 5) Naming per-branch deployments - Your approach:

flowname-dev-<branch-name>

- Keep this, but slugify branch names to avoid

/

and other characters breaking names. Generate

BRANCH_SLUG

in CI and use it in deployment names. 6) Cleanup after merge - Your approach: Optionally delete per-branch dev deployments. - Keep this. Use the CLI to delete, which is available in 3.x. -

prefect deployment delete "my-flow-dev-<branch-slug>" --yes

Process-worker-specific guidance - Dependencies: Process workers run your flows in the Python environment on that machine. For dev/prod parity: - Easiest: Pre-provision the worker’s environment with your repo’s dependencies and update it as needed when requirements change. - If you must install at runtime, add a pull step to run a script like

pip install -r requirements.txt

. This mutates the worker’s environment and can be slow; for small teams this can be acceptable for dev but I’d avoid it in prod. - Job variables: For process workers,

job_variables

map directly to ProcessJobConfiguration (env, command, working_dir, labels, stream_output). You’ll mostly use

env

. - Signature reference: ProcessJobConfiguration supports

env

,

command

,

working_dir

, etc. Concrete example: prefect.yaml for process workers - Dev runs will point to a branch; prod runs will point to main. - Dev deployment is manual-only; prod can be scheduled. - You can keep a single flow here for clarity; scale to multiple flows by adding more entries. ``` name: my-project # Pull code at run time (worker clones the repo) pull: - prefect.deployments.steps.git_clone:

repository: https://github.com/your-org/your-repo.git branch: ${BRANCH_NAME} # In prod CI, set BRANCH_NAME=main; in PR CI, set to the PR branch deployments: - name: my-flow-dev # CLI will override with a branch-specific name entrypoint: flows/my_flow.py:my_flow work_pool: name: dev schedule: null # manual-only job_variables: env: ENV: dev # Example: surface a non-secret switch. For secrets, use Prefect Blocks in code. - name: my-flow-prod entrypoint: flows/my_flow.py:my_flow work_pool: name: prod # Add a schedule if you want this to run automatically # schedule: # cron: "0 * * * *" job_variables: env: ENV: prod

GitHub Actions sketch for PR open/sync
- Simplifies your plan: no diff-detection, deploys a branch-pointer dev deployment once per PR event.
- Manual testing: trigger via UI or CLI: `prefect deployment run "my-flow-dev-<branch>"`

# PR open/synchronize - name: Slugify branch run: | echo "BRANCH_NAME=${{ github.head_ref }}" >> $GITHUB_ENV echo "BRANCH_SLUG=$(echo "${{ github.head_ref }}" | tr '/._' '-' | tr -cd '[alnum]-' | tr '[upper]' '[lower]')" >> $GITHUB_ENV - name: Prefect auth run: | prefect config set PREFECT_API_URL="${{ secrets.PREFECT_API_URL }}" prefect auth login --key "${{ secrets.PREFECT_API_KEY }}" - name: Create/Update dev deployment # Uses prefect.yaml; overrides name so each PR gets its own dev deployment env: BRANCH_NAME: ${{ env.BRANCH_NAME }} run: | prefect deploy --name "my-flow-dev-${BRANCH_SLUG}" --no-prompt

Cleanup on PR close/merge

- name: Delete dev deployment if: github.event.action == 'closed' run: | prefect deployment delete "my-flow-dev-${{ env.BRANCH_SLUG }}" --yes ``` Triggering runs manually - From the UI: open the deployment and click “Run”. - From CLI (3.x):

prefect deployment run "my-flow-dev-<branch-slug>"

Why this is simpler and where it differs - No change-detection logic: You avoid a class of mistakes where a “shared module” change doesn’t redeploy a flow. For small teams, the cost of redeploying is minimal; the reliability gain is big. - Branch pointer vs commit pin: You proposed pinning to commit. That’s reproducible but adds CI churn (redeploy for every push). With manual testing, pointing to the branch cuts CI steps and complexity. If reproducibility is critical, go back to commit pins. - Config in prefect.yaml: Puts the “what and how” of deployments under version control with the flows, so your CI becomes a thin wrapper that sets branch and name. - Process worker alignment: By using

job_variables.env

per deployment, you keep dev/prod differences close to the deployment instead of the pool, and you don’t need custom worker wrappers. Potential limitations to keep in mind - Private repos: Ensure your workers can clone the repo. Use a Git credentials block or machine user. See CI/CD guide: CI/CD with Prefect. - Secrets: Keep credentials in Prefect Secret blocks (not pool or repo env). Load them in your flows as needed. Store secrets - Dev/prod dependency drift: Since you’re on process workers, make sure both dev and prod worker environments have matching dependencies. For dev, a “pip install at run time” step is acceptable but slower; for prod, pre-provisioning is better. If you want, I can: - Draft a complete

prefect.yaml

covering multiple flows. - Provide a ready-to-paste GitHub Actions workflow file for PR open/close and for prod on merge to main.

Absolutely—here’s a simple, end-to-end walkthrough for developers and reviewers, with what happens behind the scenes at each stage. This assumes: - Process workers - Manual triggering - Per-PR dev deployments that point to the branch (not a commit) - Deployments managed via prefect.yaml and created by CI on PR open 1) Developer codes locally - What they do: - Create a feature branch - Run flows locally to test changes - Push and open a PR - Behind the scenes: - Nothing Prefect-specific yet—just local runs 2) PR opens → CI creates a per-branch dev deployment - What they do: - Open the PR and wait a minute for CI to finish - Behind the scenes: - CI sets BRANCH_NAME and a safe BRANCH_SLUG (e.g., replace slashes with dashes) - CI runs

prefect deploy

using prefect.yaml, overriding the name to include the branch slug (e.g., my-flow-dev-my-feature) - Prefect Server registers a deployment for each flow defined in prefect.yaml: - Target work pool: dev - Pull step: worker clones the PR branch at run time - No schedule: manual-only - Job variables set per deployment (e.g., ENV=dev) - Result: A branch-specific dev deployment appears in the Prefect UI 3) Reviewer (or dev) triggers a dev run - What they do (choose one): - In the UI: go to Deployments → your branch deployment → Run - Or CLI:

prefect deployment run "<FLOW_NAME>/my-flow-dev-<branch-slug>" --watch

- Behind the scenes: - A flow run is created immediately on Prefect Server - A process worker in the dev work pool picks it up - The worker clones the PR branch (it always fetches the latest commit on that branch at run time) - The worker launches the flow process with job variables (e.g., ENV=dev) applied as environment variables - Logs stream back to the UI; state changes and task durations are recorded 4) Reviewer checks the results - What they do: - Open the run in the UI, watch logs, task states, and timings - Confirm the flow used the dev environment (you can log ENV or add a quick check message) - If the flow writes artifacts/outputs, verify them (e.g., test DB rows, staging bucket files) - Re-run as needed (no redeploy required—since the deployment points to the branch, it will pick up the latest commit after each push) - Behind the scenes: - Each run pulls the current head of the branch - Prefect Server stores all historical runs for review and comparison 5) Iterate on the PR - What they do: - Push fixes/updates; re-run the dev deployment from the UI/CLI to validate - Behind the scenes: - No redeploy: the dev deployment still points to the branch; runs will use the latest commit 6) Merge to main → CI updates prod deployment(s) - What they do: - Merge when review passes - Behind the scenes: - CI runs

prefect deploy

for prod with BRANCH_NAME=main (or Git ref pinned to main) - Prod deployment(s) update to point to main and use prod job variables (e.g., ENV=prod) - If you have prod schedules, they continue or are (re)applied here 7) Cleanup the dev deployment - What they do: - Nothing; CI handles this after merge or PR close - Behind the scenes: - CI deletes the branch-specific dev deployment to keep the UI tidy:

prefect deployment delete "my-flow-dev-<branch-slug>" --yes

- Past dev flow runs remain in history unless you prune data yourself How reviewers “check changes” in practice - Locate the branch deployment in the UI (e.g., my-flow-dev-my-feature) - Trigger a run, watch logs, and confirm: - It ran against dev resources (ENV=dev and any staging credentials) - Changes behave as expected (tasks complete, outputs/artifacts look right) - Optional: include a small log line at flow start that prints the current git branch or commit SHA for traceability - Re-run after each commit push until satisfied - Approve PR; after merge, confirm prod deployment updated and, if scheduled, is healthy on its next scheduled run Notes and tips

- Keep the dev deployment manual-only to avoid accidental runs - Make sure dev process workers have: - Access to clone the repo (use a Git credentials block or a machine user if private) - The correct Python dependencies pre-installed (simplest with process workers) - Use Prefect Blocks for secrets instead of env vars on pools or deployments. Access them in code so you don’t leak credentials. See: Store secrets - Put all deploy configuration in prefect.yaml for clarity. Reference: prefect.yaml - CI/CD patterns and git-based deployments: CI/CD with Prefect - Customize job variables (environment for process workers): Customize job variables If you want, I can share a minimal prefab prefect.yaml and two short CI jobs: - PR open/close for dev deployments - Push to main for prod deployments

Totally doable. For pre-PR testing, you can create a local dev deployment that runs your working copy on your machine—no Git push required. Here’s a simple flow for the developer, what happens behind the scenes, and how it fits with your review flow. One-time setup for local testing - Developer - Ensure PREFECT_API_URL points to your Prefect Server and you’re authenticated. - Start a local process worker attached to the dev pool:

prefect worker start --type process --pool dev

- Make sure your local Python env has the project deps installed. - Behind the scenes - The worker connects to the “dev” work pool and waits for runs. - It will execute flows as a local process with the environment you define. Create/update a local dev deployment (from your working copy) - Developer - Run a small script that deploys your flow using your local filesystem as storage:

import os
    from prefect.flow import Flow
    from prefect.runner.storage import LocalStorage

    if __name__ == "__main__":
        repo_root = os.path.abspath(".")  # path to your working copy
        deployment_name = f"my-flow-dev-local-{os.getenv('USER','dev')}"
        storage = LocalStorage(path=repo_root)

        Flow.from_source(
            source=storage,
            entrypoint="flows/my_flow.py:my_flow",
        ).deploy(
            name=deployment_name,
            work_pool_name="dev",
            schedule=None,  # manual-only
            job_variables={
                "env": {"ENV": "dev"},
                "working_dir": repo_root,  # optional but handy for relative paths
            },
        )

        print(f"Created/updated deployment: {deployment_name}")

- Behind the scenes - This registers a deployment that points at your local path (LocalStorage). - No git clone; the worker will run the code directly from your filesystem. - Job variables apply per run (e.g., ENV=dev, working_dir). Trigger a test run manually - Developer/Reviewer - UI: open Deployments → select your “my-flow-dev-local-<user>” deployment → Run - Or CLI:

prefect deployment run "<FLOW_NAME>/my-flow-dev-local-<user>" --watch

- Behind the scenes - Prefect Server creates a flow run. - Your local process worker picks it up, runs Python from your working_dir against your working copy. - Logs stream to the UI; states and task timings are recorded. Iterate on code without redeploying - Developer - Edit code locally and re-run the same deployment from the UI/CLI. - You only need to redeploy if you change the entrypoint path or want to rename the deployment. - Behind the scenes - Since the deployment points to your local path, each run uses your most recent local files. Promote to branch-based review when ready - Developer - Push your branch and open a PR. - Behind the scenes - Your CI creates a per-branch dev deployment (e.g., my-flow-dev-<branch-slug>) that pulls code from the branch at run time. - Reviewers use that deployment for branch-isolated tests (as we discussed earlier). Cleanup when done - Developer/CI - Delete the local dev deployment if you want to keep the UI tidy:

prefect deployment delete "my-flow-dev-local-<user>" --yes

What reviewers will check (locally and in PR) - Trigger runs from the relevant deployment (local or branch-based). - Confirm logs and behavior match expectations (e.g., using ENV=dev resources). - Verify outputs in dev systems (test DB/staging buckets, etc.). - For traceability, log the active ENV and optionally the current commit or a “local” flag at flow start. When to use this vs the PR flow - Use the local deployment path when: - You want orchestration (UI, logging, retries) while still editing locally. - You don’t want to push WIP branches yet. - You’re running a worker on the same machine (the local path must be accessible to the worker). - Use the branch-based PR deployments when: - You’re ready for review.

- You need remote workers to execute your code. - You want branch isolation and a shared review surface for the team. Notes and caveats - Path access: LocalStorage assumes the worker can read your local path—run the worker on the same machine for this pattern. - Dependencies: With process workers, your local env must have all required packages installed. If you change deps, update/restart the worker environment. - Secrets: Keep credentials in Prefect Secret blocks and load them in your flow; avoid storing secrets in job_variables or pool env. See Store secrets. - Deployments overview and CI patterns: Deployments and CI/CD with Prefect. If you want, I can adapt that deploy script to your exact flow names/paths and add a convenience Makefile target like

make deploy-local

and

make run-local

.