Great question. Here’s a simple, robust pattern for dev/prod separation with Prefect 3.x (assuming Prefect Server on your GCP VM and code stored in GitHub). We’ll use two work pools and two deployments per flow: one for dev (tracking a branch) and one for prod (pinned to a commit). Key ideas - Separate work pools: one for dev, one for prod. Workers attached to each pool keep environments isolated. - Dev tracks a branch (e.g., develop) for fast iteration; Prod pins a commit/tag for reproducibility. - Use

Flow.from_source(...).deploy(...)

with Git storage so workers clone from GitHub at run time. - Put your GitHub token in worker environment or job variables so private repos can be pulled. Docs for reference - Work pools and workers: Work pools and workers - Deployment versioning and pinning: Deployment versioning - Using Git as storage (RunnerStorage/GitRepository): Runner storage (GitRepository) - Optional: prefect.yaml workflows: prefect.yaml guide - GitHub integration (tokens): prefect-github One-time setup on your GCP VM (Server) 1) Create two work pools - In the Prefect UI (running on your VM), create two Process work pools:

dev-pool

and

prod-pool

. This avoids CLI guessing and is the quickest path. 2) Start one worker per pool on your VM - In the UI, open each pool and click “Start a worker” to get the exact command. Example:

export PREFECT_API_URL=http://<your-vm-ip>:4200/api
export GITHUB_TOKEN=<your-gh-token-with-repo-read>
prefect worker start --pool dev-pool

…and in a second terminal:

export PREFECT_API_URL=http://<your-vm-ip>:4200/api
export GITHUB_TOKEN=<your-gh-token-with-repo-read>
prefect worker start --pool prod-pool

Tip: put the env vars into your systemd service or shell profile so workers always have them. 3) Decide your Git strategy - Dev: track branch (e.g.,

develop

) - Prod: pin to a specific commit or tag for reproducibility Code example: a simple flow In your repo, e.g., `flows/my_flow.py`:

from prefect import flow, get_run_logger

@flow
def my_flow(message: str = "hello"):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Running with message={message}")

Deployments from GitHub (dev/prod) The simplest, explicit approach is to deploy with Python using

from_source(...).deploy(...)

. This registers deployments to your Server so workers can pull the code from GitHub at run time. Create a local

deploy.py

script you can run from your laptop or CI. Make sure your terminal has

PREFECT_API_URL

set to your GCP VM’s API URL. For private repos, also set

GITHUB_TOKEN

. ``` import os from prefect import flow from prefect.flows import Flow from prefect.runner.storage import GitRepository, GitCredentials # Configure Git storage for dev (tracks a branch) dev_repo = GitRepository( url="https://github.com/<org>/<repo>.git", branch="develop", credentials=GitCredentials(access_token=os.environ.get("GITHUB_TOKEN")), ) # Configure Git storage for prod (pin a commit or tag) prod_repo = GitRepository( url="https://github.com/<org>/<repo>.git", # Option A: pin a specific commit commit_sha="abc123def4567890abc123def4567890abc123de", # Option B: or pin a release tag instead of commit with

branch="v1.2.3"

credentials=GitCredentials(access_token=os.environ.get("GITHUB_TOKEN")), ) # Entrypoint to your flow within the repo entrypoint = "flows/my_flow.py:my_flow" # Deploy Dev: tracks 'develop' branch, DEBUG logging, dev tags/params Flow.from_source(dev_repo, entrypoint=entrypoint).deploy( name="my-flow-dev", work_pool_name="dev-pool", parameters={"message": "hello from dev"}, tags=["dev"], job_variables={ "env": {

"PREFECT_LOGGING_LEVEL": "DEBUG", "ENV": "dev", # ensure worker has access if not set globally "GITHUB_TOKEN": os.environ.get("GITHUB_TOKEN", ""), } }, # Example schedule: every 5 minutes cron="*/5 * * * *", ) # Deploy Prod: pinned to commit, INFO logging, prod tags/params Flow.from_source(prod_repo, entrypoint=entrypoint).deploy( name="my-flow-prod", work_pool_name="prod-pool", parameters={"message": "hello from prod"}, tags=["prod"], job_variables={ "env": { "PREFECT_LOGGING_LEVEL": "INFO", "ENV": "prod", "GITHUB_TOKEN": os.environ.get("GITHUB_TOKEN", ""), } }, # Example schedule: daily at 02:00 UTC cron="0 2 * * *", # Optional concurrency cap to protect prod resources # concurrency_limit=1, ) print("Dev and Prod deployments registered.")

Developer workflow: new pipeline
- Prereqs (one-time on the developer machine)
  - Install Prefect 3.x and set your Server URL:

export PREFECT_API_URL=http//<your vm ip>4200/api ``` - If the repo is private, export

GITHUB_TOKEN

(or ensure workers have it via env/job_variables). - Steps 1)

git pull

the repo and create your new flow file under

flows/

. 2) Commit and push to your

develop

branch. 3) Register deployments: - Run

python deploy.py

(as above). This updates or creates dev/prod deployments on your Server. 4) Test dev: - In the UI, go to Deployments →

my-flow-dev

→ “Quick run” to trigger a run and check logs. - If you need different parameters, adjust in the UI or in your

deploy.py

and re-run it. 5) Iterate: - Update code on

develop

, push; the dev deployment (tracking the branch) will pull the latest on each run. No redeploy needed unless you change deployment config (e.g., schedule, parameters, work pool). 6) Promote to prod: - Identify the commit SHA you want to lock for prod (e.g., your release commit). - Update

prod_repo.commit_sha

in

deploy.py

, run

python deploy.py

again to update the prod deployment. - Prod runs will now use that exact code version. Developer workflow: editing an existing pipeline - For a dev-tracking deployment: - Push changes to the tracked branch; reruns will pull latest code automatically. - If you change deployment-level settings (schedule, parameters), re-run the

deploy.py

to update. - For prod: - Bump the pinned

commit_sha

(or tag) in

deploy.py

to promote a new version; re-run the deploy script. Notes and tips - Authentication to GitHub - Simplest: use a fine-scoped PAT in

GITHUB_TOKEN

. Ensure it’s present in the worker environment or passed via

job_variables.env

. - Keep secrets in environment variables or Prefect Blocks/Variables (see storing secrets). - Separation - Each pool can run on different machines or the same VM. You can also set different environment variables, resource limits, or Docker images per pool or per deployment via

job_variables

. - Scheduling and concurrency - Adjust schedules in

.deploy()

or via the UI. - Use

concurrency_limit

in

.deploy()

or global Concurrency Limits to keep dev from starving prod. - Alternative: prefect.yaml + CLI - If you prefer YAML, you can define two deployments in

prefect.yaml

(one tracking

develop

, one pinned to a commit) and run

prefect deploy

. See prefect.yaml guide. - For private repos in YAML, use a

git_clone

pull step with a token or credentials block. - If you go this route, check

prefect deploy --help

locally to see the exact flags. Would you like me to adapt the

deploy.py

and the pool/worker setup to your exact repo name, branch naming, and whether you’re using Docker or Process workers?

Short answer - Yes, you can use a single branch or a develop/main branching model. Both work well in Prefect 3.x. - Yes, GitHub Actions can handle deployments. Easiest/most reliable is to run Actions on a self‑hosted runner on your GCP VM (or otherwise make your Prefect Server API reachable from Actions). Below is a compact pattern you can copy: a single flow that takes an

env

parameter and loads dev/prod config, two deployments (dev and prod) that reference the same flow, and GitHub Actions workflows for each strategy. Helpful docs - Work pools and workers - Deployment versioning - Git storage (GitRepository) - prefect.yaml (optional) 1) One-time setup (on your GCP VM with Prefect Server) - Create two work pools in the UI:

dev-pool

and

prod-pool

(Process type is simplest). - Start a worker for each pool (ensure these env vars are set or in your service):

export PREFECT_API_URL=http://<your-vm-ip>:4200/api
export GITHUB_TOKEN=<pat-with-repo-read>
prefect worker start --pool dev-pool
# new terminal
export PREFECT_API_URL=http://<your-vm-ip>:4200/api
export GITHUB_TOKEN=<pat-with-repo-read>
prefect worker start --pool prod-pool

2) Example flow: load dev/prod config with a parameter

# flows/my_flow.py
from prefect import flow, get_run_logger

CONFIG = {
    "dev": {
        "sink_table": "my_schema.my_table_dev",
        "write_mode": "append",
    },
    "prod": {
        "sink_table": "my_schema.my_table",
        "write_mode": "overwrite",
    },
}

@flow
def my_flow(env: str = "dev", message: str = "hello"):
    logger = get_run_logger()
    if env not in CONFIG:
        raise ValueError(f"Unknown env: {env}")
    cfg = CONFIG[env]

    <http://logger.info|logger.info>(f"Running in env={env}")
    <http://logger.info|logger.info>(f"Will write to table={cfg['sink_table']} with mode={cfg['write_mode']}")
    <http://logger.info|logger.info>(f"message={message}")

    # ... your ETL/ELT logic here ...
    # write_to_table(cfg["sink_table"], mode=cfg["write_mode"])

3) Deployments: dev and prod from GitHub Create a

deploy.py

that you can run locally or in CI. This shows both strategies. Strategy A: Single-branch repo (prod pins a commit, dev tracks branch) - Dev: track

main

branch (always latest) - Prod: pin the current commit (reproducible) Strategy B: Two-branch repo (develop/main) - Dev: track

develop

- Prod: either pin a commit on

main

or track a release tag ``` # deploy.py import os from prefect.flows import Flow from prefect.runner.storage import GitRepository, GitCredentials REPO_URL = "https://github.com/<org>/<repo>.git" ENTRYPOINT = "flows/my_flow.py:my_flow" GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN") # Choose one of the two dev strategies below: # Strategy A (single branch): dev tracks main dev_repo = GitRepository( url=REPO_URL, branch="main", credentials=GitCredentials(access_token=GITHUB_TOKEN), ) # Strategy B (two branches): dev tracks develop # dev_repo = GitRepository( # url=REPO_URL, # branch="develop", # credentials=GitCredentials(access_token=GITHUB_TOKEN), # ) # PROD: pin to a commit (reproducible). In CI, set this from $GITHUB_SHA. prod_commit = os.environ.get("PROD_COMMIT_SHA", "abc123def4567890abc123def4567890abc123de") prod_repo = GitRepository( url=REPO_URL, commit_sha=prod_commit, credentials=GitCredentials(access_token=GITHUB_TOKEN), ) # Dev deployment: env='dev' Flow.from_source(dev_repo, entrypoint=ENTRYPOINT).deploy( name="my-flow-dev", work_pool_name="dev-pool", parameters={"env": "dev", "message": "hello from dev"}, tags=["dev"], job_variables={ "env": { "ENV": "dev", "GITHUB_TOKEN": GITHUB_TOKEN or "", "PREFECT_LOGGING_LEVEL": "DEBUG", } },

# run every 5 minutes (example) cron="*/5 * * * *", ) # Prod deployment: env='prod' Flow.from_source(prod_repo, entrypoint=ENTRYPOINT).deploy( name="my-flow-prod", work_pool_name="prod-pool", parameters={"env": "prod", "message": "hello from prod"}, tags=["prod"], job_variables={ "env": { "ENV": "prod", "GITHUB_TOKEN": GITHUB_TOKEN or "", "PREFECT_LOGGING_LEVEL": "INFO", } }, # daily at 02:00 UTC (example) cron="0 2 * * *", # concurrency_limit=1, # optional ) print("Dev and Prod deployments updated.")

4) GitHub Actions examples
Recommendation: use a self-hosted runner on your GCP VM so the workflow can reach your Prefect Server at localhost or your private IP. If you use GitHub-hosted runners, make sure your Prefect Server’s API is publicly reachable and secured, then set `PREFECT_API_URL` accordingly.

Dev workflow (trigger on push to develop or main depending on your strategy)

# .github/workflows/deploy-dev.yml name: Deploy Dev on: push: branches: [ develop ] # or [ main ] for single-branch strategy jobs: deploy-dev: runs-on: self-hosted # runner on your GCP VM recommended env: PREFECT_API_URL: http://localhost:4200/api GITHUB_TOKEN: ${{ secrets.GH_TOKEN_READ }} steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.11" - name: Install Prefect run: pip install --upgrade prefect - name: Deploy dev run: python deploy.py

Prod workflow (trigger on tag or push to main; pin commit using GITHUB_SHA)

# .github/workflows/deploy-prod.yml name: Deploy Prod on: push: branches: [ main ] tags: - "v*.*.*" jobs: deploy-prod: runs-on: self-hosted env: PREFECT_API_URL: http://localhost:4200/api GITHUB_TOKEN: ${{ secrets.GH_TOKEN_READ }} PROD_COMMIT_SHA: ${{ github.sha }} # pin the deployment to this commit steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.11" - name: Install Prefect run: pip install --upgrade prefect - name: Deploy prod (pin to commit) run: python deploy.py ``` 5) Developer steps (day-to-day) - New flow: - Add

flows/my_new_flow.py

with

@flow

. - Commit/push to the branch you’re using for dev (develop or main). - Run

python deploy.py

locally or let the Dev GitHub Action run to register/update the

-dev

deployment. - Test via the UI “Quick run”. - When ready for prod, merge to main (or tag a release) and let the Prod workflow run, which pins the deployment to the new commit SHA. - Edit existing flow: - Update code and push to dev branch. Dev deployment (tracking branch) pulls the latest code at run time. - Promote to prod by merging/tagging so the Prod workflow updates the pinned commit for

my-flow-prod

. Notes - Single-branch vs multi-branch: both are fine. Even on a single branch, you get strong prod guarantees by pinning

prod

to a specific commit while

dev

tracks the branch head. - Private repos: ensure

GITHUB_TOKEN

is available to the worker process (via environment or

job_variables.env

) so it can clone at run time. - You can also use environment variables or blocks for configuration instead of the inline

CONFIG

dict if you prefer centralized config management. If you share your repo URL, desired branch strategy, and whether you’re using a self‑hosted runner, I can tailor the

deploy.py

and Actions files exactly to your setup.

Short answer: Yes—you can do pure branch tracking (develop→dev, main→prod). It works fine in Prefect 3.x. Pinning prod to a commit is not required, but it’s a best practice for reproducibility, rollback, and auditability. Why teams still pin prod to a commit or tag - Reproducibility: Workers clone your code at run time. If your prod deployment tracks a branch, every run uses the current branch head. A run scheduled yesterday might execute different code today if a merge landed in between. - Rollback: With a pinned commit/tag, rolling back prod is as simple as re-pointing to the previous commit/tag. - Auditability/Change control: You know exactly what code version produced each result. This is often a compliance requirement. - Safer promotions: You can gate promotions to prod (approvals, checks) and only update prod deployments when ready. That said, many teams: - Track a branch for dev (fast iteration) - Promote to prod by updating the prod deployment to a commit SHA (automatically from GitHub Actions on merge to main or on release tag) If you prefer pure branch tracking for prod, you can. Just understand the drift/troubleshooting trade-offs. Concrete example: one flow, dev/prod deployments, with either branch tracking or commit pinning Flow that switches config by env parameter

# flows/my_flow.py
from prefect import flow, get_run_logger

CONFIG = {
    "dev": {
        "sink_table": "my_schema.my_table_dev",
        "write_mode": "append",
    },
    "prod": {
        "sink_table": "my_schema.my_table",
        "write_mode": "overwrite",
    },
}

@flow
def my_flow(env: str = "dev", message: str = "hello"):
    logger = get_run_logger()
    if env not in CONFIG:
        raise ValueError(f"Unknown env: {env}")
    cfg = CONFIG[env]

    <http://logger.info|logger.info>(f"Running env={env}")
    <http://logger.info|logger.info>(f"Writing to {cfg['sink_table']} with mode={cfg['write_mode']}")
    <http://logger.info|logger.info>(f"message={message}")

    # ... your ETL/ELT here ...

Deploy script with both options - Option 1 (branch-tracking prod): prod tracks main (no pinning) - Option 2 (pinned prod): prod pins to commit SHA (recommended)

# deploy.py
import os
from prefect.flows import Flow
from prefect.runner.storage import GitRepository, GitCredentials

REPO_URL = "<https://github.com/<org>/<repo>.git>"
ENTRYPOINT = "flows/my_flow.py:my_flow"
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN")

# DEV: track 'develop' (or 'main' if single-branch)
dev_repo = GitRepository(
    url=REPO_URL,
    branch=os.environ.get("DEV_BRANCH", "develop"),
    credentials=GitCredentials(access_token=GITHUB_TOKEN),
)

# PROD Option 1: track 'main' (no pinning)
prod_repo_branch = GitRepository(
    url=REPO_URL,
    branch=os.environ.get("PROD_BRANCH", "main"),
    credentials=GitCredentials(access_token=GITHUB_TOKEN),
)

# PROD Option 2: pin to a commit (set in CI via PROD_COMMIT_SHA)
prod_commit = os.environ.get("PROD_COMMIT_SHA")  # e.g. $GITHUB_SHA
prod_repo_pinned = (
    GitRepository(
        url=REPO_URL,
        commit_sha=prod_commit,
        credentials=GitCredentials(access_token=GITHUB_TOKEN),
    )
    if prod_commit
    else None
)

# Deploy DEV (tracks branch)
Flow.from_source(dev_repo, entrypoint=ENTRYPOINT).deploy(
    name="my-flow-dev",
    work_pool_name="dev-pool",
    parameters={"env": "dev", "message": "hello from dev"},
    tags=["dev"],
    job_variables={"env": {"ENV": "dev", "GITHUB_TOKEN": GITHUB_TOKEN or ""}},
    cron="*/5 * * * *",  # example schedule: every 5 minutes
)

# Deploy PROD using pin if provided, else branch tracking
prod_source = prod_repo_pinned or prod_repo_branch
Flow.from_source(prod_source, entrypoint=ENTRYPOINT).deploy(
    name="my-flow-prod",
    work_pool_name="prod-pool",
    parameters={"env": "prod", "message": "hello from prod"},
    tags=["prod"],
    job_variables={"env": {"ENV": "prod", "GITHUB_TOKEN": GITHUB_TOKEN or ""}},
    cron="0 2 * * *",  # example schedule: 02:00 UTC
)

print("Dev and Prod deployments updated.")

GitHub Actions - Both workflows assume your Prefect Server API is reachable (self-hosted runner on the GCP VM is simplest). Set

PREFECT_API_URL

and

GH_TOKEN_READ

secrets. Dev deploy on develop pushes

# .github/workflows/deploy-dev.yml
name: Deploy Dev
on:
  push:
    branches: [ develop ]  # or [ main ] if single-branch
jobs:
  deploy-dev:
    runs-on: self-hosted
    env:
      PREFECT_API_URL: <http://localhost:4200/api>
      GITHUB_TOKEN: ${{ secrets.GH_TOKEN_READ }}
      DEV_BRANCH: develop
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install --upgrade prefect
      - name: Deploy dev
        run: python deploy.py

Prod deploy - Branch-tracking prod (auto-advance on every merge to main):

# .github/workflows/deploy-prod-branch.yml
name: Deploy Prod (branch tracking)
on:
  push:
    branches: [ main ]
jobs:
  deploy-prod:
    runs-on: self-hosted
    env:
      PREFECT_API_URL: <http://localhost:4200/api>
      GITHUB_TOKEN: ${{ secrets.GH_TOKEN_READ }}
      PROD_BRANCH: main
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install --upgrade prefect
      - name: Deploy prod (tracks main)
        run: python deploy.py

- Pinned prod (recommended): pin to the exact commit that triggered the workflow:

# .github/workflows/deploy-prod-pinned.yml
name: Deploy Prod (pinned)
on:
  push:
    branches: [ main ]
  # or: on release tag creation
  # release:
  #   types: [published]
jobs:
  deploy-prod:
    runs-on: self-hosted
    env:
      PREFECT_API_URL: <http://localhost:4200/api>
      GITHUB_TOKEN: ${{ secrets.GH_TOKEN_READ }}
      PROD_COMMIT_SHA: ${{ github.sha }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install --upgrade prefect
      - name: Deploy prod (pin to commit)
        run: python deploy.py

What this means operationally - If you choose branch-tracking for prod: - You only need to register the prod deployment once; it will always pull the current main head at run time. Re-deploy only when you change deployment settings (schedule, parameters, work pool). - Risk: a run scheduled last week could execute new code if main advanced since scheduling. - If you choose pinned for prod: - Prod only changes when your GH Action updates the deployment to a specific commit SHA or tag. This gives consistent, auditable behavior and fast rollbacks. - Promotion == “update prod deployment to new SHA/tag.” Either approach works. If you don’t have strict compliance needs and want maximum simplicity, tracking

main

for prod is fine—just be aware of the drift trade-offs. If you want strong guarantees and easy rollbacks, use the pinned workflow for prod. References - Work pools and workers: Work pools and workers - Deployment versioning: Deployment versioning - Git storage (GitRepository): Runner storage (GitRepository) - Using prefect.yaml (optional): prefect.yaml If you share whether you want branch-tracking or pinned prod, I can tailor the deploy.py and workflows to your repo/branches exactly.