Does this dev/prod setup make sense? (I’ve no experience setting something like this up and limited prefect experience). We're using Prefect Server hosted on a VM. 1. Two VMs in GCP, each with their own Prefect server instance running (one for Prod, one for Dev). 2. On each VM's prefect instance, the variable "Environment" is set. For the dev VM instance, it will be set to "DEV" and in prod, "PROD" 3. Each pipeline we have will read in this variable at the start, and there will be some conditional logic basically that will decide what target tables, databases, etc to look at based on whether it's DEV or PROD. We'll have it read from the dev prefect instance when developing on our local machines by setting PREFECT_API_URL env variable on to the correct api endpoint of the dev instance. 4. Developers work on feature branches, test locally, and then merge into a development branch in Github. A Github actions job runs our deployment.py script (where all our deployments are written) to deploy to prefect server (setting PREFECT_API_URL environment variable appropriately before running the script) 5. When development branch is merged into main, a second Github actions script runs which does the same as the first, except this time it changes PREFECT_API_URL to the prod instance of prefect, so that when the deployments script runs, everything gets deployed there. EDIT: actually I may only need one branch in GitHub. Once a feature branch merges into main, the GitHub actions script deploys to both dev & prod

i think this makes sense except for point 3 i would think pipelines should accept a parameter called

environment: Literal[“prod”, “dev”]

or instead set the env var on the work pool base job template i understand you want to namespace prod and dev via env var but that shouldn’t mean that all pipelines running against those respective servers need to run on the same machine as your server (unless that’s what you want, and workers would run the same VM as the server). people normally like to run their workers in a different place than the server so they can independently allocate resources. i might check out work pools in the docs, as you can set the default ENVIRONMENT there and then that work pool config is stored server side but workers will give all new flow runs that env without you having to set that env on the server machine

@Nate Thanks for feedback. I need to do some reading up on how exactly work pools, workers etc work. ATM we just have prefect server installed on a single VM and using default work pool I think. So in your example, the dev & prod version of a pipeline will be visible in the prefect UI after deploying, but using different work pools (e.g dev workpool and prod workpool), which could have workers on different VMs (these machines are where the code is actually executed)? And for using the Literal, is this recommended over using variables? It’s what we use currently for some of our pipelines, but noticed we’ve never investigated how the variable functionality works in Prefect.

work pools: building parameters for deployments

Thanks. That makes sense. Those vids are very useful! So if I had a process work pool (e.g., dev-workpool-1) would it make sense to have multiple workers running from different VMs? Would the main advantage of this be that if one worker failed for some reason, the other would stick be polling the workpool and pick up the work?

yea redundancy for workers is up to you but generally speaking i would suggest that you don't need more than • 1 worker per work pool (1

prefect worker start

process per work pool) • 1 work pool per type of infra (process, docker) per ENVIRONMENT (dev, prod)

Thanks @Nate. This has helped. I think I’ve got a slightly better dev/prod ci/cd plan in mind now. It’s not perfect but will hopefully be suitable.