Hi everyone, We are getting started with Prefect Orion and are excited about the possibilities. Our central use case is coordinating lots of data jobs that integrate into one master database. We'd like to set up a continuous integration (CI) environment to ensure that when we create or rev Flows that these Flows are fully integration/regression tested. Our current idea (open to other approaches) is to set things up so that when we go to put a new Flow in CI all the Flows (including dependents) can be fired against the CI database and not the production database. It would seem like this requires some way to either maintain separate Flows or set a parameter that goes through all flows that indicates whether this is a dev run, a CI run, or a production run. Has anyone done anything like this? If so, how did you accomplish it? Any other ideas/feedback welcome. Thanks!

we do this by simply using prefect workspaces, environments in github actions to build+deploy and secrets in each environments k8s cluster

Thanks, this sounds like what I am trying to accomplish Just so I understand, you all essentially maintain a separate kubernetes cluster (and separate Prefect instance) for CI vs staging vs prod? Is that right? How do you all keep these environments reasonably in sync?

Hi, yes we have 3 workspaces in the Prefect Cloud and a k8s cluster per environment, we keep them in sync using github actions per environment which deploy on push

Thanks, so helpful! So if I'm guessing right, the Github actions are something like - when you push changes in a Flow to x repo, push it to the CI workspace. Is that right? Also, how do you trigger dependent jobs in CI? Does that just happen on its own when the new Flow is committed or is there some way of triggering them manually? Another also, once CI succeeds how does the Flow get pushed to your production workspace?

We use github action 'changed files' and deploy the which flow(s) that have been added or changed. After deploy we use another github action to run the flows. Currently we manually merge between the branches and push to trigger the deploy

Thank you, this has been so helpful! It seems that at least one other set of people has tried to do this: https://discourse.prefect.io/t/prefect-repository-example-with-a-ci-cd-github-actions-template-by-henning-holgersen/517 I am going to try to figure out what they are doing as well.

@Eric Ellsworth yeah, I actually took Hennings project as our starter for ten and it gave me the base to build out from (I was new to python then)

That's awesome! I mentioned it to my colleagues and we are going to look into using it.

cool, let me know how you get on