https://prefect.io logo
Title
j

Joshua Greenhalgh

04/20/2023, 4:27 PM
Are there any links to CI/CD best practices with prefect v2 - I am transitioning from v1 and would like to keep things as close as I can - I currently bundle all my flows into a single container and run on GKE - I think I will probably move to using cloud run but would like to keep the general pattern of merging to main, building a container and then thats the container which runs the flow?
1
I suppose a related Q - lots of the v2 docs seem to suggest setting up config for deployments , blocks and the like in the UI - I don't want to have to do this - I want this kind of setup specified in code that is run in my CD pipeline - what are best practices here?
j

Joseph Thickpenny Ryan

04/20/2023, 4:46 PM
Not sure about best practice, but sounds very similar to what we've been doing so I'm chime in with a few suggestions. Note that we're using Prefect Cloud We had a lot of config defined in the
Flow
and the registration command in Prefect 1 and basically all of that stuff has been moved out into the
Deployment
object in Prefect 2, which ended up being the primary hurdle We have ended up having a combination of a single infrastructure block per target environment and combining that with the ability to provide infrastructure overrides in the
Deployment
, so that we don't end up in a situation where people are having to write a flow, a deployment, and an infrastructure block. It seems to work but might not be ideal for everyone, we did it to keep as many of the Prefect 1 deployment ideas as possible and avoid everyone needing to understand the underlying runtime Effectively we have a script that scans a directory of flows, reads a config file for flow metadata (schedules etc), builds a
Deployment
object based on that definition and applies that to the cloud There are some other considerations here, it's worth reading about the file system blocks and how you can read code from there, but we already jumped through a lot of hoops to get it working so didn't venture further
:gratitude-thank-you: 1
b

Bianca Hoch

04/21/2023, 2:36 PM
Hey Joshua!
I want this kind of setup specified in code that is run in my CD pipeline - what are best practices here?
You can certainly set this up programmatically, similarly to how Joseph explained. We do have some CI/CD examples in our Recipes repository you can reference as well.
Also, you're more than welcome to schedule a rubber duck with us if it'd be easier to talk through the setup as opposed to messaging async