https://prefect.io logo
m

Maity

09/11/2023, 5:55 AM
Any advice for packaging Prefect projects so that they may be reused for different tenants/environments, as well as programmatically/deterministically tracking those deployments? For example I could have the same flow deployed for 6 different customers, across 3 different Prefect servers (customer a, customer b, a common managed server for customers which don't want to provide their own infra). Normally, if I wasn't using Prefect, I have a "code project" and a "deployment project", where the code project would contain the source code. The source code allows injection of different environment variables to support the various configs. The source code is containerized and deployed to my ACR so it can be reused whenever I need it with different AKS envs and secrets. Then I have a deployment project, which references the code projects containers/helm charts, and injects the customer specific environment variables into them.
I could do something almost identical if I was using long-lived infrastructure with the flow.serve method. Where each flow.serve would basically be its own helm deployment for each customer environment. prefect deploy is great - but keeping the prefect.yaml within the source code is not ideal for me, as my flows can be reused multiple times within the same Prefect server, or across many different Prefect servers. I would prefer to setup a separate project of many folders Customers/[CustomerName]/prefect-[env].yaml This project would represent a deterministic way to deploy every single flow for that prefect server. To achieve what I'm after, would it be best to create a prefect.yaml within the source code and create a docker image which: downloads the source accepts environment vars to pass through to the prefect.yaml the dockerfile runs prefect deploy --all any other setup code for that particular project Then encapsulate all of that in within a series of helm releases, in such that every time I deploy the chart, the prefect deploy runs and then exits. That way I can deterministically manage in different Git repos many different Prefect server implementations for my various customers, which utilize my shared library of Prefect projects. Seems like a lot of steps when the prefect.yaml at a conceptual level seems to follow the same pattern as helmfile. Can we have a project of prefect.yamls in the same vein of my helmfile project?