(Crossposted from Discourse)
Hello,
I manage a custom workflow manager stack that handles the scheduling and implementing of (mostly) ETL processes to download and store data. I’ve been investigating Prefect OSS and have decided to make the next step and start testing whether it can provide improvements for my team that wouldn’t require the management and upkeep on our custom build. In beginning to map things out, I’ve come to a fork in the road that I’d love some guidance on how to handle.
Our current system has a persistent scheduler that stores schedules in a MongoDB collection and creates future jobs based upon those schedules. Those schedules persist between deployments because the MongoDB data is untouched during deployment and just reused every time we deploy/rebuild. They are editable in a UI.
When I first started researching Prefect a year ago (2.0 was still in beta), deployments I think were handled or viewed much differently. Now, it appears as though a deployment is more 1:1 with a flow, or is at least more presented that way. The goal from a scheduling standpoint is to allow non-developers to hop into the Prefect UI and add/edit a schedule and then when we were to deploy a new build, the schedule would persist.
With that said, when is it appropriate to deploy a flow again? Never? Changing the flow file allows for scheduling to persist. How does flow.server() play into this? The quickstart guidance explains that it is the easiest way to deploy a flow, but that rarely makes it the correct way.
I appreciate any insight you all can provide on this topic and look forward to following along here. Thanks again.