We are currently using Github releases tags more specificall

Question

We are currently using Github releases tags more specifically and CircleCI to publish flows into AWS ECR we are doing this from a monolithic repository so that our data scientists can publish their flows in a standardized way without the need for them to worry about infrastructure etc We use a hash on the flow file to determine if the file has changed since the last release and deploy all the flows that have changed We are running into an issue in Staging whereby one release from one person is clobbering the release made by another person since they are working in different branches Does anyone have a similar setup How have you addressed the issue above I supposed some kind of version ordering would address this issue As in if the registered version in Cloud is gt than the version to be deployed then don t deploy

Samuel Hinton · Answer

We re coming up to a similar decision about uploading flows on tag using a pipeline so commenting so I can also see what the recs are for this situation

Anna Geller · Answer

There are so many ways you could approach it I will give you some suggestions as food for thought and you can decide what suits your use case best 1 Decouple ECR image build from flow registration process you mentioned ECR build process being used to publish flows does it mean you are using Docker storage and bounding code dependencies with flow code storage If so decoupling both from each other i e using ECR build to package code dependencies and sth like GitHub or S3 storage for flow storage may help here 2 Add multiple workflow definitions i e a different workflow for each team use case Git branch to decentralize the process and make it easier to manage and reduce the surface area of some users stepping on each other s toes 3 You mentioned you use hash I assume you mean serialized flow hash as in `flow register project name= xxx idempotency key=flow serialized hash ` if so using CLI prefect register instead can help here as it automatically detects flow changes and you can use it to register only directories relevant for your team say if data scientists have their own folder in your shared repo their CI CD workflow might include the command ```prefect register project xxx p data science flows ```

Jacob Blanco · Answer

1 We do some kind of a hybrid where we package most of the dependencies in a base Docker image and then create a new image with the flow published again to ECR upon deployment It s an interesting thought and maybe a change we can consider while moving to 2 0 2 The multiple workflow definition is definitely an interesting approach The most straight forward way I can envision doing this is to a split out the repo into multiple owners or b split out the Flows folder structure by Team Owner rather than Theme Database like we do now this would also help us with issues around Codeownership for PR reviews 3 Actually our pipeline pre dates the `idempotency key` and we don t calculate our own and instead use the git diff between the latest staging production tag and the latest production tag which is an imperfect solution of course We are indeed using `flow register` so your point still stands

Anna Geller · Answer

Catching up after holidays LMK if there is anything you d like to discuss here I have no more ideas probably best if you explore various options and iterate on it and reflect what works and what doesn t

Jacob Blanco · Answer

Thank you Anna you ve given me a lot of food for thought

Jacob Blanco · Answer

Sorry to bring this thread back from the depths of Tartarus but I ve had a closer look at your proposed solution using what I assume is a combination of Github storage with DockerRun Is that correct The benefits I see are Way less tags deployments registrations since you only need to register a change to the flow if the git ref is changed or the structure of the flow has been changed This will in turn shorten the testing cycle so instead of make code change push to branch create staging tag wait for re registration run flow to Make code change push to branch run flow We can even have per flow branch mapping configuration so we couldn t clobber each other code changes We could even as you suggested maintain totally parallel branches for each team so re registration can happen in totally independent flows Also by decoupling the base image from the flow it will shorten the cycle of updating flow dependencies because all we need to do is deploy the new base image with the latest tag and it will get picked up in the next flow run It also reduces the overall number of images stored everywhere so maintenance is less of a hassle The only downside I see is that we need to teach people when the flow has to be re registered or is there a way to determine that from the flow object I m thinking we can trigger that on merge into the feature branch

Anna Geller · Answer

before we dive deeper here and before you start any process optimizations what s your upgrade plan to Prefect 2 0 not sure whether optimizing the process for 1 0 makes sense now it depends on when how do you plan to migrate

Anna Geller · Answer

definitely separating the code dependencies from flow storage is the right first step to make redeploying changes to shared modules base images more independent of flow code changes not to give you a disappointing answer but everything else is something you would need to experiment with your team to see what works best for them education is for sure part of it but reregistering the flow from Prefect CLI should work fine from CI if no change to flow metadata occurred the registration is skipped this makes it all quite straightforward

Jacob Blanco · Answer

gt before we dive deeper here and before you start any process optimizations what s your upgrade plan to Prefect 2 0 not sure whether optimizing the process for 1 0 makes sense now it depends on when how do you plan to migrate We are still waiting on confirmation of when Prefect 2 0 will be released so that we can start working backwards towards an upgrade plan We are in the midst of renewing our contract are moving to 2 0 as a result I ve not had a deep look at 2 0 and how that impacts deployment yet but I think philosophically speaking separating the dependency from the flow definition makes sense regardless I ll take a look more deeply at 2 0 and see what our to be deployment pipeline should look like and maybe come back with more specific questions Thanks

Anna Geller · Answer

gt separating the dependency from the flow definition makes sense regardless Couldn t agree more 100 That makes sense Keep us posted