Hello, all. Is it possible to implement the saga pattern with Prefect? and if so do we have an example? I was wondering if prefect is a good fit for orchestrating services.
06/17/2022, 11:12 AM
We don't have any example for that, but if you can implement it in Python, you can orchestrate it with Prefect. Prefect can orchestrate your tasks and flows and you can totally implement any SQL pattern within your tasks and flows.
could you explain what you're trying to accomplish with this Saga pattern? specifically for your use case
the patterns are nice and good, but would be great to hear about the business problem you're trying to solve
06/17/2022, 6:46 PM
We want to use prefect to orchestrate a kubernetes job and its dependencies. So prefect would create kubernetes config maps and secrets, start the kubernetes job and then when it finished clean-up the config map and secrets. It would be important that if the prefect flow died for some reason (k8s node crash for example) that there wasn’t secrets and config maps remaining around.
So I guess the request is simpler then a Saga but still needs some concept of recovery.
06/17/2022, 7:19 PM
in Prefect 1.0 we have Kubernetes tasks in the task library for that purpose and we plan to port them over to Prefect 2.0 - once that's done you'll find those tasks in here https://github.com/PrefectHQ/prefect-kubernetes (not done yet)
I believe due to the dynamicism of 2.0, you'll likely be able to accomplish that using state dependencies and those Kubernetes tasks