https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Michael Maletich

06/17/2022, 10:55 AM
Hello, all. Is it possible to implement the saga pattern with Prefect? and if so do we have an example? I was wondering if prefect is a good fit for orchestrating services.
a

Anna Geller

06/17/2022, 11:12 AM
We don't have any example for that, but if you can implement it in Python, you can orchestrate it with Prefect. Prefect can orchestrate your tasks and flows and you can totally implement any SQL pattern within your tasks and flows.
m

Michael Maletich

06/17/2022, 1:55 PM
Thank you, I think the implementation of the Saga pattern requires some primitives beyond just pure Python. Some workflow engines support it and some do not. For example Airflow requires an external service to track state: https://github.com/timozerrer/saga-pattern-airflow. But other workflow engines like temporal have support for the Saga pattern: https://github.com/temporalio/samples-java/blob/main/src/main/java/io/temporal/samples/hello/HelloSaga.java. I’m guessing Prefect would also require an external service?
a

Anna Geller

06/17/2022, 2:32 PM
could you explain what you're trying to accomplish with this Saga pattern? specifically for your use case
the patterns are nice and good, but would be great to hear about the business problem you're trying to solve
m

Michael Maletich

06/17/2022, 6:46 PM
We want to use prefect to orchestrate a kubernetes job and its dependencies. So prefect would create kubernetes config maps and secrets, start the kubernetes job and then when it finished clean-up the config map and secrets. It would be important that if the prefect flow died for some reason (k8s node crash for example) that there wasn’t secrets and config maps remaining around.
So I guess the request is simpler then a Saga but still needs some concept of recovery.
a

Anna Geller

06/17/2022, 7:19 PM
in Prefect 1.0 we have Kubernetes tasks in the task library for that purpose and we plan to port them over to Prefect 2.0 - once that's done you'll find those tasks in here https://github.com/PrefectHQ/prefect-kubernetes (not done yet)
I believe due to the dynamicism of 2.0, you'll likely be able to accomplish that using state dependencies and those Kubernetes tasks
2 Views