Hello, all. Is it possible to implement the saga pattern with Prefect? and if so do we have an example? I was wondering if prefect is a good fit for orchestrating services.

We don't have any example for that, but if you can implement it in Python, you can orchestrate it with Prefect. Prefect can orchestrate your tasks and flows and you can totally implement any SQL pattern within your tasks and flows.

Thank you, I think the implementation of the Saga pattern requires some primitives beyond just pure Python. Some workflow engines support it and some do not. For example Airflow requires an external service to track state: https://github.com/timozerrer/saga-pattern-airflow. But other workflow engines like temporal have support for the Saga pattern: https://github.com/temporalio/samples-java/blob/main/src/main/java/io/temporal/samples/hello/HelloSaga.java. I’m guessing Prefect would also require an external service?

could you explain what you're trying to accomplish with this Saga pattern? specifically for your use case

We want to use prefect to orchestrate a kubernetes job and its dependencies. So prefect would create kubernetes config maps and secrets, start the kubernetes job and then when it finished clean-up the config map and secrets. It would be important that if the prefect flow died for some reason (k8s node crash for example) that there wasn’t secrets and config maps remaining around.

in Prefect 1.0 we have Kubernetes tasks in the task library for that purpose and we plan to port them over to Prefect 2.0 - once that's done you'll find those tasks in here https://github.com/PrefectHQ/prefect-kubernetes (not done yet)