Hi all, software architect here exploring whether Prefect might be a good option for running some rather complex flows in a production SaaS product which involve a mix of... 1. Multiple users with different roles, all submitting their own inputs and reacting to the progress of the flow in different ways. 2. Calls to various services (some long-running). 3. Quite a bit of branching logic based on the inputs provided by users and other services, and potentially even the need to repeat, loop, and/or add/remove steps midway through the flow as needed. 4. Potentially multiple different terminal states for the flow (e.g., we can terminate if we have data X and Y, or else if we have data Z, X, and A). 5. The ability to run and A/B test different versions of flows for different users simultaneously. Not sure if something that hairy is abusing Prefect's focus on data engineering, or is even possible in Prefect...but I look forward to finding out and getting to know you all. :)

Welcome! Sounds like a complex use-case 🙂

I have been for integrating a SaaS app with enterprise customers, and our first tests were pretty good, although I found that the branching logic is still lacking somewhat. the

case

statements help, but it starts to get messy when your branching is core in the flow, or when you have many branches. Reacting to flow progress doesn’t seems like that would be a problem, since the

state_handler

logic in Prefect is very powerful. My use case also involved long running tasks (+ 30min) and async flows, and that worked just fine. At the end of the day, if you think of it as almost pure Python, with powerful orchestration in the backend, it definitely should be possible. I would love to hear the results of your experiments!

Thanks for the detailed feedback! I'll keep you posted.

Task mapping works great, using the dask executer things can run distributed, which is great, I’ve only tested locally, but shouldn’t be a problem in production, specially if you have any experience with k8s.