https://prefect.io logo
Title
s

Steven Das

01/06/2021, 2:12 PM
Hi all, software architect here exploring whether Prefect might be a good option for running some rather complex flows in a production SaaS product which involve a mix of... 1. Multiple users with different roles, all submitting their own inputs and reacting to the progress of the flow in different ways. 2. Calls to various services (some long-running). 3. Quite a bit of branching logic based on the inputs provided by users and other services, and potentially even the need to repeat, loop, and/or add/remove steps midway through the flow as needed. 4. Potentially multiple different terminal states for the flow (e.g., we can terminate if we have data X and Y, or else if we have data Z, X, and A). 5. The ability to run and A/B test different versions of flows for different users simultaneously. Not sure if something that hairy is abusing Prefect's focus on data engineering, or is even possible in Prefect...but I look forward to finding out and getting to know you all. :)
👋 2
z

Zanie

01/06/2021, 3:48 PM
Welcome! Sounds like a complex use-case 🙂
w

wiretrack

01/06/2021, 3:57 PM
I have been for integrating a SaaS app with enterprise customers, and our first tests were pretty good, although I found that the branching logic is still lacking somewhat. the
case
statements help, but it starts to get messy when your branching is core in the flow, or when you have many branches. Reacting to flow progress doesn’t seems like that would be a problem, since the
state_handler
logic in Prefect is very powerful. My use case also involved long running tasks (+ 30min) and async flows, and that worked just fine. At the end of the day, if you think of it as almost pure Python, with powerful orchestration in the backend, it definitely should be possible. I would love to hear the results of your experiments!
s

Steven Das

01/06/2021, 4:20 PM
Thanks for the detailed feedback! I'll keep you posted.
In particular strong opinions on the task looping and mapping features to support repetitious tasks with an unknown number of iterations?
w

wiretrack

01/06/2021, 4:23 PM
Task mapping works great, using the dask executer things can run distributed, which is great, I’ve only tested locally, but shouldn’t be a problem in production, specially if you have any experience with k8s.
👍 1
I don’t know how that would work with unknown number of iterations, that might be a challenge, since I usually feed the task mapping a list generated by another task (usually a collection i got from mongo with credentials and parameters to each client), so that might be a thing to look into
or maybe the “raw” way of doing things and ‘while True’ within a task, which seems odd, but might work