Hi, just looking at prefect over writing something custom for my team, hopefully I can ask a couple questions :). We only have about a dozen task types, although some are parallel, depending on data. My product manager wants to be able to reconfigure a flow’s “dag” (sequence of tasks) , as configuration, through the user interface without modifying any source code files or involving developers. The flows also involve parallel steps as defined by data (read an input file and process all entries in the input file in parallel, then aggregate the output and perform another sequential step when all are done). These steps would also be involved in the changes through configuration. This is pretty hard, isn’t it? Obviously predecessor and subsequent tasks would have to understand each other’s input and output, if any. I could see pre-defining a range of possible flows and switching between the entire flow by configuration as simpler and a possible solution. Ideas or comments? Thank you!!

Since Prefect flows are entirely dynamic, you wouldn't even need to define multiple ahead of time. You could write a collection of tasks, and have which tasks the flow executes be determined entirely by normal conditional python logic, where perhaps parameters to the flow function are the control mechanism.

The UI functionality you’re talking about is the realm of RPA software. You could pair that with prefect to achieve what you want. The parallelism can be handled by prefect’s asynchronous nature, by the RPA software in scheduling mutiple flows, or by using dask within your flows. Each will fit a different set of tasks/flows and type of parallelism better. If you need help implementing, let me know. Big fan of the software.