What is the best practice for running sub-flows in parallel (not thread concurrent), while keeping all the benefits of UI observability?
The
above thread addresses the issue partly, suggesting the use of
run_deployment
to create sub-flows. While this works technically, it partially severs the link between the main flow and its subflows. When I try the
run_deployment
sub-flow method, the sub-flows do appear within the main flow, I get a blank execution graph (see image 1).
I even have a task within the main flow, but that doesn't show up on the execution graph either (see image 2).
So while I THINK
run_deployment
is working as expected, I certainly can't verify that parallel execution is working via the UI.
## Situation and additional requirements
I need to be able to run a flow of sub-flows where each sub-flow runs in parallel, at the same time. The sub-flows can also have their own sub-flows, and they all need to be classified as flows (rather than tasks) so that any of the sub components can be run independently if/when needed.
I also need to be able to return the state from each of the sub-flows so that if any of them fail, I can mark the main flow as failed. (Something I haven't figured out yet.)
It would be great to have the ability to return a result to the main flow, but that might be a tough ask with
run_deployment
, which is another reason using
run_deployment
seems like a compromise compared to the usual seamless functionality.
## Conclusion
Wishlist summary
• Create sub-flows that can run in parallel
• Have all the normal UI functionality work the same as any other sub-flow
• Ability to get the state of all sub-flows within the main flow
• Ability to return results from sub-flows within the main flow
Please let me know if there is a better way to create parallel sub-flows. I'm happy to submit an issue, but I wanted to ask here in case I'm missing something obvious! 😉