@Marvin can I run subflows in parallel?

It seems that the documents available don't explicitly mention whether subflows can be run in parallel in Prefect 2.x. However, Prefect 2.x is designed to be more intuitive and offers a natural way to organize flows into sub-components, suggesting a level of modularity that might support parallel execution. To get a definitive answer, you might want to refer to the Prefect documentation directly or explore the community resources. Would you like me to search for more specific information or perhaps look into GitHub issues related to this topic?

@Marvin yes please search

Yes, running subflows in parallel is a topic that is actively discussed in Prefect's GitHub issues. Here are some of the discussions that might be relevant to your question: 1. Add `Flow.submit` interface for subflows (#6689): This issue suggests that currently, subflows can only be run in the main process and only tasks can be submitted to other infrastructure to run in the background. There's a proposal for allowing flows to be submitted to run in the background, which implies that running subflows in parallel could be possible in future versions of Prefect. 2. Long running flow with subflows incorrectly displays as "crashed" (#10991): This bug report indicates that subflows can be spawned in parallel using

asyncio

. However, there's an issue with the UI marking the parent flow as "crashed" even though the subflows are still running. These GitHub issues indicate that the ability to run subflows in parallel has been recognized and is being considered for the development of Prefect, but there might be some current limitations or bugs you should be aware of. Keep an eye on the Prefect GitHub repository and the official documentation for updates on this feature.

you're free to asyncio.gather async subflows, but this is concurrent, not true parallelism there's also an experimental feature we've recently introduced to submit many flow runs in a non-blocking way, which you can find here