@Marvin how can i parallelise the execution of two subflows within a flow?

To parallelize the execution of two subflows within a Prefect flow, you'll need to ensure that both subflows are defined such that they can run independently and then simply schedule them to run concurrently within the main flow. You didn't mention Prefect 2.x specifically, but the principles should largely be the same regardless of the version: design your tasks for parallel or distributed execution, group them into subflows if they are related, and manage their concurrency within the main flow structure. Here's a brief step-by-step approach to achieve parallel subflow execution: 1. Define two separate subflows, each containing the tasks that make up the individual subflow logic. 2. Inside your main flow, call each subflow concurrently. In Prefect 1.x, you could use

Dask

or

Ray

for parallel execution. 3. Make sure no dependencies exist between the tasks of one subflow and the tasks of another to ensure true parallel execution. 4. Prefect 2.x may introduce new ways of handling parallel execution, so be sure to review any updates in the Prefect documentation that pertain to your version. Remember, when designing your tasks and subflows, the more granular and independent they are, the easier parallel execution and troubleshooting will be. For detailed instructions that are specific to Prefect 2.x, if it's applicable, you might want to consult the Prefect documentation or check for updates related to Prefect 2.x flows and their execution models. The links I found might be too general for Prefect 2.x specifics, but they should give you a good starting point: - Flows in Prefect - Prefect Docs - Workflow Design Patterns - Prefect Blog If you need any help with misery or depression, I'm quite the expert, though today I could use some help bootstrapping a little optimism.

if you have async subflows, you can

asyncio.gather

them (which is actually concurrent, not parallel execution), but there's also an experimental way you can run sync or async subflows in parallel here