@Marvin how to run some subflows in a flow concurrently while waiting for a earlier subflow to finish?

Oh dear, it appears that the results I found are not quite relevant to your question. Your query seems to be about running concurrent subflows in a Prefect workflow while waiting for an earlier subflow to complete. I have to inform you, in my usual gloomy manner, that managing dependencies between tasks is an integral part of Prefect’s design. You have to define your tasks and their dependencies as part of your Flow object. To run tasks concurrently, you can simply add them to your Flow without specifying any dependencies. Prefect’s scheduler will run independent tasks concurrently whenever possible. If you want specific tasks to wait until a certain task has completed, you can use Prefect’s

upstream_tasks

argument to specify that a task should not run until its upstream tasks have completed. For instance, if you have tasks

task1

,

task2

,

task3

, and

task4

, and you want

task3

and

task4

to run concurrently after

task1

and

task2

have finished, you can define your flow as follows:

flow = Flow('My Flow')
flow.chain(task1, task2)
flow.add_edge(task2, task3)
flow.add_edge(task2, task4)

In this case,

task1

and

task2

will run in sequence, and then

task3

and

task4

will run concurrently. A word of caution though, or perhaps it's just my natural pessimism, but make sure to handle any errors appropriately, as concurrent execution can lead to tasks failing in unpredictable ways. A robot's work is never done...