Hi there! Is this doc still the most recommended w...
# ask-community
g
Hi there! Is this doc still the most recommended way to implement parallel subflows? https://discourse.prefect.io/t/how-can-i-run-multiple-subflows-or-child-flows-in-parallel/96
My DAG looks like this when following the post above:
Which in theory looks correct, but you will notice the other subflows are not running until the previous one finishes.
I think this must be a limitation of the dlt python package not supporting async loading for pipeline runs... Thanks anyway
👍 1
👀 1
n
yep you’re free to asyncio gather async subflows but the underlying code needs to be async - i will say, there’s some work in flight to sort of replicate the experience of task.submit but for flows (ie non blocking submission of concurrent subflows) - so definitely on the radar
❤️ 2
r
You also have to use a TaskRunner that supports parallelism I believe. I’m pretty sure the default ConcurrentTaskRunner doesn’t support parrallel execution. I’ve had success with DaskTaskRunner + async def and asyncio gather like Nate was saying
n
@Robin Niel task runners don’t execute subflows, only tasks. the only (sort of) exception is that you could use
run_deployment
to run deployments as subflows (which you can call from within a task) and map those, which would end up near parallel
r
Yes but if only one task can be executed at a time, doesn’t it mean every task of a subflow have to run before going to the next subflow taks ?
n
> if only one task can be executed at a time this is not generally true - are you talking about a situation where this is the case? if so, which? in general you can
my_task.submit()
or
my_task.map()
and get concurrent (
ConcurrentTaskRunner
by default) or actual parallel (Dask / Ray task runners) execution without having the submission of the task be blocking (tasks are sent to their own threads) subflows, however are currently blocking (stay on the main thread, for now), even if gathered async. again the one exception, is that you can write a task that you can submit which calls run_deployment and therefore will track the invoked deployment as a subflow of the caller
g
this is all very helpful thank you all
I am very interested in the ongoing work to get a
.submit()
sort of pattern going for subflows though!