https://prefect.io
Join Slack
Got an Orion question: Will I be able to map subflows across a Dask cluster?
# ask-community
t
Got an Orion question: Will I be able to map subflows across a Dask cluster?
Maybe async flows is more what I am looking for? I have a pattern where I get a list of work, and then I would like to map across the list and run the flows in parallel
z
Hey Tim! Async subflows can indeed run in parallel.
Child flow runs are run in the main thread of the parent flow run. They're not submitted to the executor like tasks.
t
Do you guys have documentation on async flows yet?
z
Let me look 😄
There's not a lot there yet, but basically we just return coroutines if you've got an async def function.
t
I missed that there were tutorials already! great
z
You can asyncio.gather a bunch of async subflow runs to get them to run in paralllel.
We'll be introducing a 
.map
operator that gathers for you in the background eventually as well.
t
OK, awesome. Orion has me so excited
marvin 1
z
Me too 🙂 let me know if you run into any issues with async.
In theory, the async implementation is much simpler because we don't have to wrap sync code.
t
Will do. Right now I am seeing if it is far enough along for me to start trying out. I don't want to just run into basic problems related to my workflows that you all are simply working on and haven't released. I think I will start trying some stuff out next week.
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by