Iām having a hard time understanding task runners vs. async tasks and subflows. Why would one use
ConcurrentTaskRunner
/`DaskTaskRunner` /`RayTaskRunner` vs. using some combination of async tasks and/or async subflows? Any tips?
ā 1
z
Zanie
01/13/2023, 11:34 PM
When using the concurrent task runner, async tasks will run on the event loop and sync tasks will run in worker threads.
When using Dask or Ray, tasks (async or sync) can run on multiple worker processes, perhaps distributed across multiple machines.
Currently, subflows are always run on the main thread. This means that an async subflow is the only way to run multiple subflows concurrently. The exception to this is that you can run deployments concurrently from a flow, here your subflow runs will run via the agent and have their own dedicated infrastructure.