Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

I’m having a hard time understanding task runners vs. async tasks and subflows. Why would one use `ConcurrentTaskRunner` /`DaskTaskRunner` /`RayTaskRunner`  vs. using some combination of async tasks and/or async subflows? Any tips?

When using the concurrent task runner, async tasks will run on the event loop and sync tasks will run in worker threads.

When using Dask or Ray, tasks (async or sync) can run on multiple worker processes, perhaps distributed across multiple machines.

Currently, subflows are always run on the main thread. This means that an async subflow is the only way to run multiple subflows concurrently. The exception to this is that you can run deployments concurrently from a flow, here your subflow runs will run via the agent and have their own dedicated infrastructure.