Thomas Stack
05/14/2024, 2:06 PMMarvin
05/14/2024, 2:06 PMDaskTaskRunner
or RayTaskRunner
, which enable tasks to run in parallel across multiple workers, even if the tasks are part of different flow runs.
For example, with a RayTaskRunner
, you could submit tasks to be executed immediately, and the task runner will manage the execution of these tasks in parallel, across available computing resources, as demonstrated here: Running parallel tasks with Ray.
Remember, you can control task execution order via dependencies as well, by using the wait_for
parameter when calling a task to specify upstream task dependencies. Here's the documentation for controlling task execution order.
In short, simultaneous execution is part of the normal behavior in Prefect and it has mechanisms to ensure that this is handled smoothly. But how exactly it handles it depends on the configuration of task runners and flow design.