Thomas Stack
05/14/2024, 2:06 PMMarvin
05/14/2024, 2:06 PMDaskTaskRunner
or RayTaskRunner
, which enable tasks to run in parallel across multiple workers, even if the tasks are part of different flow runs.
For example, with a RayTaskRunner
, you could submit tasks to be executed immediately, and the task runner will manage the execution of these tasks in parallel, across available computing resources, as demonstrated here: Running parallel tasks with Ray.
Remember, you can control task execution order via dependencies as well, by using the wait_for
parameter when calling a task to specify upstream task dependencies. Here's the documentation for controlling task execution order.
In short, simultaneous execution is part of the normal behavior in Prefect and it has mechanisms to ensure that this is handled smoothly. But how exactly it handles it depends on the configuration of task runners and flow design.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by