@Marvin How would I await a list of tasks submitted to a ray cluster?

To await a list of tasks submitted to a Ray cluster using Prefect, you'll need to utilize the

RayTaskRunner

. Here's a brief overview: 1. Install the

prefect-ray

package: This is necessary to use the Ray integration with Prefect. You can install it using pip:

pip install -U prefect-ray

2. Set up your flow with the

RayTaskRunner

: You can specify the

RayTaskRunner

when defining your flow. If you already have a Ray cluster running, you can provide the connection URL via an address argument. 3. Parallel execution using Ray: When you submit tasks to Ray using Prefect, they are distributed and run in parallel across available workers on the Ray cluster. For more detailed instructions and code examples, you can visit the Prefect documentation: - Prefect 2.x documentation: Task Runners with Ray & Dask - Prefect 3.x documentation: Prefect Ray Integration Ray is great for parallelism, but do be aware of its limitations, such as experimental support on Python 3.12 and non-x86/64 architectures. Just a minor hiccup in an otherwise exciting journey to task parallelism!

https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-ray/tests/test_task_runners.py#L526-L540 or more simply

@flow(task_runner=RayTaskRunner())
def f():
  return some_task.map(range(100)).result()

where

.result()

will wait for all the concurrent mapped tasks to finish