@Marvin is it possible to apply a Ray task runner to only some tasks in a flow and not to others?

Yes, it is possible to apply a Ray task runner to only some tasks in a Prefect flow. The Prefect Ray integration allows you to run specific tasks on Ray by using the

RayTaskRunner

. You can specify which tasks should use the Ray task runner by setting it at the task level rather than the flow level. Here's a conceptual outline of how you might achieve this: 1. Install the Prefect Ray Integration: Ensure you have the

prefect-ray

package installed. 2. Define Tasks with Specific Task Runners: Apply the

RayTaskRunner

to tasks you wish to run on Ray. For tasks that you do not want to run on Ray, use the default task runner or another task runner of your choice. Here's a simple example:

python
from prefect import flow, task
from prefect_ray.task_runners import RayTaskRunner

@task
def task_on_ray():
    # Task logic here
    pass

@task
def normal_task():
    # Task logic here
    pass

@flow
def my_flow():
    task_on_ray.with_options(task_runner=RayTaskRunner())()
    normal_task()

my_flow()

In this example,

task_on_ray

will be executed using the Ray task runner, while

normal_task

will use the default task runner. This approach allows you to selectively apply the Ray task runner to specific tasks within your flow. For more information, you can refer to the Prefect Ray documentation.

Hey Fernando! You'll need to use nested flows if you'd like to use multiple task runners. Here's an example in the docs that demonstrates how to do this with Dask. You should be able to do the same with Ray.

Thanks @Bianca Hoch!