Hello people, do any of you use the prefect-ray integration (with or without the RayTaskRunner) at the production level? We are currently wondering as we have examined many solutions to accommodate it but they either lose state tracking in the dashboard or involve having a n head nodes, 1 for each flow because of how the serialization and dependency resolution works. This is especially untenable for multiple environments where developers could be testing multiple models or workflows.

@Nate 👀

hey @Mitch - do you have a code example or two where you can show exactly what's violating your expectations? that'd be super helpful

hey @Nate I can probably summarize some of the approaches we are considering. We're wondering if other people have come to similar conclusions or how they have structured their own integrations. Current Options for Integrating Ray and Prefect 1. Ray Client (

ray://…:10001

) ◦ Driver (e.g., Prefect worker) off-cluster (or not a Ray node); connects to the head’s Ray Client port 10001. ◦ Works with

RayTaskRunner

. ◦ "Fat head" requirement: Head Client server deserializes on some paths → may import the same module-scope deps as your flow (decorators, evaluated hints, closures). ▪︎

runtime_env

(

pip

,

working_dir

) applies to workers (and Ray Job drivers in Jobs), not the Client server on the head. ◦ Mitigations ▪︎ Standard Ray pattern is to defer imports / avoid module-scope heavy libs → fights decorators and typing; challenging to port legacy code. ▪︎ Head image = worker stack: version skew → head tracks every workflow pin; GPU head. ▪︎ Runtime install on head → really fragile, slow, must still match workers. ▪︎ Multiple heads (clusters) → challenging K8s management and CI/CD, large resource usage. 2. Ray Job ◦ Launcher submits a Ray Job; driver runs in-cluster;

runtime_env

can cover driver + workers. ◦ Works with

RayTaskRunner

without needing to pass the head node address. ◦ Excellent for per-run / per-user deps without baking everything on the head. ▪︎ 0-CPU head + e.g.

entrypoint_num_cpus=1

on submit_job can place the driver on a worker. ◦ Makes coding pattern very different from standard prefect. Hard to call deployments across Ray jobs. 3. Sidecar + shared Ray socket ◦ Prefect worker as second container in the Ray head/worker pod; shared

/tmp/ray

(

emptyDir

) →

ray.init(address="auto")

finds the raylet. ◦ Works with

RayTaskRunner

without needing to pass the head node address. ◦ Driver in sidecar; Ray container can stay stock. No

ray://

→ no Client deserialization on the head for that path. ◦ Cost: long-lived poller unless you add scale-to-zero (below). Head sidecar: simplest. Worker-group sidecar: scalable, not to zero for free. ◦ Mitigation ▪︎ KEDA: ScaledObject + minReplicaCount: 0; trigger on queue depth. Ray Jobs + minimal submitter + KEDA can shrink the always-on footprint. 4. Ephemeral cluster per run ◦ New Ray cluster per submission. Cold start ~tens of seconds to 1m+. Often splits YAML/lifecycle across repos and teams. ◦ Works with

RayTaskRunner

without needing to pass the head node address. Other Concepts Considered 1. Multiple worker groups (KubeRay, etc.) ◦ Several

workerGroupSpecs

, different images; custom resources on nodes; tasks request resources={...} (or framework equivalents); autoscaler matches demand. ◦ Does not fix Client head imports if the driver still uses

ray://

— only routes work. ◦ "Slim head": pair heterogeneous workers with a driver that does not unpickle your code on the head (Ray Jobs, sidecar + address="auto", etc.), not Client alone. ◦ Unpickling / images: Ray

cloudpickles

into the object store; worker groups don’t remove serialization — they change where unpickle happens and which image must satisfy imports. ▪︎ `ray://`: Client server on head may unpickle → “No module named …” on head despite good workers. ▪︎ In-cluster driver (Job, sidecar, local

address="auto"

): driver pickles; workers unpickle at run; head doesn’t need your app packages for that path. Control-plane traffic still hits the head; difference is head importing user code or not. ◦ Split modules if two worker images need different stacks: one module that imports both A and B forces both images to load both unless A-tasks and B-tasks live in separate modules with only their own top-level imports. Driver may still import all flow modules; each group only needs deps for modules its tasks load. 2.

RayTaskRunner

vs

@ray.remote

◦

@ray.remote

with stdlib-only may work under Client while the flow file imports heavy libs — narrow graph. ◦

prefect[ray]

often widens imports → same head pain as the flow. Prefer in-cluster

address="auto"

or Jobs when head and worker images won’t match.

hey @Carlos Alberto da Costa Filho - thanks for the writeup!! very helpful i'm thinking that its worth adding some color to the docs, so i've opened a PR here any review that you could give would be super valuable!

hey @Nate this looks great!! thanks for documenting these patterns. doing more exploration on this, i found another pattern, documented below. I'm also wondering if I'm reinventing the wheel at this point! Prefect Pod Joins Ray Cluster • Job pod `ray start`s to cluster GCS (<head>:6379) with

--num-cpus=0 --num-gpus=0

, waits for

/tmp/ray/session_latest/sockets/raylet

, then runs the flow with

RayTaskRunner

/

ray.init

→

address="auto"

(local raylet). Not

ray://

. • Driver pickles; workers unpickle (cloudpickle → object store). Head stays control plane — no app deps on head for that path. Driver image needs all module-scope imports; split task modules by worker group if A/B images must not share top-level imports. • Zero resources: no tasks on driver; no fake capacity; not a KubeRay pod. Head still gets scheduling / heartbeats; only head-side user imports (Client path) go away. • Slim head: since pickling happens on your Prefect worker with your image of choice, you can fully control dependencies. Unpickling happends on the workers - you can provide their environments via Ray worker groups or pip/conda/uv installs. Implementation Option A: image

ENTRYPOINT

• Create an entrypoint on the Docker image that will be used for the deployment of your flow. • The entrypoint calls:

#!/usr/bin/env bash
set -euo pipefail
: "${RAY_HEAD_GCS:?}"
ray start --address="${RAY_HEAD_GCS}" --num-cpus=0 --num-gpus=0 --block &
until [[ -S /tmp/ray/session_latest/sockets/raylet ]]; do sleep 1; done
exec "$@"

Implementation Open B:

job_variable

highjacking • Pass a "command" in the

job_variables

submitted to

Flow.deploy

• Prior to running prefect, the pod joins the ray cluster.

_SHELL_SCRIPT = (
    f"ray start --address={RAY_HEAD_GCS} --num-cpus=0 --num-gpus=0 --block & "
    "until [ -S /tmp/ray/session_latest/sockets/raylet ]; do sleep 1; done; "
    "exec prefect flow-run execute"
)
RAY_DRIVER_COMMAND = f"/bin/bash -lc '{_SHELL_SCRIPT}'"

# When registering the deployment (shape depends on your helpers):
# job_variables={"command": RAY_DRIVER_COMMAND}

great, thanks @Carlos Alberto da Costa Filho! the updated docs are now live

amazing! thanks