Mitch
04/01/2026, 5:06 PM@flow(task_runner=RayTaskRunner(
address="<ray://your-cluster-address>",
init_kwargs={
"runtime_env": {
"pip": ["pandas", "requests", "scikit-learn"], # List all top-level imports here
"env_vars": {"PYTHONPATH": "."}
}
}
))
def my_flow():
print(f"using prefect-ray tasks: Shouting from {socket.gethostname()}: {os.getpid()}")
shout_task.map(range(10)).wait()
print("Shouting done")Marvin
04/01/2026, 5:06 PMMarvin
04/01/2026, 5:09 PMfrom prefect import flow, task
from prefect_ray.task_runners import RayTaskRunner
@task
def shout_task(i):
import socket, os
return f"using prefect-ray tasks: Shouting from {socket.gethostname()}: {os.getpid()} - {i}"
@flow(
task_runner=RayTaskRunner(
address="ray://<your-cluster-address>:10001",
init_kwargs={
"runtime_env": {
"pip": ["pandas", "requests", "scikit-learn"],
# Often preferable to rely on working_dir/py_modules instead of PYTHONPATH
# "working_dir": ".", # Upload your project directory
"env_vars": {"PYTHONPATH": "."}
}
}
)
)
def my_flow():
futures = [shout_task.submit(i) for i in range(10)]
return [f.result() for f in futures]
if __name__ == "__main__":
my_flow()
Tips to keep CI/deploy sane:
- Avoid installing on the driver: keep worker-only imports inside tasks.
- If you want zero installs on the head node, don’t let it schedule tasks:
- Start head with no resources: ray start --head --num-cpus=0
- Or in Ray cluster config, set head node CPU to 0 or use resource tags and require them on tasks.
- For faster, reproducible prod runs, pre-bake a common base image for cluster nodes and use runtime_env only for small deltas (or pin exact versions in runtime_env for determinism).
- Prefer runtime_env["working_dir"] or py_modules to ship your code to the cluster, instead of relying on PYTHONPATH.
If you can share:
- How you’re running Ray (VMs vs K8s, Ray operator, Docker images),
- Whether the head is allowed to run tasks today,
- Whether your flow imports those libraries at flow scope,
…I can suggest the most direct setup to avoid installs on the head and keep environments predictable while you’re still on Prefect 2.x.