<@ULVA73B9P> We are investigating prefect-ray inte...
# ask-marvin
m
@Marvin We are investigating prefect-ray integration in our current workflows. One issue we're having is that the head node needs to have the same images or dependencies as the workers, which appears to be a non started because the workers can install packages at run time, making this solution untenable (complicated ci, deployment, etc). Is it true that the running the following won't actually install them on the head node???
Copy code
@flow(task_runner=RayTaskRunner(
    address="<ray://your-cluster-address>",
    init_kwargs={
        "runtime_env": {
            "pip": ["pandas", "requests", "scikit-learn"], # List all top-level imports here
            "env_vars": {"PYTHONPATH": "."}
        }
    }
))
def my_flow():
    print(f"using prefect-ray tasks: Shouting from {socket.gethostname()}: {os.getpid()}")
    shout_task.map(range(10)).wait()
    print("Shouting done")
m
thought for 158.1 seconds
Short answer: runtime_env.pip does not install anything on your local driver process. It is applied by Ray on the cluster side for the job. Those packages will be prepared on any Ray node that actually runs your tasks/actors for that job — which can include the head node if Ray schedules work there. If the head node never gets tasks, it won’t install there. Key points for Prefect 2.x + Ray: - RayTaskRunner passes init_kwargs straight through to ray.init(...). The runtime_env you set is Ray’s, not Prefect’s. - runtime_env.pip/conda/etc are installed into a Ray-managed environment on nodes where your job runs tasks, not into your local flow process. - Your local flow (driver) still needs any packages it imports at flow scope. Put heavy/worker-only imports inside tasks to avoid requiring them on the driver. - If you don’t want anything installed on the head node, prevent it from scheduling tasks (e.g., start head with 0 CPUs or set placement/resources so tasks only land on worker nodes). About your snippet: the idea is right, but note that in Prefect 2.x you’ll typically submit tasks rather than use .map/.wait. Here’s a minimal 2.x example that mirrors your intent:
Copy code
from prefect import flow, task
from prefect_ray.task_runners import RayTaskRunner

@task
def shout_task(i):
    import socket, os
    return f"using prefect-ray tasks: Shouting from {socket.gethostname()}: {os.getpid()} - {i}"

@flow(
    task_runner=RayTaskRunner(
        address="ray://<your-cluster-address>:10001",
        init_kwargs={
            "runtime_env": {
                "pip": ["pandas", "requests", "scikit-learn"],
                # Often preferable to rely on working_dir/py_modules instead of PYTHONPATH
                # "working_dir": ".",  # Upload your project directory
                "env_vars": {"PYTHONPATH": "."}
            }
        }
    )
)
def my_flow():
    futures = [shout_task.submit(i) for i in range(10)]
    return [f.result() for f in futures]

if __name__ == "__main__":
    my_flow()
Tips to keep CI/deploy sane: - Avoid installing on the driver: keep worker-only imports inside tasks. - If you want zero installs on the head node, don’t let it schedule tasks: - Start head with no resources:
ray start --head --num-cpus=0
- Or in Ray cluster config, set head node CPU to 0 or use resource tags and require them on tasks. - For faster, reproducible prod runs, pre-bake a common base image for cluster nodes and use runtime_env only for small deltas (or pin exact versions in runtime_env for determinism). - Prefer
runtime_env["working_dir"]
or
py_modules
to ship your code to the cluster, instead of relying on
PYTHONPATH
. If you can share: - How you’re running Ray (VMs vs K8s, Ray operator, Docker images), - Whether the head is allowed to run tasks today, - Whether your flow imports those libraries at flow scope, …I can suggest the most direct setup to avoid installs on the head and keep environments predictable while you’re still on Prefect 2.x.