Laksh Aithani
05/07/2022, 7:16 PMfrom prefect import flow, task
from prefect.task_runners import RayTaskRunner
@task(retries=3, retry_delay_seconds=5)
def say_hello(name):
print(f"hello {name}")
kwargs = dict() # works on my Mac
kwargs = dict(address="ray://**.***.**.***:10001", init_kwargs=dict(runtime_env={"pip": []})) # fails on my Mac, works on Ray head node
@flow(task_runner=RayTaskRunner(**kwargs))
def greetings(names):
for name in names:
say_hello(name)
if __name__ == "__main__":
greetings(["arthur", "trillian", "ford", "marvin"])
when running the this flow on my Mac and not connecting to the remote client but spinning up a local Ray cluster, it works. When trying to use the remote client, it fails when on my Mac with the following error:
19:38:37.575 | INFO | Task run 'say_hello-d71d0552-0' - Crash detected! Execution was interrupted by an unexpected exception.
19:38:37.616 | INFO | Task run 'say_hello-d71d0552-1' - Crash detected! Execution was interrupted by an unexpected exception.
19:38:37.628 | INFO | Task run 'say_hello-d71d0552-2' - Crash detected! Execution was interrupted by an unexpected exception.
19:38:37.641 | INFO | Task run 'say_hello-d71d0552-3' - Crash detected! Execution was interrupted by an unexpected exception.
weirdly then, when on the head node of the ray cluster, but still connecting to the ray cluster with the remote client connection, it works.
Anyone know why this may be the case? (edited)Kevin Kho
05/07/2022, 9:56 PMAnna Geller
05/07/2022, 11:44 PMLaksh Aithani
05/08/2022, 5:48 PM