Hi I m seeing a strange issue when using Ray inside a flow t

Question

Hi I m seeing a strange issue when using Ray inside a flow that doesn t use `RayTaskRunner` Ray is initialized from within a `task` and an `ActorPool` is used to run some logic and then another `task` runs which may take a few minutes to run After the task using Ray completes after a minute or so the entire flow crashes with the following error ```gcs rpc client h 533 Failed to connect to GCS within 60 seconds GCS may have been killed It s either GCS is terminated by `ray stop` or is killed unexpectedly If it is killed unexpectedly see the log file gcs server out The program will terminate ``` I have a minimal example here Saving that to a file `ray test py` and running `python ray test py` should reproduce the issue I see the issue on Ubuntu 18 04 but not on Mac

Aric Huang · Answer

This is using `ray==2 4 0` which should be the latest version

Aric Huang · Answer

I switched to using `RayTaskRunner` and it doesn t crash anymore but it doesn t seem like it should be required to use `RayTaskRunner` to use Ray within a task

Aric Huang · Answer

One thing I found is that when running the example flow when the task using Ray runs a `gcs server` process starts which is expected ```$ ps aux | grep gcs server aric 7796 2 8 0 0 806636 27740 pts 6 Sl+ 19 55 0 00 opt pyenv versions 3 8 12 lib python3 8 site packages ray core src ray gcs gcs server redis address=10 128 0 80 redis port=6379 gcs server port=0 metrics agent port=65472 node ip address=10 128 0 80 ```

Aric Huang · Answer

However as soon as the task completes the `gcs server` process is listed as `defunct` ```$ ps aux | grep gcs server aric 7796 2 5 0 0 0 0 pts 6 Z+ 19 55 0 00 gcs server lt defunct gt ```

Aric Huang · Answer

Any thoughts on why this is happening This came up when porting an existing flow from Prefect V1 to V2 The V1 version of the flow works fine with `ray==2 4 0` on the same Ubuntu 18 04 hosts so something seems to be different in V2

Aric Huang · Answer

Equivalent code using regular Python functions does not have this issue using this minimal example <https gist github com concreted 37d039f6184add94e912a08322140cc3>

Aric Huang · Answer

^^ When running that script the `gcs server` process stays active after the function using Ray returns

Aric Huang · Answer

My understanding from Ray documentation is that `gcs server` is started automatically when Ray is initialized and should be automatically shutdown when the Python process exits <https docs ray io en latest ray core api doc ray shutdown html> ```This will automatically run at the end when a Python process that uses Ray exits``` However when running with Prefect it seems to be getting terminated prematurely

Jacob Danovitch · Answer

I m not 100 what the origin of the issue is though maybe it has something to do with whatever magic async stuff Prefect does behind the scenes but the reason that it works with `RayTaskRunner` is because it calls `ray init ` before running your tasks which you don t If you add it to your script like this ```if name == main ray init ray test ``` Or like this ``` flow log prints=True def ray test ray init test sleep ``` It doesn t crash Tested on Ubuntu 20 04 5 LTS with `prefect==2 10 4` and `ray==2 3 0` the crash is reproducible without `ray init ` and disappears with either of the two fixes above