Justin Trautmann
10/04/2022, 3:15 PMCrash detected! Request to <https://api.prefect.cloud/api/accounts/[...]/workspaces/[...]/task_runs/> failed.
...
RuntimeError: The connection pool was closed while 65 HTTP requests/responses were still in-flight.
this is most likely not related to local network issues as it is reproducible across different networks. when using a local orion server instead, the flow succeeds without any issues and a couple of days ago, the flow ran successfully on prefect cloud.
I am submitting ~100 parallel tasks using the RayTaskRunner with a local cluster.
Any help is much appreciated.
python 3.8.10
prefect 2.4.5
prefect-ray 0.2.0.post2Toby Rahloff
10/04/2022, 4:16 PMprefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/2.../workspaces/.../task_runs/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
2022-10-04 18:13:47,715 - distributed.nanny - ERROR - Worker process died unexpectedly
2022-10-04 18:13:47,715 - distributed.nanny - ERROR - Worker process died unexpectedly
❯ prefect version
Version: 2.4.5
API version: 0.8.1
Python version: 3.8.10
Git commit: dbe27317
Built: Thu, Sep 29, 2022 3:05 PM
OS/Arch: linux/x86_64
Profile: dev
Server type: cloud
Zach Angell
10/05/2022, 1:09 PMToby Rahloff
10/05/2022, 1:55 PMFaheem Khan
10/05/2022, 11:33 PM