Hi All, With prefect 2.4.0, prefect-dask 0.2.0 I a...
# prefect-community
Hi All, With prefect 2.4.0, prefect-dask 0.2.0 I am receiving Internal server error and these errors are random, sometime the flow works fine next time flow crashes with
internal server error
. Other errors that appears from agent logs are,
RunTime error: The connection pool was closed while 9 HTTP requests/responses were still in flight.
2022-09-20 23:54:20,960 - distributed.client - ERROR -
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/distributed/client.py", line 1527, in _close
await self.scheduler_comm.close()
23:54:20.968 | ERROR   | Flow run 'nickel-pronghorn' - Crash detected! Execution was interrupted by an unexpected exception.
23:54:20.970 | DEBUG   | Flow run 'nickel-pronghorn' - Crash details:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1345, in report_flow_run_crashes
File "/usr/local/lib/python3.10/contextlib.py", line 697, in __aexit__
cb_suppress = await cb(*exc_details)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 939, in create_task_run_then_submit
task_run = await create_task_run(
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 984, in create_task_run
task_run = await flow_run_context.client.create_task_run(
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1721, in create_task_run
response = await <http://self._client.post|self._client.post>(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 279, in send
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 225, in raise_for_status
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://prefect-server:4200/api/task_runs/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
Hey @Faheem Khan We have an internal ticket open to track these intermittent internal server errors through the Cloud API. Would you be able to provide minimum example of the flow you're running as well as the deployment?
@Mason Menges I am using local server with dask task runner in docker containers. I don't find an issue with prefect 2.0.4 and flows run fine same code but prefect 2.4 upgrade I find my flows crashing randomly with these errors. I think it might be a bug in prefect-dask. I have downgraded to prefect 2.04 and so far no errors, will provide a minimal example soon. Thanks🙂