Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi All, With prefect 2.4.0, prefect-dask 0.2.0 I am receiving Internal server error and these errors are random, sometime the flow works fine next time flow crashes with `internal server error`. Other errors that appears from agent logs are, `RunTime error: The connection pool was closed while 9 HTTP requests/responses were still in flight.`

`2022-09-20 23:54:20,960 - distributed.client - ERROR -` 
`Traceback (most recent call last):`
  `File "/usr/local/lib/python3.10/site-packages/distributed/client.py", line 1527, in _close`
    `await self.scheduler_comm.close()`
`asyncio.exceptions.CancelledError`
`23:54:20.968 | ERROR   | Flow run 'nickel-pronghorn' - Crash detected! Execution was interrupted by an unexpected exception.`
`23:54:20.970 | DEBUG   | Flow run 'nickel-pronghorn' - Crash details:`
`Traceback (most recent call last):`
  `File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1345, in report_flow_run_crashes`
    `yield`
  `File "/usr/local/lib/python3.10/contextlib.py", line 697, in __aexit__`
    `cb_suppress = await cb(*exc_details)`
  `File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__`
    `raise exceptions[0]`
  `File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 939, in create_task_run_then_submit`
    `task_run = await create_task_run(`
  `File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 984, in create_task_run`
    `task_run = await flow_run_context.client.create_task_run(`
  `File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1721, in create_task_run`
    `response = await <http://self._client.post|self._client.post>(`
  `File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post`
    `return await self.request(`
  `File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request`
    `return await self.send(request, auth=auth, follow_redirects=follow_redirects)`
  `File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 279, in send`
    `response.raise_for_status()`
  `File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 225, in raise_for_status`
    `raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__`
`prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://prefect-server:4200/api/task_runs/>'`
`Response: {'exception_message': 'Internal Server Error'}`
`For more information check: <https://httpstatuses.com/500>`

Hey <@U03C7MF5LGM> We have an internal ticket open to track these intermittent internal server errors through the Cloud API. Would you be able to provide minimum example of the flow you're running as well as the deployment?

<@U03B84QJ78A> I am using local server with dask task runner in docker containers. I don't find an issue with prefect 2.0.4 and flows run fine same code but prefect 2.4 upgrade I find my flows crashing randomly with these errors. I think it might be a bug in prefect-dask. I have downgraded to prefect 2.04 and so far no errors, will provide a minimal example soon. Thanks:slightly_smiling_face: