https://prefect.io logo
Join Slack
Powered by
Hi Prefects, One of my flows crashed with an `http...
# ask-community
t
Hi Prefects, One of my flows crashed with an 
httpx.WriteError
. After running for about 1h and completing 371 tasks the flow entered a crashed state with the following trace:
👀 1
Copy code
Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last):
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/asyncio/selector_events.py", line 916, in write
    n = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 49, in write
    await self._stream.send(item=buffer)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in send
    await self._call_sslobject_method(self._ssl_object.write, item)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/anyio/streams/tls.py", line 168, in _call_sslobject_method
    await self.transport_stream.send(self._write_bio.read())
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1297, in send
    raise self._protocol.exception
anyio.BrokenResourceError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 313, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/http2.py", line 146, in handle_async_request
    raise exc
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/http2.py", line 110, in handle_async_request
    await self._send_request_body(request=request, stream_id=stream_id)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/http2.py", line 223, in _send_request_body
    await self._write_outgoing_data(request)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/http2.py", line 372, in _write_outgoing_data
    raise exc
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_async/http2.py", line 360, in _write_outgoing_data
    await self._network_stream.write(data_to_send, timeout)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 49, in write
    await self._stream.send(item=buffer)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/path/to/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.WriteError

The above exception was the direct cause of the following exception:

httpx.WriteError
Just checked the agent logs and this error seems to happen from time to time. However, this is the first time that it actually crashed a flow.
b
Hi Tim, thanks for reporting this here. What version of Prefect are you using by chance?
Also, would mind describing what the tasks are doing? Are the tasks processing large amounts of data?
As a first troubleshooting step, you could try bumping the 
PREFECT_API_REQUEST_TIMEOUT
. You can preview the default for that value by running the following command in the CLI: 
prefect config view --show-defaults | grep TIMEOUT
You can adjust the value by running this line: 
prefect config set PREFECT_API_REQUEST_TIMEOUT='30'
Also, we recommend upgrading your Prefect version to 2.5.0 or above to benefit from the additional client stability fixes<https://github.com/PrefectHQ/prefect/pull/7090> which were made.
j
hello bianca, hello tim-oliver, were you able to find a mitigation strategy for this? i'm facing similar issues right now and increasing the api request timeout didn't help. what do you think about adding 
status.HTTP_408_REQUEST_TIMEOUT
to the list of retry codes in the base client?
8 Views
Open in Slack
PreviousNext
Powered by