I think the size of the object returned by one of my tasks i Prefect Community #ask-community

I think the size of the object returned by one of ...

Berislav Lopac

03/28/2024, 12:12 AM

I think the size of the object returned by one of my tasks is breaking Prefect... 🤔 I have an object that is around 23MB in size, and after the task completes the flow stops executing, and about 3 minutes the flow fails with the following error:

Copy code

00:09:42.549 | ERROR   | Flow run 'humongous-squid' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/http2.py", line 183, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
httpcore.LocalProtocolError: 1

Any ideas?

✅ 1

Berislav Lopac

03/28/2024, 12:13 AM

If I explicitly send a smaller object (a list, actually), the flow completes correctly.

Berislav Lopac

03/28/2024, 12:24 AM

Just found this discussion: https://github.com/PrefectHQ/prefect/issues/7442 and following the advice there disabled HTTP/2; now I'm getting a different error:

Copy code

00:22:39.903 | ERROR   | Flow run 'arrogant-boar' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/http11.py", line 143, in handle_async_request
    raise exc
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/http11.py", line 113, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/http11.py", line 186, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_async/http11.py", line 224, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 31, in read
    with map_exceptions(exc_map):
  File "/home/berislav/.pyenv/versions/3.11.8/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/berislav/.virtualenvs/dataflows/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

Berislav Lopac

03/28/2024, 10:17 AM

OK, it looks like this is caused by a very large result of a task; does anyone have any recommendations on how to handle large task results? 🤔

Berislav Lopac

03/28/2024, 10:19 AM

Ah, finally a sensible message:

Copy code

{'exception_message': 'Invalid request received.', 'exception_detail': [{'loc': ['body', 'parameters'], 'msg': 'Flow run parameters must be less than 512KB when serialized.', 'type': 'value_error'}

Jeff Hale

03/28/2024, 1:08 PM

Here’s a discussion of dealing with big data in the docs that might be helpful.

Berislav Lopac

03/28/2024, 2:30 PM

Ah thank you, missed that! thank you

Berislav Lopac

03/28/2024, 2:31 PM

Yeah, I pretty much solved my issues with option 3 😄

🙌 1

23 Views

Open in Slack

Previous Next