Hi Prefect experts! I use prefect cloud, with my w...
# ask-community
p
Hi Prefect experts! I use prefect cloud, with my work pools in AWS Fargate. After upgrading to Prefect 3, I started getting the following errors after each flow run:
Copy code
ERROR | QueueingLogExporter - Failed to export batch: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Traceback (most recent call last):
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
                ^^^^^^^^^^^^^^^^^^^
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
              ^^^^^^^^^^^^^^^^^^
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse
    httplib_response = super().getresponse()
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/usr/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:
...

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/prefect/telemetry/services.py", line 35, in _handle_batch
    self._otlp_exporter.export(items)
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 167, in export
    resp = self._export(serialized_data)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 136, in _export
    return <http://self._session.post|self._session.post>(
    ^^^^^^^^^^^^^^^^^^^
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/repo-root/apps/logging_detection/.venv/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service <prefect.telemetry.services.QueueingLogExporter object at 0x7f0964d43c80> failed to process batch of size 1
However, my flow finishes with success. I get these logs in my prefect worker logs, and in my flow run logs in AWS CloudWatch, but not in the flow run logs in Prefect. Any hints would be appreciated!
b
Hey Progre55 (sick username btw)! This is the first time I'm seeing an error like this. 👀 Would you be willing to open a GitHub issue?