https://prefect.io logo
Title
s

Slackbot

12/14/2022, 12:53 PM
This message was deleted.
d

Deepanshu Aggarwal

12/14/2022, 1:21 PM
Encountered exception during execution: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions yield File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request raise exc File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request raise exc File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 205, in _receive_event raise RemoteProtocolError(msg) httpcore.RemoteProtocolError: Server disconnected without sending a response. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1247, in orchestrate_task_run result = await run_sync(task.fn, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread return await anyio.to_thread.run_sync(call, cancellable=True) File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/opt/prefect/flows/process_sso_integrations_flows/sso_flow.py", line 85, in sso_token_executor output = run_deployment( File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 197, in coroutine_wrapper return run_async_from_worker_thread(async_fn, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 148, in run_async_from_worker_thread return anyio.from_thread.run(call) File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run return asynclib.run_async_from_thread(func, *args) File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread return f.result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 131, in run_deployment flow_run = await client.read_flow_run(flow_run_id) File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 1443, in read_flow_run response = await self._client.get(f"/flow_runs/{flow_run_id}") File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get return await self.request( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 160, in send await super().send(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send response = await self._send_handling_auth( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth response = await self._send_handling_redirects( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects response = await self._send_single_request(request) File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request response = await transport.handle_async_request(request) File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request with map_httpcore_exceptions(): File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.RemoteProtocolError: Server disconnected without sending a response.
c

Christopher Boyd

12/14/2022, 3:21 PM
what’s the server in this case?
Also, would you be able to move your tracebacks into the thread ?
d

Deepanshu Aggarwal

12/14/2022, 4:48 PM
i think i deleted it by mistake
Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 916, in write
    n = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 33, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1274, in receive
    raise self._protocol.exception
anyio.BrokenResourceError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 30, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadError

The above exception was the direct cause of the following exception:

httpx.ReadError
im using aws eks to host my orion server with our own postgresql
c

Christopher Boyd

12/15/2022, 1:29 PM
that was a concern I had, that disconnected by server says that it was not a graceful disconnect - possibly too many connections are being opened, or a connection being opened and not responded to, so the server disconnects the remote end forcefully (e.g. with a RST instead of a FIN)
d

Deepanshu Aggarwal

12/15/2022, 1:40 PM
yes , we are trying to make a lot of connections . but is there a work around for it ? any method to maintain a large number of connections ?
c

Christopher Boyd

12/15/2022, 2:19 PM
that would be at your database
you would either need to be able to handle many connections at the database, ensure that you aren’t opening and not closing connections at the flow, and potentially organizing / scheduling them in a way that distributes the connections
also, are you connecting from inside the same cluster, to the service of postgres, or outbound to an external source that is not located in the same cluster?
d

Deepanshu Aggarwal

12/15/2022, 2:23 PM
its outside the cluster . we have postgre running on rds but inside the same virtual network
how to close connections at the flow ?