xposting: kube worker is losing connection to server https://prefect-community.slack.com/archives/C048K0MGHNK/p1682211404185689 here are some logs, where the last one is just

ConnectionTerminated error_code:0

Worker 'KubernetesWorker 3fe10498-8e37-4bca-bc38-f1adf48ee842' started!
last_stream_id:19999, additional_data:None>
httpx.RemoteProtocolError: <ConnectionTerminated error_code:0,
raise mapped_exc(message) from exc
line 77, in map_httpcore_exceptions
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py",
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
with map_httpcore_exceptions():
line 352, in handle_async_request
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[truncated...]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
runs_response = await self._get_scheduled_flow_runs()
372, in get_and_submit_flow_runs
File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line
await workload()
line 46, in critical_service_loop
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/services.py",
Traceback (most recent call last):
The above exception was the direct cause of the following exception:
last_stream_id:19999, additional_data:None>
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:0,
raise RemoteProtocolError(self._connection_error_event)
139, in handle_async_request
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http2.py", line
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[truncated...]
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py",
yield
line 60, in map_httpcore_exceptions
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py",
Traceback (most recent call last):
During handling of the above exception, another exception occurred:
state ConnectionState.CLOSED
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in
raise ProtocolError(
process_input
[truncated...]
During handling of the above exception, another exception occurred:
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
func, target_state = self._transitions[(self.state, input_)]
process_input
File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 224, in
Traceback (most recent call last):
Examples of recent errors:
Failed the last 3 attempts. Please check your environment and configuration.
Unable to load step function: {fully_qualified_name}. Attempting install of {requires}.

That looks similar, if not quite the same as a problem I was having, I ended up adding

PREFECT_API_ENABLE_HTTP2=False

to the environment to disable the newer http library and I haven't had the problem over the last month or so

thank you, I'll try it!