https://prefect.io logo
Join Slack
Powered by
Hi , i have an bash script which starts prefect wo...
# prefect-cloud
r
Hi , i have an bash script which starts prefect worker queues (prefect agent start -q "name" ) every hour and for some reason one of the agent says its unhealthy most of the time , any reasons what could the issues? (my understanding was there was no responce from that queue but i am more curious as to whats causing this issue and that specific queue alone!) and one of my flow runs crashed and it threw and 500 responce error , it says it was internal server error any leads on this would help! (I am using prefect 2 with server running on amazon ec2 instance and i use prefect cloud if that helps) , thanks in advance
👀 1
was going through the logs and found this , ERROR | prefect.agent - Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED , h2.exceptions.ProtocolError: Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
m
Hey @Revanth, nothing immediately comes to mind here but would you mind providing some more details about the flow that triggered this error, what task runner are you using? How is the flow being triggered, i.e. is this running on a schedule or is it being triggered as part of a different flow run? do you have an example of the flow code?
r
hey mason thanks for reaching out , i am using ConcurrentTaskRunner and the flow is triggered through schedule and i am using Deployment.build_from_flow and passing in the function
the error raises when i run this bash script , prefect agent start -q [work queue 1] & prefect agent start -q [work queue 2] & prefect agent start -q [work queue 3] && fg , it throws the error which i mentioned in the messages
Agent started! Looking for work from queue(s): user-analytics-computation-queue...
073728.146 | ERROR | prefect.agent - Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED Traceback (most recent call last): File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input func, target_state = self.transitions[(self.state, input)] KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.RECV_PING: 14>) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 114, in handle_async_request status, headers = await self._receive_response( File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 231, in _receive_response event = await self._receive_stream_event(request, stream_id) File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 262, in _receive_stream_event await self._receive_events(request, stream_id) File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 283, in _receive_events events = await self._read_incoming_data(request) File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 345, in _read_incoming_data events = self._h2_state.receive_data(data) File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 1463, in receive_data events.extend(self._receive_frame(frame)) File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 1487, in _receive_frame frames, events = self._frame_dispatch_table[frame.class](frame) File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 1760, in _receive_ping_frame events = self.state_machine.process_input( File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input raise ProtocolError( h2.exceptions.ProtocolError: Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions yield File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request raise exc File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request raise LocalProtocolError(exc) 
# pragma: nocover
 
httpcore.LocalProtocolError: Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
queue_runs = await self.client.get_runs_in_work_queue(
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
response = await <http://self._client.post|self._client.post>(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
await super().send(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
07:37:32.732 | ERROR   | prefect.agent - Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.RECV_PING: 14>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 114, in handle_async_request
status, headers = await self._receive_response(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 231, in _receive_response
event = await self._receive_stream_event(request, stream_id)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 262, in _receive_stream_event
await self._receive_events(request, stream_id)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 283, in _receive_events
events = await self._read_incoming_data(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 345, in _read_incoming_data
events = self._h2_state.receive_data(data)
File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 1463, in receive_data
events.extend(self._receive_frame(frame))
File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 1487, in _receive_frame
frames, events = self._frame_dispatch_table[frame.__class__](frame)
File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 1760, in _receive_ping_frame
events = self.state_machine.process_input(
File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
raise LocalProtocolError(exc)  _# pragma: nocover_
httpcore.LocalProtocolError: Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
queue_runs = await self.client.get_runs_in_work_queue(
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
response = await <http://self._client.post|self._client.post>(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
await super().send(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
07:37:33.169 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
🙌 1
m
Brilliant, Yeah we seen this come up a few times recently and this definitely seems like it could be related 😄
r
not really sure about issue tried reading through bunch of open issue on github but all of them went over my head! , for an work around wrote few bash scripts which would start the work queue when the flow was supposed to run and pause the work queue after a while , i am hopping this would help me solve the issue , any other methods would you recommend?
m
For clarification you're not starting an agent as part of the flow run correct?
r
not part of the flow run
i am using CronSchedule to schedule my flows , the bash scripts are to start the agent when the flow was supposed to run , example let us i have scheduled my flow run at 12 am what i am currently doing is running an bash script (prefect agent start -q work_queue_name) at 11:55 and assumming the flow ends in 10 minutes , so after 10 minutes i am running an another bash script (prefect work-queue pause work_queue_name) , hope i gave some useful info , sorry english isnt my first language!
m
Hey @Revanth I think this might be related to this issue https://github.com/PrefectHQ/prefect/issues/7442
6 Views
Open in Slack
PreviousNext
Powered by