https://prefect.io logo
Join Slack
Powered by
Hi all, I've been using Prefect Cloud for almost a...
# prefect-cloud
m
Hi all, I've been using Prefect Cloud for almost a year and a half now, with almost no troubles - so kudos for that 🙌 Over the last few days however, we have a bunch of flows stuck in 
Pending
state, and for the life of me I can't figure out why. The work pool is ready, the agent is healthy (running on EC2) and other subsequent flow runs are taking place on the same infrastructure. Can anybody help/ suggest where I should investigate please?
Copy code
Apr 12 00:59:51 blah.eu-west-1.compute.internal prefect[2122]: 00:59:51.824 | INFO    | prefect.agent - Submitting flow run '6d673031-0a18-4e81-8738-7ced55e4dfc3'
Apr 12 00:59:51 blah.eu-west-1.compute.internal prefect[2122]: 00:59:51.826 | INFO    | prefect.agent - Submitting flow run '9bbe21f3-2e46-4051-9238-e3eadc705d4d'
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: 01:00:53.881 | ERROR   | prefect.agent - Failed to update state of flow run '6d673031-0a18-4e81-8738-7ced55e4dfc3'
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: Traceback (most recent call last):
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/h2/connection.py", line 224, in process_input
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: func, target_state = self._transitions[(self.state, input_)]
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: During handling of the above exception, another exception occurred:
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: Traceback (most recent call last):
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpcore/_async/http2.py", line 106, in handle_async_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: await self._send_request_headers(request=request, stream_id=stream_id)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpcore/_async/http2.py", line 203, in _send_request_headers
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/h2/connection.py", line 766, in send_headers
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/h2/connection.py", line 229, in process_input
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: "Invalid input %s in state %s" % (input_, old_state)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: During handling of the above exception, another exception occurred:
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: Traceback (most recent call last):
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: yield
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: resp = await self._pool.handle_async_request(req)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: raise exc
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: response = await connection.handle_async_request(request)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: return await self._connection.handle_async_request(request)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpcore/_async/http2.py", line 142, in handle_async_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: raise LocalProtocolError(exc)  # pragma: nocover
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: The above exception was the direct cause of the following exception:
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: Traceback (most recent call last):
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/prefect/agent.py", line 466, in _propose_pending_state
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: state = await propose_state(self.client, Pending(), flow_run_id=flow_run.id)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/prefect/engine.py", line 1785, in propose_state
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: force=force,
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/prefect/client/orion.py", line 1625, in set_flow_run_state
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: json=dict(state=state_create.dict(json_compatible=True), force=force),
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_client.py", line 1861, in post
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: extensions=extensions,
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_client.py", line 1533, in request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: return await self.send(request, auth=auth, follow_redirects=follow_redirects)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/prefect/client/base.py", line 245, in send
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: httpx.LocalProtocolError,
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/prefect/client/base.py", line 187, in _send_with_retry
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: response = await request()
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_client.py", line 1624, in send
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: history=[],
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_client.py", line 1651, in _send_handling_auth
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: history=history,
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: response = await self._send_single_request(request)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_client.py", line 1722, in _send_single_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: response = await transport.handle_async_request(request)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: resp = await self._pool.handle_async_request(req)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/usr/lib64/python3.7/contextlib.py", line 130, in __exit__
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: self.gen.throw(type, value, traceback)
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: File "/home/prefect/venv/lib64/python3.7/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: raise mapped_exc(message) from exc
Apr 12 01:00:53 blah.eu-west-1.compute.internal prefect[2122]: httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Here's an exception that occurred in the agent logs at the same time as a flow was scheduled to run, but instead got stuck in pending state
we're running a pretty old version of the prefect agent
Copy code
(venv) [prefect@.. ~]$ prefect --version
2.7.8
3 Views
Open in Slack
PreviousNext
Powered by