Amith M
08/09/2025, 5:50 AM.serve
deployment on gcp. I have everything setup and the deployment gets the job and reports it back to the prefect server, but it never starts the job, it goes to "pending" state. It then goes back for listening for jobs again. the worker/runner never starts.
and sometimes every other run, there is some form of httpx error in the logs.
i have attached the logsAmith M
08/09/2025, 5:51 AM2025-08-09 11:16:23.724 IST
05:46:23.624 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of '_get_and_submit_flow_runs'
2025-08-09 11:16:24.124 IST
05:46:23.825 | DEBUG | prefect.runner - Querying for flow runs scheduled before 2025-08-09 05:46:33.825421+00:00
2025-08-09 11:16:29.425 IST
05:46:28.724 | DEBUG | prefect.runner - Discovered 1 scheduled_flow_runs
2025-08-09 11:16:30.362 IST
05:46:29.624 | DEBUG | prefect.runner - Limit slot acquired for flow run '06896e0a-c2e0-763c-8000-49de11cfeac9'
2025-08-09 11:16:31.925 IST
05:46:30.362 | INFO | prefect.flow_runs.runner - Runner 'runner-bdc947cf-443d-4a8c-ae83-09e78c3fd564' submitting flow run '06896e0a-c2e0-763c-8000-49de11cfeac9'
2025-08-09 11:16:40.224 IST
05:46:38.724 | INFO | prefect.flow_runs.runner - Opening process...
2025-08-09 11:17:20.124 IST
05:47:17.826 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of '_get_and_submit_flow_runs'
2025-08-09 11:17:21.024 IST
05:47:20.125 | DEBUG | prefect.runner - Querying for flow runs scheduled before 2025-08-09 05:47:30.124994+00:00
2025-08-09 11:17:22.625 IST
05:47:22.624 | DEBUG | prefect.events.clients - Connection closed with 3/10 attempts
2025-08-09 11:17:24.824 IST
05:47:23.724 | DEBUG | prefect.events.clients - Reconnecting...
2025-08-09 11:17:27.825 IST
05:47:27.124 | INFO | prefect.flow_runs.runner - Completed submission of flow run '06896e0a-c2e0-763c-8000-49de11cfeac9'
2025-08-09 11:17:52.525 IST
05:47:40.024 | DEBUG | prefect.client - Encountered retryable exception during request. Another attempt will be made in 1.9134020393203932s. This is attempt 1/6.
2025-08-09 11:17:52.525 IST
Traceback (most recent call last): File "/opt/venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/opt/venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request resp = await self._pool.handle_async_request(req)
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
2025-08-09 11:17:52.525 IST
raise exc from None
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
2025-08-09 11:17:52.525 IST
response = await connection.handle_async_request(
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
2025-08-09 11:17:52.525 IST
return await self._connection.handle_async_request(request)
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
2025-08-09 11:17:52.525 IST
raise exc
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
2025-08-09 11:17:52.525 IST
) = await self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
2025-08-09 11:17:52.525 IST
event = await self._receive_event(timeout=timeout)
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 231, in _receive_event
2025-08-09 11:17:52.525 IST
raise RemoteProtocolError(msg)
2025-08-09 11:17:52.525 IST
httpcore.RemoteProtocolError: Server disconnected without sending a response.
2025-08-09 11:17:52.525 IST
The above exception was the direct cause of the following exception:
2025-08-09 11:17:52.525 IST
Traceback (most recent call last): File "/opt/venv/lib/python3.12/site-packages/prefect/client/base.py", line 268, in _send_with_retry response = await send(request, *send_args, **send_kwargs)
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
2025-08-09 11:17:52.525 IST
response = await self._send_handling_auth(
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
2025-08-09 11:17:52.525 IST
response = await self._send_single_request(request)
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
2025-08-09 11:17:52.525 IST
response = await transport.handle_async_request(request)
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/opt/venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
2025-08-09 11:17:52.525 IST
with map_httpcore_exceptions():
2025-08-09 11:17:52.525 IST
^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 11:17:52.525 IST
File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
2025-08-09 11:17:52.525 IST
self.gen.throw(value)
File "/opt/venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
2025-08-09 11:17:52.525 IST
raise mapped_exc(message) from exc
2025-08-09 11:17:53.558 IST
httpx.RemoteProtocolError: Server disconnected without sending a response.
2025-08-09 11:18:06.924 IST
05:48:06.124 | DEBUG | prefect.runner - Discovered 0 scheduled_flow_runs
2025-08-09 11:18:22.225 IST
05:48:20.925 | DEBUG | prefect.runner - Querying for flow runs scheduled before 2025-08-09 05:48:30.524753+00:00
2025-08-09 11:18:24.524 IST
05:48:24.425 | DEBUG | prefect.runner - Discovered 0 scheduled_flow_runs
2025-08-09 11:18:35.124 IST
05:48:34.624 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of '_get_and_submit_flow_runs'
2025-08-09 11:18:37.125 IST
05:48:35.424 | DEBUG | prefect.runner - Querying for flow runs scheduled before 2025-08-09 05:48:45.125197+00:00
2025-08-09 11:18:41.524 IST
05:48:41.226 | DEBUG | prefect.runner - Discovered 0 scheduled_flow_runs
Nate
08/09/2025, 7:41 PMNate
08/09/2025, 7:41 PMBrendan Dalpe
08/11/2025, 1:24 PMprefect.events.clients - Connection closed with 3/10 attempts
httpx.RemoteProtocolError: Server disconnected without sending a responseAgree with Nate. Do you have some kind of security tools between your GCP resource and the Internet? Looks like something is actively closing the connection. It's not "timing out" which would usually indicate there is a a different issue with lack of connectivity.
Amith M
08/11/2025, 5:49 PM