Joseph Wibowo
11/26/2025, 9:30 AMnsertId: "6926c7cc0001a1ede3b68210"
labels: {4}
logName: "projects/asdf-prod/logs/run.googleapis.com%2Fstderr"
payload: "textPayload"
receiveLocation: "us-east1"
receiveTimestamp: "2025-11-26T09:26:36.189588891Z"
resource: {2}
severity: "ERROR"
textPayload: "Traceback (most recent call last):
File "/opt/prefect/.venv/lib/python3.12/site-packages/prefect/client/base.py", line 268, in _send_with_retry
response = await send(request, *send_args, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
with map_httpcore_exceptions():
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response."
timestamp: "2025-11-26T09:26:36.106989Z"
traceSampled: false
}Marvin
11/26/2025, 9:30 AMMarvin
11/26/2025, 9:35 AMPREFECT_API_REQUEST_TIMEOUT (seconds, default 60)
- PREFECT_CLIENT_MAX_RETRIES (default 5)
- PREFECT_CLIENT_RETRY_JITTER_FACTOR (default 0.2)
- PREFECT_CLIENT_RETRY_EXTRA_CODES (extra HTTP status codes to retry)
If you want to make the client more tolerant to brief blips
# longer per-request timeout if network/API is occasionally slow
export PREFECT_API_REQUEST_TIMEOUT=120
# allow more attempts during spikes
export PREFECT_CLIENT_MAX_RETRIES=10
To confirm it’s being retried (and not fatal)
# temporarily increase logging detail
export PREFECT_LOGGING_LEVEL=DEBUG
export PREFECT_DEBUG_MODE=true
Relevant docs
- Settings reference (timeouts/retries/logging): Prefect 3 Settings Reference
- Managing settings/how to set env vars: Manage Settings
- Adding logging: Add Logging
- Workers overview: Workers Concept
- GCP worker guide: GCP Worker Guide
Note on GCP worker types
- In Prefect 3.x the Cloud Run worker types are cloud-run (v1 API) and cloud-run-v2 (v2 API). There is no cloud-run-job worker type. If you’re using Cloud Run Jobs, use one of those two types (we generally recommend cloud-run-v2 for new setups).
When to worry
- You see sustained waves of these errors accompanied by failed flow runs or missed state updates.
- They repeat past your configured PREFECT_CLIENT_MAX_RETRIES.
- You’re self-hosting the API and also seeing high CPU/connection resets—then consider scaling your API (e.g., Cloud Run max instances/concurrency), or increasing request timeout on the platform side.
If you can share:
- Are any flow runs failing, or is this just log noise?
- Prefect version, and whether you’re on Prefect Cloud or self-hosted (Cloud Run logs suggest self-hosted)?
I can tailor next steps (timeouts, retries, or scaling knobs) based on that.