<@ULVA73B9P> I am seeing this in my orchestrator a...
# ask-marvin
j
@Marvin I am seeing this in my orchestrator after i spin up a bunch of cloud job worker deployments...should i be worried?
Copy code
nsertId: "6926c7cc0001a1ede3b68210"
labels: {4}
logName: "projects/asdf-prod/logs/run.googleapis.com%2Fstderr"
payload: "textPayload"
receiveLocation: "us-east1"
receiveTimestamp: "2025-11-26T09:26:36.189588891Z"
resource: {2}
severity: "ERROR"
textPayload: "Traceback (most recent call last):
  File "/opt/prefect/.venv/lib/python3.12/site-packages/prefect/client/base.py", line 268, in _send_with_retry
    response = await send(request, *send_args, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/opt/prefect/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response."
timestamp: "2025-11-26T09:26:36.106989Z"
traceSampled: false
}
m
thought for 297.5 seconds
Short answer: probably not. That error usually indicates a transient network hiccup where the server closed the connection before sending a response. Prefect’s client treats this as a retryable condition and will automatically retry with exponential backoff. If your flow runs and state updates are succeeding, the occasional stack trace during bursts (like spinning up many workers) is typically harmless. What it is - httpx.RemoteProtocolError = the server dropped the connection mid-request. - Common during traffic spikes, load balancer resets, or brief API unavailability. - Prefect’s client automatically retries these exceptions. How Prefect handles it (Prefect 3.x) - Retries are built-in for RemoteProtocolError and other transient errors with exponential backoff + jitter. - Defaults you can tune via env vars: -
PREFECT_API_REQUEST_TIMEOUT
(seconds, default 60) -
PREFECT_CLIENT_MAX_RETRIES
(default 5) -
PREFECT_CLIENT_RETRY_JITTER_FACTOR
(default 0.2) -
PREFECT_CLIENT_RETRY_EXTRA_CODES
(extra HTTP status codes to retry) If you want to make the client more tolerant to brief blips
Copy code
# longer per-request timeout if network/API is occasionally slow
export PREFECT_API_REQUEST_TIMEOUT=120

# allow more attempts during spikes
export PREFECT_CLIENT_MAX_RETRIES=10
To confirm it’s being retried (and not fatal)
Copy code
# temporarily increase logging detail
export PREFECT_LOGGING_LEVEL=DEBUG
export PREFECT_DEBUG_MODE=true
Relevant docs - Settings reference (timeouts/retries/logging): Prefect 3 Settings Reference - Managing settings/how to set env vars: Manage Settings - Adding logging: Add Logging - Workers overview: Workers Concept - GCP worker guide: GCP Worker Guide Note on GCP worker types - In Prefect 3.x the Cloud Run worker types are
cloud-run
(v1 API) and
cloud-run-v2
(v2 API). There is no
cloud-run-job
worker type. If you’re using Cloud Run Jobs, use one of those two types (we generally recommend
cloud-run-v2
for new setups). When to worry - You see sustained waves of these errors accompanied by failed flow runs or missed state updates. - They repeat past your configured
PREFECT_CLIENT_MAX_RETRIES
. - You’re self-hosting the API and also seeing high CPU/connection resets—then consider scaling your API (e.g., Cloud Run max instances/concurrency), or increasing request timeout on the platform side. If you can share: - Are any flow runs failing, or is this just log noise? - Prefect version, and whether you’re on Prefect Cloud or self-hosted (Cloud Run logs suggest self-hosted)? I can tailor next steps (timeouts, retries, or scaling knobs) based on that.