https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I keep getting errors that look like ...
# ask-marvin
j
@Marvin I keep getting errors that look like this:
Copy code
09:08:27.342 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 18.951345016723153s. This is attempt 4/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout
m
✅ thought for 290.4 seconds
j
@Marvin This is then followed by
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 268, in _send_with_retry
    response = await send(request, *send_args, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
m
✅ thought for 253.2 seconds
Thanks for sharing the trace — that’s Prefect’s client retrying a request after an HTTP read timeout from the API. This usually points to intermittent network slowness, a proxy/load balancer holding the connection too long, or an endpoint that’s taking longer than your current timeout allows. Assuming you’re on Prefect 3.x, here are pragmatic steps to reduce or eliminate these: What’s happening - httpcore.ReadTimeout: the HTTP response wasn’t received within the configured timeout. - Prefect logs “Encountered retryable exception… attempt X/Y” because the SDK will retry transient network failures by default. Quick fixes you can try 1) Increase the API request timeout - Default is 60s. Try bumping to 120–300s if you see timeouts on slow/loaded networks. - Set via profile/CLI:
Copy code
prefect config set PREFECT_API_REQUEST_TIMEOUT=120
- Or via env var:
Copy code
export PREFECT_API_REQUEST_TIMEOUT=120
2) Increase retry budget - Defaults to 5 retries; raise if your network is bursty:
Copy code
prefect config set PREFECT_CLIENT_MAX_RETRIES=8
- You can also expand which HTTP codes are retried if needed:
Copy code
prefect config set PREFECT_CLIENT_RETRY_EXTRA_CODES="429,503"
3) Optional: enable HTTP/2 (often better on lossy links; ensure your endpoint supports it) - Prefect Cloud does; many self-hosted LBs do as well.
Copy code
prefect config set PREFECT_API_ENABLE_HTTP2=true
4) Reduce API call volume from logging (if you log a lot) - Prefect batches logs every 2s but high-volume logs still mean frequent requests. You can: - Increase the interval:
Copy code
prefect config set PREFECT_LOGGING_TO_API_BATCH_INTERVAL=5
- Temporarily disable API logging (you’ll lose logs in the UI during that period):
Copy code
prefect config set PREFECT_LOGGING_TO_API_ENABLED=false
5) If you construct a client manually, pass HTTPX timeout settings directly - Useful for fine control of connect/read/write timeouts:
Copy code
import httpx
  from prefect.client.orchestration import PrefectClient

  timeout = httpx.Timeout(connect=30, read=180, write=30, pool=60)
  async with PrefectClient(
      api="<your PREFECT_API_URL>",
      api_key="<your PREFECT_API_KEY>",
      httpx_settings={"timeout": timeout},
  ) as client:
      ...
6) Network / proxy checks - If you’re behind a corporate proxy, ensure standard env vars are set: 
HTTPS_PROXY
, 
HTTP_PROXY
, and 
NO_PROXY
(include your API host in NO_PROXY if you bypass). - For self-hosted/API behind a load balancer: confirm upstream/proxy idle and read timeouts are ≥ your client timeout. - Confirm TLS/cert chain if using custom certs; for testing only, you can set:
Copy code
prefect config set PREFECT_API_TLS_INSECURE_SKIP_VERIFY=true
(Not recommended for production.) Useful references - Settings reference (all configurable settings, env vars, and profiles): Prefect 3 Settings Reference - Rate limits and retries context: Rate limits in Prefect Cloud - API keys (if you need to rotate/regenerate): Manage API Keys A couple of quick questions to help narrow it down: - Are you on Prefect Cloud or self-hosted? If self-hosted, is there a proxy/load balancer in front of the API? - Do these timeouts eventually succeed after retries, or do they fail jobs? - Where do you see them most (worker logs, flow run logs, deployment commands)? - Any proxies/firewalls in the environment (Kubernetes, Airflow, corporate network)? If you share those details, I can recommend more targeted settings or infra tweaks.
Thanks for sharing the traceback. An 
httpx.ReadTimeout
means the Prefect client didn’t receive a response from the API within the configured timeout window. In Prefect 3.x, the default API request timeout is 60 seconds. Quick mitigations - Increase the API request timeout (recommended first step) - Environment variable:
Copy code
export PREFECT_API_REQUEST_TIMEOUT=120
- Or set it in your Prefect profile:
Copy code
prefect config set PREFECT_API_REQUEST_TIMEOUT=120
- Increase client retry settings (helps with transient slowness) - You can set:
Copy code
export PREFECT_CLIENT_MAX_RETRIES=8
    export PREFECT_CLIENT_RETRY_JITTER_FACTOR=0.3
    # Optionally add extra HTTP codes to retry
    export PREFECT_CLIENT_RETRY_EXTRA_CODES=429,502,503,504
Per-process override (code) - If you want to raise the timeout for just one process or call, pass 
httpx_settings
to the Prefect client:
Copy code
from prefect.client.orchestration import get_client

  # Async usage
  async with get_client(httpx_settings={"timeout": 120}) as client:
      # make API calls that may be slow
      ...

  # Sync usage
  with get_client(httpx_settings={"timeout": 120}, sync_client=True) as client:
      ...
Verify your current settings
Copy code
prefect config view
You should see 
PREFECT_API_REQUEST_TIMEOUT
, 
PREFECT_CLIENT_MAX_RETRIES
, etc. in the output if set. Common causes and troubleshooting tips - Self-hosted behind a proxy/load balancer: - Ensure upstream timeouts exceed your client timeout (e.g., Nginx `proxy_read_timeout`/`proxy_send_timeout`, ALB idle timeout). - If using Cloudflare or similar, be aware of default 100s limits for requests. - Network/proxy/DNS: - Intermittent corporate proxies or DNS issues can cause slow reads. Check `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` env vars and ensure the API host is reachable from the runtime environment. - Server/API health: - Check Prefect Server (or Prefect Cloud workspace) logs/metrics for slow queries or high CPU/memory. - If this started recently, confirm there’s no maintenance or incident and that you’re on a recent Prefect version. - Scope of the issue: - Does it happen on specific endpoints (e.g., creating flow runs, setting state, listing runs) or all calls? - Is it consistent or intermittent? A couple of questions to help us narrow it down - Are you on Prefect Cloud or self-hosted? If self-hosted, are you behind Nginx/ALB/Cloudflare? - Which operation triggers the timeout (e.g., creating a deployment, running a flow, listing runs)? - Is this consistent or sporadic? Any recent infrastructure/network changes? - Can you share the full stack trace (including the “above exception”)? References - Settings reference (env vars like 
PREFECT_API_REQUEST_TIMEOUT
, 
PREFECT_CLIENT_MAX_RETRIES
): Settings reference - Client settings model (retry-related env vars): prefect.settings.models.client - API settings model (request timeout default and env var): prefect.settings.models.api
Open in Slack
PreviousNext
Powered by