https://prefect.io logo
Join Slack
Powered by
```File "/usr/local/lib/python3.12/site-packages/h...
# marvin-ai
r
Copy code
File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/usr/local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/prefect/chrombpnet-registry-proj/app/workflows/submit.py", line 387, in process_job_by_type
    result = await variant_scoring_submit_flow(scoring_input, temp_table_id, chromosomes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1391, in run_flow_async
    async with engine.start():
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1309, in start
    async with self.initialize_run():
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1234, in initialize_run
    self.flow_run = await self.create_flow_run(self.client)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1079, in create_flow_run
    return await client.create_flow_run(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 568, in create_flow_run
    response = await self.request("POST", "/flow_runs/", json=flow_run_create_json)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 53, in request
    return await self._client.send(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 330, in send
    response = await self._send_with_retry(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 250, in _send_with_retry
    response = await send(request, *send_args, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError
my prefect flow that used to work on prefect managed infra is no longer starting sub-flows. do you know why @Marvin
m
✅ thought for 318.7 seconds
Thanks for sharing the traceback — that helps. The key line is this:
Copy code
httpx.ReadError
...
in create_flow_run -> POST /flow_runs/
That means the SDK tried to register a subflow run with the Prefect API, but the HTTP connection was closed before a response was received. In practice, we most often see this when: - Many subflows are started in parallel and you hit API rate limits or overwhelm the connection pool - There’s a transient network hiccup (server closed idle connection, load balancer dropped the connection, etc.) - Less commonly, an auth or URL misconfiguration (but that usually shows as 401/403/404 instead of a ReadError) A few targeted steps to get you unblocked quickly: 1) Add backoff/retries for client requests - Increase client retries to smooth over transient ReadErrors:
Copy code
export PREFECT_CLIENT_MAX_RETRIES=8
export PREFECT_CLIENT_RETRY_JITTER_FACTOR=0.3
(These map to the Prefect client retry settings in 3.x.) 2) Throttle subflow creation - If you are starting many subflows at once, add a small rate limit to reduce bursts of POST /flow_runs calls:
Copy code
from prefect import flow
from prefect.concurrency.asyncio import rate_limit
from anyio import create_task_group

@flow
async def child_flow(x): 
    return x

@flow
async def parent_flow(params):
    async with rate_limit(ops=50, per=60), create_task_group() as tg:
        for p in params:
            tg.start_soon(child_flow, p)
Tune ops/per to your workload. 3) Turn on debug logs to confirm what’s happening
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
Look for messages about retries or rate limits around the time subflows start. 4) Upgrade the runtime image - If your managed-infra deployment uses an older image, upgrading to the latest Prefect 3.x image can pick up stability fixes in httpx/httpcore and the SDK. 5) Sanity check environment - Make sure the managed-infra environment still has a valid 
PREFECT_API_URL
and 
PREFECT_API_KEY
(these should be set automatically by Prefect Cloud, but worth confirming if anything changed recently). - Check Prefect Cloud Status if this began suddenly. Useful references: - Troubleshoot Prefect Cloud - Understanding rate limits in Prefect Cloud - Global concurrency/rate limiting To help pinpoint the cause, could you share: - Your Prefect SDK version (
prefect version
) and whether this started after an upgrade - Roughly how many subflows you’re starting concurrently - Whether the error is intermittent or happens consistently - A recent parent flow run ID and timestamp — I can look for signs of rate limiting or network churn around that time Once we know those details, we can suggest the minimal throttle/retry settings to stabilize your runs.
3 Views
Open in Slack
PreviousNext
Powered by