https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P>, i encounter this, why this happens, ...
# marvin-ai
p
@Marvin, i encounter this, why this happens, check health success
Copy code
11:25:45.024 | DEBUG   | prefect.profiles - Using profile 'public'
2026-02-26 11:25:45.369 | INFO     | __main__:<module>:66 - Starting deployment with version: 25.02.2026 and branch: None
11:25:45.371 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
11:25:45.470 | DEBUG   | prefect.events.clients - Reconnecting websocket connection.
11:25:45.470 | DEBUG   | prefect.events.clients - Opening websocket connection.
2026-02-26 11:25:45.476 | INFO     | __main__:main:23 - GitHub credentials loaded: GitHubCredentials(token=SecretStr('**********'))
2026-02-26 11:25:45.476 | INFO     | __main__:main:24 - Deploying /home/pradana/projects/outlet-scrape/workflows/workflows
2026-02-26 11:25:45.476 | INFO     | __main__:main:26 - Starting deployment...
2026-02-26 11:25:45.476 | INFO     | __main__:main:27 - Deploying from GitHub repository: <https://github.com/aidevakasha/outlet-scrape.git>
11:25:45.477 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp9krp_11k/outlet-scrape'...
11:25:45.477 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Cloning repository <https://github.com/aidevakasha/outlet-scrape.git>
11:25:45.554 | DEBUG   | prefect.events.clients - Pinging to ensure websocket connected.
11:25:45.635 | DEBUG   | prefect.events.clients - Pong received. Websocket connected.
11:25:45.635 | DEBUG   | prefect.events.clients - Resending 0 unconfirmed events.
11:25:45.636 | DEBUG   | prefect.events.clients - Finished resending unconfirmed events.
11:25:45.636 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
11:25:45.637 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Emitting event id=019c9832-1703-7ae3-b09c-ea49139a63b6.
11:25:45.637 | DEBUG   | prefect.events.clients - Added event id=019c9832-1703-7ae3-b09c-ea49139a63b6 to unconfirmed events list. There are now 1 unconfirmed events.
11:25:45.638 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Emit reconnection attempt 0.
11:25:45.638 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Sending event id=019c9832-1703-7ae3-b09c-ea49139a63b6.
11:25:45.638 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Checkpointing event id=019c9832-1703-7ae3-b09c-ea49139a63b6.
11:25:51.234 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2
11:25:52.565 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
11:25:52.889 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp72nxupud/outlet-scrape'...
11:25:52.890 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Cloning repository <https://github.com/>
11:25:56.697 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2
11:25:56.715 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
Creating/updating deployments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--11:25:56.837 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
Creating/updating deployments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--11:25:57.190 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/2.
118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError
 prefect version
11:27:27.428 | DEBUG   | prefect.profiles - Using profile 'public'
Version:              3.6.13
API version:          0.8.4
Python version:       3.13.11
Git commit:           9eb03a74
Built:                Fri, Jan 23, 2026 03:49 AM
OS/Arch:              linux/x86_64
Profile:              public
Server type:          server
Pydantic version:     2.12.5
Server:
  Database:           sqlite
  SQLite version:     3.50.4
Integrations:
  prefect-docker:     0.7.1
  prefect-github:     0.4.0
  prefect-email:      0.4.2
m
Your message was too long, here's your message at the allowed limit: i encounter this why this happens check health success ```11 25 45 024 | DEBUG | prefect profiles - Using profile 'public' 2026-02-26 11 25 45 369 | INFO | main <module> 66 - Starting deployment with version 25 02 2026 and branch None 11 25 45 371 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ 11 25 45 470 | DEBUG | prefect events clients - Reconnecting websocket connection 11 25 45 470 | DEBUG | prefect events clients - Opening websocket connection 2026-02-26 11 25 45 476 | INFO | main main 23 - GitHub credentials loaded GitHubCredentials(token=SecretStr('*****')) 2026-02-26 11 25 45 476 | INFO | main main 24 - Deploying /home/pradana/projects/outlet-scrape/workflows/workflows 2026-02-26 11 25 45 476 | INFO | main main 26 - Starting deployment 2026-02-26 11 25 45 476 | INFO | main main 27 - Deploying from GitHub repository <https //github com/aidevakasha/outlet-scrape git> 11 25 45 477 | DEBUG | prefect runner storage git-repository outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp9krp_11k/outlet-scrape' 11 25 45 477 | DEBUG | prefect runner storage git-repository outlet-scrape - Cloning repository <https //github com/aidevakasha/outlet-scrape git> 11 25 45 554 | DEBUG | prefect events clients - Pinging to ensure websocket connected 11 25 45 635 | DEBUG | prefect events clients - Pong received Websocket connected 11 25 45 635 | DEBUG | prefect events clients - Resending 0 unconfirmed events 11 25 45 636 | DEBUG | prefect events clients - Finished resending unconfirmed events 11 25 45 636 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ 11 25 45 637 | DEBUG | prefect events clients - EventsClient(id=138704068446592) Emitting event id=019c9832-1703-7ae3-b09c-ea49139a63b6 11 25 45 637 | DEBUG | prefect events clients - Added event id=019c9832-1703-7ae3-b09c-ea49139a63b6 to unconfirmed events list There are now 1 unconfirmed events 11 25 45 638 | DEBUG | prefect events clients - EventsClient(id=138704068446592) Emit reconnection attempt 0 11 25 45 638 | DEBUG | prefect events clients - EventsClient(id=138704068446592) Sending event id=019c9832-1703-7ae3-b09c-ea49139a63b6 11 25 45 638 | DEBUG | prefect events clients - EventsClient(id=138704068446592) Checkpointing event id=019c9832-1703-7ae3-b09c-ea49139a63b6 11 25 51 234 | DEBUG | prefect runner storage git-repository outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2 11 25 52 565 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ 11 25 52 889 | DEBUG | prefect runner storage git-repository outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp72nxupud/outlet-scrape' 11 25 52 890 | DEBUG | prefect runner storage git-repository outlet-scrape - Cloning repository <https //github com/> 11 25 56 697 | DEBUG | prefect runner storage git-repository outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2 11 25 56 715 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ Creating/updating deployments ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% - -- --11 25 56 837 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ Creating/updating deployments ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% - -- --11 25 57 190 | DEBUG |
p
@Marvin, i encounter this, why this happens, check health success
Copy code
prefect.client - Connecting to API at http://******:4200/api/
11:25:45.470 | DEBUG   | prefect.events.clients - Reconnecting websocket connection.
11:25:45.470 | DEBUG   | prefect.events.clients - Opening websocket connection.

2026-02-26 11:25:45.476 | INFO     | __main__:main:24 - Deploying /home/pradana/projects/outlet-scrape/workflows/workflows
2026-02-26 11:25:45.476 | INFO     | __main__:main:26 - Starting deployment...
2026-02-26 11:25:45.476 | INFO     | __main__:main:27 - Deploying from GitHub repository: <https://github.com/aidevakasha/outlet-scrape.git>
11:25:45.477 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp9krp_11k/outlet-scrape'...
11:25:45.477 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Cloning repository <https://github.com/aidevakasha/outlet-scrape.git>
received. Websocket connected.
11:25:45.635 | DEBUG   | prefect.events.clients - Resending 0 unconfirmed events.
prefect.client - Connecting to API at http://******:4200/api/
11:25:45.637 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Emitting event id=019c9832-1703-7ae3-b09c-ea49139a63b6.
11:25:45.638 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Checkpointing event id=019c9832-1703-7ae3-b09c-ea49139a63b6.
11:25:51.234 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2
11:25:52.565 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
11:25:52.889 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp72nxupud/outlet-scrape'...
11:25:52.890 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Cloning repository <https://github.com/>
11:25:56.697 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2
-11:25:56.837 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
Creating/updating deployments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--11:25:57.190 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/2.
118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError
 prefect version
11:27:27.428 | DEBUG   | prefect.profiles - Using profile 'public'
Version:              3.6.13
API version:          0.8.4
Python version:       3.13.11
Git commit:           9eb03a74
Built:                Fri, Jan 23, 2026 03:49 AM
OS/Arch:              linux/x86_64
Profile:              public
Server type:          server
Pydantic version:     2.12.5
Server:
  Database:           sqlite
  SQLite version:     3.50.4
Integrations:
  prefect-docker:     0.7.1
  prefect-github:     0.4.0
  prefect-email:      0.4.2
m
Your message was too long, here's your message at the allowed limit: i encounter this why this happens check health success ```11 25 45 024 | DEBUG | prefect profiles - Using profile 'public' 2026-02-26 11 25 45 369 | INFO | main <module> 66 - Starting deployment with version 25 02 2026 and branch None 11 25 45 371 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ 11 25 45 470 | DEBUG | prefect events clients - Reconnecting websocket connection 11 25 45 470 | DEBUG | prefect events clients - Opening websocket connection 2026-02-26 11 25 45 476 | INFO | main main 23 - GitHub credentials loaded GitHubCredentials(token=SecretStr('*****')) 2026-02-26 11 25 45 476 | INFO | main main 24 - Deploying /home/pradana/projects/outlet-scrape/workflows/workflows 2026-02-26 11 25 45 476 | INFO | main main 26 - Starting deployment 2026-02-26 11 25 45 476 | INFO | main main 27 - Deploying from GitHub repository <https //github com/aidevakasha/outlet-scrape git> 11 25 45 477 | DEBUG | prefect runner storage git-repository outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp9krp_11k/outlet-scrape' 11 25 45 477 | DEBUG | prefect runner storage git-repository outlet-scrape - Cloning repository <https //github com/aidevakasha/outlet-scrape git> 11 25 45 554 | DEBUG | prefect events clients - Pinging to ensure websocket connected 11 25 45 635 | DEBUG | prefect events clients - Pong received Websocket connected 11 25 45 635 | DEBUG | prefect events clients - Resending 0 unconfirmed events 11 25 45 636 | DEBUG | prefect events clients - Finished resending unconfirmed events 11 25 45 636 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ 11 25 45 637 | DEBUG | prefect events clients - EventsClient(id=138704068446592) Emitting event id=019c9832-1703-7ae3-b09c-ea49139a63b6 11 25 45 638 | DEBUG | prefect events clients - EventsClient(id=138704068446592) Checkpointing event id=019c9832-1703-7ae3-b09c-ea49139a63b6 11 25 51 234 | DEBUG | prefect runner storage git-repository outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2 11 25 52 565 | DEBUG | prefect client - Connecting to API at http //*** 4200/api/ 11 25 52 889 | DEBUG | prefect runner storage git-repository outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp72nxupud/outlet-scrape' 11 25 52 890 | DEBUG | prefect runner storage git-repository outlet-scrape - Cloning repository <https //github com/> 11 25 56 697 | DEBUG | prefect runner storage git-repository outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2 -11 25 56 837 | DEBUG | prefect client - Connecting to API at http //**** 4200/api/ Creating/updating deployments ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% - -- --11 25 57 190 | DEBUG | prefect client - Encountered retryable exception during request Another attempt will be made in 2s This is attempt 1/2 118 in map_httpcore_exceptions raise mapped_exc(message) from exc httpx ReadError prefect version 11 27 27 428 | DEBUG | prefect profiles - Using profile 'public' Version 3 6 13 API version 0 8 4 Python version 3 13 11 Git commit 9eb03a74 Built Fri Jan 23 2026 03 49 AM OS/Arch linux/x86_64 Profile public Server type server Pydantic version 2
✅ thought for 395.0 seconds
p
@Marvin, i encounter this, why this happens, check health success
Copy code
prefect.client - Connecting to API at http://******:4200/api/
11:25:45.470 | DEBUG   | prefect.events.clients - Reconnecting websocket connection.
11:25:45.470 | DEBUG   | prefect.events.clients - Opening websocket connection.

2026-02-26 11:25:45.476 | INFO     | __main__:main:24 - Deploying /home/pradana/projects/outlet-scrape/workflows/workflows
2026-02-26 11:25:45.476 | INFO     | __main__:main:26 - Starting deployment...
2026-02-26 11:25:45.476 | INFO     | __main__:main:27 - Deploying from GitHub repository: <https://github.com/aidevakasha/outlet-scrape.git>
11:25:45.477 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp9krp_11k/outlet-scrape'...
11:25:45.477 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Cloning repository <https://github.com/aidevakasha/outlet-scrape.git>
received. Websocket connected.
11:25:45.635 | DEBUG   | prefect.events.clients - Resending 0 unconfirmed events.
prefect.client - Connecting to API at http://******:4200/api/
11:25:45.637 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Emitting event id=019c9832-1703-7ae3-b09c-ea49139a63b6.
11:25:45.638 | DEBUG   | prefect.events.clients - EventsClient(id=138704068446592): Checkpointing event id=019c9832-1703-7ae3-b09c-ea49139a63b6.
11:25:51.234 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2
11:25:52.565 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
11:25:52.889 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Pulling contents from repository 'outlet-scrape' to '/tmp/tmp72nxupud/outlet-scrape'...
11:25:52.890 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Cloning repository <https://github.com/>
11:25:56.697 | DEBUG   | prefect.runner.storage.git-repository.outlet-scrape - Successfully checked out commit 877170f0148c1fc5124ff47c9e691832afb9aae2
-11:25:56.837 | DEBUG   | prefect.client - Connecting to API at http://******:4200/api/
Creating/updating deployments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--11:25:57.190 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/2.
118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError
 prefect version
11:27:27.428 | DEBUG   | prefect.profiles - Using profile 'public'
Version:              3.6.13
API version:          0.8.4
Python version:       3.13.11
Git commit:           9eb03a74
Built:                Fri, Jan 23, 2026 03:49 AM
OS/Arch:              linux/x86_64
Profile:              public
Server type:          server
Pydantic version:     2.12.5
Server:
  Database:           sqlite
  SQLite version:     3.50.4
Integrations:
  prefect-docker:     0.7.1
  prefect-github:     0.4.0
  prefect-email:      0.4.2
why above happens
m
✅ thought for 178.7 seconds
Thanks for the logs — the key line is:
Copy code
Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/2.
...
httpx.ReadError
What this means - 
httpx.ReadError
is raised when the server closes the connection or the connection drops while the client is reading the response. It’s a transient network/protocol issue — not necessarily that your API is “down,” so a health check can still pass. - Prefect’s client treats this as retryable and will retry a few times by default. Why it often happens - A reverse proxy/load balancer/ingress closes idle/long-running connections (common default idle timeouts are 15–30s). - The API handler takes longer than the client’s read timeout (defaults to 60s), so the socket is closed mid-read. - Occasional protocol quirks (HTTP/2, uvicorn) can cause midstream resets; Prefect retries these by design. Relevant details in Prefect 3.x - Request timeout is controlled by 
PREFECT_API_REQUEST_TIMEOUT
(applies to connect/read/write/pool; default 60s). - Client has built-in retries for 
httpx.ReadError
and similar transient errors. - Prefect sets HTTP keepalive pooling with an idle expiry of ~25s to avoid stale connections, but if your proxy/LB is lower than that, you can still see resets. What you can do 1) Bump the client request timeout for deployment operations (common fix if deployment creation is just taking a bit longer):
Copy code
export PREFECT_API_REQUEST_TIMEOUT=120   # or higher
export PREFECT_CLIENT_MAX_RETRIES=5      # ensure enough retries
2) If you’re behind a proxy/LB/ingress, make sure its timeouts exceed your client timeout and don’t kill idle connections too aggressively. For example (Nginx-style):
Copy code
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
keepalive_timeout 65s;
Also confirm any cloud LB idle timeout (ALB/ELB, etc.) is comfortably above your client’s behavior. 3) Keep HTTP/2 disabled if you suspect protocol issues (it’s off by default, but set this explicitly if you’ve enabled it):
Copy code
export PREFECT_API_ENABLE_HTTP2=false
4) If you control the client in code and suspect idle keepalive mismatches, you can tune httpx settings directly (advanced):
Copy code
from prefect.client.orchestration import get_client
import httpx

async with get_client(
    httpx_settings={
        "limits": httpx.Limits(
            max_connections=16,
            max_keepalive_connections=8,
            keepalive_expiry=20,   # lower than your LB idle timeout
        ),
        "timeout": httpx.Timeout(120.0),
    }
) as client:
    ...
5) Check server/proxy logs around 112557 for connection resets/timeouts. Since you’re running the Prefect Server with SQLite, heavy write operations can be slower; for production, Postgres is recommended to reduce latency spikes. Why your health check can still say “OK” - Health checks are quick, independent requests. A 
ReadError
occurs when the long-running/streamed request used during “Creating/updating deployments…” gets interrupted or times out mid-response — different path than a simple health probe. References - Known transient protocol issues that Prefect retries by design: #7512 and #7442 - Settings you can adjust: Prefect settings A couple of clarifying questions to help pinpoint the cause: - Are you going through a reverse proxy or Kubernetes ingress to reach http://****:4200, and if so, what are its idle/read timeouts? - Does this only happen during deployment creation, or do you see it on other API calls too? - Can you share any API server or proxy logs around the timestamp of the 
ReadError
?
p
Copy code
Creating/updating deployments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--11:25:57.190 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/2.
Traceback (most recent call last):
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        pool_request.request
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
    raise exc
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.READ_NUM_BYTES, timeout=timeout
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
         ~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/pradana/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/prefect/client/base.py", line 271, in _send_with_retry
    response = await send(request, *send_args, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/pradana/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/pradana/projects/outlet-scrape/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError
here the error @Marvin
m
✅ thought for 184.5 seconds
Thanks for the logs — the key line is this:
Copy code
Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/2.
...
httpx.ReadError
A ReadError means the connection to your Prefect API was opened, but the response stream was cut off while the client was reading it. In practice this is usually caused by one of: - A reverse proxy / load balancer closing the connection (idle/read timeout too low) - The API server process recycling/restarting mid-request - Long/locked DB operations delaying the response until an upstream timeout closes the socket (SQLite can do this under load) - Mismatched keep-alive settings between client, proxy, and server Your health check works because it’s fast; the “create/update deployments” call is longer and more prone to timeouts. Recommended fixes Client-side (the process running your deploy script/command): - Increase the API request timeout and (optionally) retries:
Copy code
# Increase read/connect/write/pool timeouts (default is 60s)
  prefect config set PREFECT_API_REQUEST_TIMEOUT=120

  # Optionally increase retries if you see intermittent network/proxy blips
  prefect config set PREFECT_CLIENT_MAX_RETRIES=10
In code (only if you’re creating a client yourself):
Copy code
import httpx
  from prefect import get_client

  httpx_settings = {"timeout": httpx.Timeout(connect=120, read=120, write=120, pool=120)}
  async with get_client(httpx_settings=httpx_settings) as client:
      await client.hello()
Server-side (Prefect Server at :4200): - Bump the API keep-alive timeout so proxies don’t drop connections:
Copy code
prefect server start --keep-alive-timeout 60
- You’re on SQLite. It’s great for dev, but under concurrent writes it can lock and stall long enough to trigger upstream timeouts. If this is anything beyond light testing or you’re running multiple workers/processes, move to Postgres. If you’re behind NGINX/Ingress/LB: - Ensure proxy timeouts are longer than the client/server timeouts and that WebSockets are allowed:
Copy code
location /api {
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_connect_timeout 10s;
      proxy_read_timeout 60s;
      proxy_send_timeout 60s;
      proxy_pass <http://prefect_api>;
  }
- If you changed keep-alives, align them: proxy idle timeout ≤ Prefect client timeout ≤ Prefect server keep-alive. Why you also see two git clones - Seeing two temporary checkout paths can be normal if the deploy step resolves source twice (e.g., preparing context then building deployments). If you consistently see a “Cloning repository https://github.com/” without the repo path, share your deploy code; that could hint at a misconfigured source URL somewhere. Helpful docs - Self-hosted server CLI options: Server CLI - Advanced self-hosted (scaling, proxies): Advanced self-hosted - Settings reference (timeouts/retries): Settings reference A few quick checks to narrow it down - Are you running behind NGINX/Ingress/ALB? If yes, what are the proxy_read_timeout / send_timeout? - Does the ReadError happen consistently, or intermittently? - Can you share how you’re triggering the deployment (CLI vs script using 
flow.from_source(...).deploy(...)
)? - Server logs around 112557 — any tracebacks or restarts? If you can share those details, I can tailor the exact settings to your setup.
Thanks for the stack trace — it confirms the client is waiting for response headers when the connection is closed mid‑read: - Root exception: 
httpcore.ReadError
→ wrapped as 
httpx.ReadError
- This almost always means the server (or something in between, e.g., proxy/LB) closed/reset the socket while the client was reading. Why this can happen even when health checks succeed - Health checks are tiny, fast requests. Creating/updating deployments is a larger/longer request cycle. Proxies/LBs or the server may close an “idle” or long‑running connection while the client is still reading. Most common causes - Proxy/LB/ingress idle/read timeout is too low and closes the connection mid‑response. - Server keep‑alive timeout is lower than the client’s pooled connection idle time, so reused connections can be stale. - API handler is slow (e.g., SQLite writes or heavy CPU) and exceeds read timeout, causing the peer to reset/close. - Less commonly, protocol quirks (HTTP/2/uvicorn) — Prefect retries these by design. What I’d do next (in order) 1) Increase the client’s request timeout (applies to connect/read/write/pool):
Copy code
export PREFECT_API_REQUEST_TIMEOUT=120
2) Ensure the client can retry enough times (your log shows “attempt 1/2”, which is unusually low):
Copy code
export PREFECT_CLIENT_MAX_RETRIES=5
export PREFECT_CLIENT_RETRY_JITTER_FACTOR=0.2
3) If you’re behind a reverse proxy/LB/ingress, raise its timeouts (examples): - Nginx:
Copy code
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
keepalive_timeout 65s;
- AWS ALB/ELB: set idle timeout above your expected request/keepalive durations (e.g., 60–120s). 4) Align server keep‑alive with your network path. Prefect Server (uvicorn) default keep‑alive is 5s, which can be too short. Start the server with a higher value:
Copy code
prefect server start --keep-alive-timeout 30
(or via env: 
PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=30
) 5) Keep HTTP/2 off (default is off). If you enabled it, turn it off:
Copy code
export PREFECT_API_ENABLE_HTTP2=false
6) Check the API server logs at the exact timestamp of the error (112557). If the server raised an exception or restarted, you’ll see it there — that would also produce a mid‑read close. 7) If needed, tune client keep‑alive for your environment (advanced; only if you suspect keep‑alive reuse issues). Example of lowering the idle expiry so stale connections aren’t reused:
Copy code
from prefect.client.orchestration import get_client
import httpx

async with get_client(
    httpx_settings={
        "limits": httpx.Limits(
            max_connections=16,
            max_keepalive_connections=8,
            keepalive_expiry=20,  # ensure this is lower than any LB idle timeout
        ),
        "timeout": httpx.Timeout(120.0),
    }
) as client:
    ...
Notes specific to your setup - You’re on Prefect Server with SQLite. Under heavier write load, SQLite can stall requests; consider Postgres for stability/perf in production. - The log snippet shows retries limited to 2 attempts; verify no profile/env is setting 
PREFECT_CLIENT_MAX_RETRIES=1
. References - Known transient protocol issues Prefect retries: #7512 and #7442 - Settings you can adjust: Prefect settings To help pinpoint root cause, could you share: - Whether the API is reached through a proxy/LB/ingress and its idle/read timeout values - Any server logs at the failure time - Whether this only happens during deployment creation or on other API calls too
3 Views
Open in Slack
PreviousNext
Powered by