https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm getting this error frequently whi...
# ask-marvin
p
@Marvin I'm getting this error frequently which cause my worker to crash (using prefect 3.3.5) I already increased some environment variable like PREFECT_API_REQUEST_TIMEOUT but it still crash
Copy code
Jul  7 08:49:17 dtfcin01 bash[49787]: 08:49:17.671 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
Jul  7 08:49:17 dtfcin01 bash[49787]: 08:49:17.671 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
Jul  7 08:49:19 dtfcin01 bash[49787]: 08:49:19.555 | INFO    | uvicorn.access - 10.10.20.80:61578 - "GET /api/work_pools/process-pool HTTP/1.1" 200
Jul  7 08:49:19 dtfcin01 bash[49787]: 08:49:19.566 | INFO    | uvicorn.access - 10.10.20.80:61578 - "POST /api/work_pools/process-pool/workers/heartbeat HTTP/1.1" 204
Jul  7 08:49:20 dtfcin01 bash[49787]: 08:49:20.756 | INFO    | prefect.server.services.recentdeploymentsscheduler - Scheduled 0 runs.
Jul  7 08:49:22 dtfcin01 bash[49787]: 08:49:22.671 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
Jul  7 08:49:22 dtfcin01 bash[49787]: 08:49:22.672 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
Jul  7 08:49:23 dtfcin01 bash[49787]: 08:49:23.826 | INFO    | uvicorn.access - 10.10.20.80:61578 - "POST /api/work_pools/process-pool/get_scheduled_flow_runs HTTP/1.1" 200
Jul  7 08:49:24 dtfcin01 bash[62202]: 08:49:24.527 | ERROR   | prefect.flow_runs.worker - Failed to submit flow run 'afefe645-2f6d-4e9f-89c4-265e5cfeccb8' to infrastructure.
Jul  7 08:49:24 dtfcin01 bash[62202]: Traceback (most recent call last):
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/runner/runner.py", line 589, in execute_flow_run
Jul  7 08:49:24 dtfcin01 bash[62202]:    flow_run = await self._client.read_flow_run(flow_run_id)
Jul  7 08:49:24 dtfcin01 bash[62202]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 668, in read_flow_run
Jul  7 08:49:24 dtfcin01 bash[62202]:    response = await self.request(
Jul  7 08:49:24 dtfcin01 bash[62202]:               ^^^^^^^^^^^^^^^^^^^
..
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
Jul  7 08:49:24 dtfcin01 bash[62202]:    response = await self._send_single_request(request)
Jul  7 08:49:24 dtfcin01 bash[62202]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
Jul  7 08:49:24 dtfcin01 bash[62202]:    response = await transport.handle_async_request(request)
Jul  7 08:49:24 dtfcin01 bash[62202]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
Jul  7 08:49:24 dtfcin01 bash[62202]:    with map_httpcore_exceptions():
Jul  7 08:49:24 dtfcin01 bash[62202]:         ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/contextlib.py", line 158, in __exit__
Jul  7 08:49:24 dtfcin01 bash[62202]:    self.gen.throw(value)
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
Jul  7 08:49:24 dtfcin01 bash[62202]:    raise mapped_exc(message) from exc
Jul  7 08:49:24 dtfcin01 bash[62202]: httpx.PoolTimeout
...
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/contextlib.py", line 158, in __exit__
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     self.gen.throw(value)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     raise to_exc(exc) from exc
Jul  7 08:49:24 dtfcin01 bash[62202]:    | httpcore.PoolTimeout
Jul  7 08:49:24 dtfcin01 bash[62202]:    | The above exception was the direct cause of the following exception:
Jul  7 08:49:24 dtfcin01 bash[62202]:    |
Jul  7 08:49:24 dtfcin01 bash[62202]:    | Traceback (most recent call last):
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/runner/runner.py", line 589, in execute_flow_run
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     flow_run = await self._client.read_flow_run(flow_run_id)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 668, in read_flow_run
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     response = await self.request(
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 53, in request
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     return await self._client.send(request)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/client/base.py", line 330, in send
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     response = await self._send_with_retry(
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/client/base.py", line 250, in _send_with_retry
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     response = await send(request, *send_args, **send_kwargs)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     response = await self._send_handling_auth(
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     response = await self._send_handling_redirects(
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     response = await self._send_single_request(request)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     response = await transport.handle_async_request(request)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     with map_httpcore_exceptions():
Jul  7 08:49:24 dtfcin01 bash[62202]:    |          ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/contextlib.py", line 158, in __exit__
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     self.gen.throw(value)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     raise mapped_exc(message) from exc
Jul  7 08:49:24 dtfcin01 bash[62202]:    | httpx.PoolTimeout
Jul  7 08:49:24 dtfcin01 bash[62202]:    +------------------------------------
Jul  7 08:49:24 dtfcin01 bash[49787]: 08:49:24.624 | INFO    | prefect.server.events.triggers - Automation 62695b6b-94b5-4b85-8ba8-e76986a547d4, trigger acdbabde-7c8e-4922-9c35-fe1ad36951ea covers event 'prefect.flow-run.Crashed' (94222ce3-1d39-4017-a72c-74ae9744e90b) for 'prefect.flow-run.afefe645-2f6d-4e9f-89c4-265e5cfeccb8' at '2025-07-07T08:49:24.581136+00:00'
Jul  7 08:49:24 dtfcin01 bash[49787]: 08:49:24.628 | INFO    | uvicorn.access - 10.10.20.80:61578 - "POST /api/flow_runs/afefe645-2f6d-4e9f-89c4-265e5cfeccb8/set_state HTTP/1.1" 201
Jul  7 08:49:24 dtfcin01 bash[62202]: 08:49:24.629 | INFO    | prefect.flow_runs.worker - Reported flow run 'afefe645-2f6d-4e9f-89c4-265e5cfeccb8' as crashed: Flow run could not be submitted to infrastructure:
Jul  7 08:49:24 dtfcin01 bash[62202]: ExceptionGroup('unhandled errors in a TaskGroup', [PoolTimeout('')])
m
Your message was too long, here's your message at the allowed limit: I'm getting this error frequently which cause my worker to crash (using prefect 3 3 5) I already increased some environment variable like PREFECT_API_REQUEST_TIMEOUT but it still crash ```Jul 7 08 49 17 dtfcin01 bash[49787] 08 49 17 671 | INFO | prefect server services marklateruns - Finished monitoring for late runs Jul 7 08 49 17 dtfcin01 bash[49787] 08 49 17 671 | INFO | prefect server services failexpiredpauses - Finished monitoring for late runs Jul 7 08 49 19 dtfcin01 bash[49787] 08 49 19 555 | INFO | uvicorn access - 10 10 20 80 61578 - GET /api/work_pools/process-pool HTTP/1 1 200 Jul 7 08 49 19 dtfcin01 bash[49787] 08 49 19 566 | INFO | uvicorn access - 10 10 20 80 61578 - POST /api/work_pools/process-pool/workers/heartbeat HTTP/1 1 204 Jul 7 08 49 20 dtfcin01 bash[49787] 08 49 20 756 | INFO | prefect server services recentdeploymentsscheduler - Scheduled 0 runs Jul 7 08 49 22 dtfcin01 bash[49787] 08 49 22 671 | INFO | prefect server services failexpiredpauses - Finished monitoring for late runs Jul 7 08 49 22 dtfcin01 bash[49787] 08 49 22 672 | INFO | prefect server services marklateruns - Finished monitoring for late runs Jul 7 08 49 23 dtfcin01 bash[49787] 08 49 23 826 | INFO | uvicorn access - 10 10 20 80 61578 - POST /api/work_pools/process-pool/get_scheduled_flow_runs HTTP/1 1 200 Jul 7 08 49 24 dtfcin01 bash[62202] 08 49 24 527 | ERROR | prefect flow_runs worker - Failed to submit flow run 'afefe645-2f6d-4e9f-89c4-265e5cfeccb8' to infrastructure Jul 7 08 49 24 dtfcin01 bash[62202] Traceback (most recent call last) Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/prefect/runner/runner py line 589 in execute_flow_run Jul 7 08 49 24 dtfcin01 bash[62202] flow_run = await self _client read_flow_run(flow_run_id) Jul 7 08 49 24 dtfcin01 bash[62202] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/prefect/client/orchestration/_flow_runs/client py line 668 in read_flow_run Jul 7 08 49 24 dtfcin01 bash[62202] response = await self request( Jul 7 08 49 24 dtfcin01 bash[62202] ^^^^^^^^^^^^^^^^^^^ Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/httpx/_client py line 1694 in _send_handling_redirects Jul 7 08 49 24 dtfcin01 bash[62202] response = await self _send_single_request(request) Jul 7 08 49 24 dtfcin01 bash[62202] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/httpx/_client py line 1730 in _send_single_request Jul 7 08 49 24 dtfcin01 bash[62202] response = await transport handle_async_request(request) Jul 7 08 49 24 dtfcin01 bash[62202] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/httpx/_transports/default py line 393 in handle_async_request Jul 7 08 49 24 dtfcin01 bash[62202] with map_httpcore_exceptions() Jul 7 08 49 24 dtfcin01 bash[62202] ^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/contextlib py line 158 in exit Jul 7 08 49 24 dtfcin01 bash[62202] self gen throw(value) Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/httpx/_transports/default py line 118 in map_httpcore_exceptions Jul 7 08 49 24 dtfcin01 bash[62202] raise mapped_exc(message) from exc Jul 7 08 49 24 dtfcin01 bash[62202] httpx
p
@Marvin I'm getting this error frequently which cause my worker to crash (using prefect 3.3.5) I already increased some environment variable like PREFECT_API_REQUEST_TIMEOUT but it still crash
Copy code
Jul  7 08:49:17 dtfcin01 bash[49787]: 08:49:17.671 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
Jul  7 08:49:17 dtfcin01 bash[49787]: 08:49:17.671 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
Jul  7 08:49:19 dtfcin01 bash[49787]: 08:49:19.555 | INFO    | uvicorn.access - 10.10.20.80:61578 - "GET /api/work_pools/process-pool HTTP/1.1" 200
Jul  7 08:49:19 dtfcin01 bash[49787]: 08:49:19.566 | INFO    | uvicorn.access - 10.10.20.80:61578 - "POST /api/work_pools/process-pool/workers/heartbeat HTTP/1.1" 204
Jul  7 08:49:20 dtfcin01 bash[49787]: 08:49:20.756 | INFO    | prefect.server.services.recentdeploymentsscheduler - Scheduled 0 runs.
Jul  7 08:49:22 dtfcin01 bash[49787]: 08:49:22.671 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
Jul  7 08:49:22 dtfcin01 bash[49787]: 08:49:22.672 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
Jul  7 08:49:23 dtfcin01 bash[49787]: 08:49:23.826 | INFO    | uvicorn.access - 10.10.20.80:61578 - "POST /api/work_pools/process-pool/get_scheduled_flow_runs HTTP/1.1" 200
Jul  7 08:49:24 dtfcin01 bash[62202]: 08:49:24.527 | ERROR   | prefect.flow_runs.worker - Failed to submit flow run 'afefe645-2f6d-4e9f-89c4-265e5cfeccb8' to infrastructure.
Jul  7 08:49:24 dtfcin01 bash[62202]: Traceback (most recent call last):
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/runner/runner.py", line 589, in execute_flow_run
Jul  7 08:49:24 dtfcin01 bash[62202]:    flow_run = await self._client.read_flow_run(flow_run_id)
Jul  7 08:49:24 dtfcin01 bash[62202]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:  File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 668, in ..
Jul  7 08:49:24 dtfcin01 bash[62202]: httpx.PoolTimeout
...
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/contextlib.py", line 158, in __exit__
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     self.gen.throw(value)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     raise to_exc(exc) from exc
Jul  7 08:49:24 dtfcin01 bash[62202]:    | httpcore.PoolTimeout
Jul  7 08:49:24 dtfcin01 bash[62202]:    | The above exception was the direct cause of the following exception:
...
Jul  7 08:49:24 dtfcin01 bash[62202]:    |          ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/contextlib.py", line 158, in __exit__
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     self.gen.throw(value)
Jul  7 08:49:24 dtfcin01 bash[62202]:    |   File "/data/miniconda3/envs/datafetch/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     raise mapped_exc(message) from exc
Jul  7 08:49:24 dtfcin01 bash[62202]:    | httpx.PoolTimeout
Jul  7 08:49:24 dtfcin01 bash[62202]:    +------------------------------------
Jul  7 08:49:24 dtfcin01 bash[49787]: 08:49:24.624 | INFO    | prefect.server.events.triggers - Automation 62695b6b-94b5-4b85-8ba8-e76986a547d4, trigger acdbabde-7c8e-4922-9c35-fe1ad36951ea covers event 'prefect.flow-run.Crashed' (94222ce3-1d39-4017-a72c-74ae9744e90b) for 'prefect.flow-run.afefe645-2f6d-4e9f-89c4-265e5cfeccb8' at '2025-07-07T08:49:24.581136+00:00'
Jul  7 08:49:24 dtfcin01 bash[49787]: 08:49:24.628 | INFO    | uvicorn.access - 10.10.20.80:61578 - "POST /api/flow_runs/afefe645-2f6d-4e9f-89c4-265e5cfeccb8/set_state HTTP/1.1" 201
Jul  7 08:49:24 dtfcin01 bash[62202]: 08:49:24.629 | INFO    | prefect.flow_runs.worker - Reported flow run 'afefe645-2f6d-4e9f-89c4-265e5cfeccb8' as crashed: Flow run could not be submitted to infrastructure:
Jul  7 08:49:24 dtfcin01 bash[62202]: ExceptionGroup('unhandled errors in a TaskGroup', [PoolTimeout('')])
m
Your message was too long, here's your message at the allowed limit: I'm getting this error frequently which cause my worker to crash (using prefect 3 3 5) I already increased some environment variable like PREFECT_API_REQUEST_TIMEOUT but it still crash ```Jul 7 08 49 17 dtfcin01 bash[49787] 08 49 17 671 | INFO | prefect server services marklateruns - Finished monitoring for late runs Jul 7 08 49 17 dtfcin01 bash[49787] 08 49 17 671 | INFO | prefect server services failexpiredpauses - Finished monitoring for late runs Jul 7 08 49 19 dtfcin01 bash[49787] 08 49 19 555 | INFO | uvicorn access - 10 10 20 80 61578 - GET /api/work_pools/process-pool HTTP/1 1 200 Jul 7 08 49 19 dtfcin01 bash[49787] 08 49 19 566 | INFO | uvicorn access - 10 10 20 80 61578 - POST /api/work_pools/process-pool/workers/heartbeat HTTP/1 1 204 Jul 7 08 49 20 dtfcin01 bash[49787] 08 49 20 756 | INFO | prefect server services recentdeploymentsscheduler - Scheduled 0 runs Jul 7 08 49 22 dtfcin01 bash[49787] 08 49 22 671 | INFO | prefect server services failexpiredpauses - Finished monitoring for late runs Jul 7 08 49 22 dtfcin01 bash[49787] 08 49 22 672 | INFO | prefect server services marklateruns - Finished monitoring for late runs Jul 7 08 49 23 dtfcin01 bash[49787] 08 49 23 826 | INFO | uvicorn access - 10 10 20 80 61578 - POST /api/work_pools/process-pool/get_scheduled_flow_runs HTTP/1 1 200 Jul 7 08 49 24 dtfcin01 bash[62202] 08 49 24 527 | ERROR | prefect flow_runs worker - Failed to submit flow run 'afefe645-2f6d-4e9f-89c4-265e5cfeccb8' to infrastructure Jul 7 08 49 24 dtfcin01 bash[62202] Traceback (most recent call last) Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/prefect/runner/runner py line 589 in execute_flow_run Jul 7 08 49 24 dtfcin01 bash[62202] flow_run = await self _client read_flow_run(flow_run_id) Jul 7 08 49 24 dtfcin01 bash[62202] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 7 08 49 24 dtfcin01 bash[62202] File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/prefect/client/orchestration/_flow_runs/client py line 668 in Jul 7 08 49 24 dtfcin01 bash[62202] httpx PoolTimeout Jul 7 08 49 24 dtfcin01 bash[62202] | File /data/miniconda3/envs/datafetch/lib/python3 12/contextlib py line 158 in exit Jul 7 08 49 24 dtfcin01 bash[62202] | self gen throw(value) Jul 7 08 49 24 dtfcin01 bash[62202] | File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/httpcore/_exceptions py line 14 in map_exceptions Jul 7 08 49 24 dtfcin01 bash[62202] | raise to_exc(exc) from exc Jul 7 08 49 24 dtfcin01 bash[62202] | httpcore PoolTimeout Jul 7 08 49 24 dtfcin01 bash[62202] | The above exception was the direct cause of the following exception Jul 7 08 49 24 dtfcin01 bash[62202] | ^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 7 08 49 24 dtfcin01 bash[62202] | File /data/miniconda3/envs/datafetch/lib/python3 12/contextlib py line 158 in exit Jul 7 08 49 24 dtfcin01 bash[62202] | self gen throw(value) Jul 7 08 49 24 dtfcin01 bash[62202] | File /data/miniconda3/envs/datafetch/lib/python3 12/site-packages/httpx/_transports/default py line 118 in map_httpcore_exceptions Jul 7 08 49 24 dtfcin01 bash[62202] | raise mapped_exc(message) from exc Jul 7 08 49 24 dtfcin01 bash[62202] | httpx PoolTimeout Jul 7 08 49 24 dtfcin01 bash[62202] +------------------------------------ Jul 7 08 49 24 dtfcin01 bash[49787]
p
@Marvin I'm getting this error frequently which cause my worker to crash (using prefect 3.3.5) I already increased some environment variable like PREFECT_API_REQUEST_TIMEOUT but it still crash
Copy code
...
Jul  7 08:49:24 dtfcin01 bash[62202]:    |     raise mapped_exc(message) from exc
Jul  7 08:49:24 dtfcin01 bash[62202]:    | httpx.PoolTimeout
Jul  7 08:49:24 dtfcin01 bash[62202]:    +------------------------------------
Jul  7 08:49:24 dtfcin01 bash[49787]: 08:49:24.624 | INFO    | prefect.server.events.triggers - Automation 62695b6b-94b5-4b85-8ba8-e76986a547d4, trigger acdbabde-7c8e-4922-9c35-fe1ad36951ea covers event 'prefect.flow-run.Crashed' (94222ce3-1d39-4017-a72c-74ae9744e90b) for 'prefect.flow-run.afefe645-2f6d-4e9f-89c4-265e5cfeccb8' at '2025-07-07T08:49:24.581136+00:00'
Jul  7 08:49:24 dtfcin01 bash[49787]: 08:49:24.628 | INFO    | uvicorn.access - 10.10.20.80:61578 - "POST /api/flow_runs/afefe645-2f6d-4e9f-89c4-265e5cfeccb8/set_state HTTP/1.1" 201
Jul  7 08:49:24 dtfcin01 bash[62202]: 08:49:24.629 | INFO    | prefect.flow_runs.worker - Reported flow run 'afefe645-2f6d-4e9f-89c4-265e5cfeccb8' as crashed: Flow run could not be submitted to infrastructure:
Jul  7 08:49:24 dtfcin01 bash[62202]: ExceptionGroup('unhandled errors in a TaskGroup', [PoolTimeout('')])
m
✅ thought for 20.6 seconds
Based on your error logs and the research, here's what's happening and how to address it: The 
PoolTimeout
error you're experiencing is typically related to connection pool exhaustion when your worker is trying to communicate with the Prefect server. This can happen when: 1. There are too many concurrent API requests 2. The server is taking too long to respond 3. Network latency issues between worker and server Here are the steps to resolve this: 1. Adjust Connection Timeouts You've already tried 
PREFECT_API_REQUEST_TIMEOUT
, but there are additional settings you can configure:
Copy code
bash
   # Add these to your worker environment
   PREFECT_API_REQUEST_TIMEOUT=30  # Increase if needed (in seconds)
   PREFECT_API_KEEPALIVE_TIMEOUT=5
   PREFECT_API_POOL_CONNECTIONS=100  # Adjust based on your workload
   PREFECT_API_MAX_RETRIES=3
2. Optimize Task Submission Rate If you're submitting a large number of tasks very quickly, consider: - Using task batching - Implementing rate limiting in your flow - Using 
task_runner
settings to control concurrency:
Copy code
python
   from prefect import flow
   from prefect.task_runners import ConcurrentTaskRunner

   @flow(task_runner=ConcurrentTaskRunner(max_concurrent_tasks=10))
   def your_flow():
       # Your flow code
3. Check Network Configuration - Ensure stable network connectivity between worker and server - Consider running the worker closer to the server to reduce latency - Check if any firewalls or proxies are interfering with connections 4. Server-side Settings If you're running your own Prefect server, adjust these settings:
Copy code
bash
   PREFECT_SERVER_API_HOST="0.0.0.0"
   PREFECT_API_DATABASE_CONNECTION_TIMEOUT=60
   PREFECT_API_DATABASE_POOL_SIZE=20
   PREFECT_API_DATABASE_MAX_OVERFLOW=10
5. Infrastructure Considerations - Ensure your server has adequate resources (CPU/memory) - Monitor server load during high-task periods - Consider scaling your server infrastructure if needed If you continue to experience issues after implementing these changes, you might want to: 1. Implement exponential backoff in your flow retries 2. Break very large flows into smaller sub-flows 3. Consider using work queues to distribute load across multiple workers Let me know if you need help implementing any of these solutions or if you'd like to explore other options.
Open in Slack
PreviousNext
Powered by