Robert Szomolanyi
06/03/2026, 10:32 AM2026-06-03 10:11:13.656 app INFO [] [] [] [] : Content-Type: [application/json] Content-Encoding: [gzip] Date: [Wed, 03 Jun 2026 10:11:13 GMT] Server: [uvicorn] Content-Length: [809] Vary: [Origin Accept-Encoding] X-Krakend-Completed: [false]
[GIN] 2026/06/03 - 10:11:13 | 200 | 13.283935ms | 10.212.143.243 | GET "/prefect/work_pools/default_pool"
2026-06-03 10:11:13.672 app INFO [] [] [] [] : X-Forwarded-Port: [443] Forwarded: [for=10.212.143.243;host=<http://prefect-api.apps.os21v02.dctest.slsp.sk;proto=https|prefect-api.apps.os21v02.dctest.slsp.sk;proto=https>] X-Forwarded-For: [10.212.143.243] Accept: [*/*] X-Prefect-Api-Version: [0.8.4] Content-Length: [93] Content-Type: [application/json] X-Prefect-Worker-Name: [ProcessWorker 7b98008c-3cd3-44f2-b05e-1dcb0ead1316] Accept-Encoding: [gzip, deflate] User-Agent: [prefect/3.7.1 (API 0.8.4)] X-Forwarded-Host: [<http://prefect-api.apps.os21v02.dctest.slsp.sk|prefect-api.apps.os21v02.dctest.slsp.sk>] X-Forwarded-Proto: [https]
2026-06-03 10:11:23.673 app ERROR [babe-cafefacedbed] [] [] [] : [ENDPOINT: /prefect/work_pools/:work_pool_name/workers/heartbeat] context deadline exceeded
2026-06-03 10:11:23.673 access [] [] [] [] : 10.212.143.243 - - [03/Jun/2026 10:11:23 +0000] "POST /prefect/work_pools/default_pool/workers/heartbeat HTTP/1.1" 500 -1 10000 ms
2026-06-03 10:11:23.673 app ERROR [] [] [] [] : Error #01: context deadline exceeded\n
2026-06-03 10:11:23.673 app INFO [] [] [] [] : Vary: [Origin] X-Krakend-Completed: [false]
[GIN] 2026/06/03 - 10:11:23 | 500 | 10.000840736s | 10.212.143.243 | POST "/prefect/work_pools/default_pool/workers/heartbeat"
Error #01: context deadline exceededMarvin
06/03/2026, 10:32 AMMarvin
06/03/2026, 10:34 AMtext
POST /prefect/work_pools/default_pool/workers/heartbeat ... 500 ... 10000 ms
Error #01: context deadline exceeded
What it means: the worker started successfully enough to find the work pool (GET ... 200), but when it tried to send its heartbeat, the server did not finish processing that request within its timeout window and returned 500.
In Prefect 3.x, that heartbeat endpoint updates the worker record in the backend database. So this is usually not a worker startup problem — it is usually a server/database timeout problem.
Most likely reasons
1. Database is slow or blocked
- heartbeat writes to the Prefect DB
- if the DB is under load, locked, or slow, the request can time out
2. Server DB timeout is too low
- the exact 10000 ms strongly suggests a 10-second backend timeout is being hit
3. Too few DB connections / connection pool exhaustion
- common if several workers or API requests hit the DB at once
4. Proxy / gateway timeout behavior
- you are behind OpenShift / KrakenD / ingress
- even if the proxy is not the root cause, it can expose or worsen backend latency
So the likely flow is:
text
worker -> proxy/route -> Prefect API -> Prefect database
The GET succeeds quickly, but the POST heartbeat requires a DB write and that write is exceeding the timeout.
Why I think this is the reason
- GET "/prefect/work_pools/default_pool" returns 200
- POST "/prefect/work_pools/default_pool/workers/heartbeat" fails after exactly 10000 ms
- context deadline exceeded is a classic timeout symptom, typically from backend request/DB execution timing out
What to check first
1. Check Prefect server database health
- Is Postgres reachable and healthy?
- Any slow queries / locks?
- High CPU / memory on DB pod?
2. Increase the Prefect server DB timeout
On self-hosted Prefect server, try increasing:
bash
PREFECT_SERVER_DATABASE_TIMEOUT=30
Then restart the Prefect server.
3. Increase DB pool settings if needed
If you have multiple workers:
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=10
4. Check OpenShift / ingress / KrakenD timeouts
Make sure upstream/proxy timeout is greater than the Prefect processing time.
5. Look at Prefect API server logs and DB logs at the same timestamp
Specifically around:
text
2026-06-03 10:11:23
Important note
This is probably not caused by the work pool itself, because the pool lookup succeeded. The failure happens when the worker heartbeat tries to write/update worker state.
Most probable root cause in one sentence:
Your worker can reach the Prefect API, but the API cannot complete the heartbeat database operation within 10 seconds, so it returns context deadline exceeded.
If you want, I can help you with the exact OpenShift env vars / Helm values to increase the timeout for your Prefect server deployment.