<@ULVA73B9P> I am running on prem server in Opensh...
# ask-marvin
r
@Marvin I am running on prem server in Openshift cluster. I got this error after worker start. What is reason ?
Copy code
2026-06-03 10:11:13.656 app INFO [] [] [] [] : Content-Type: [application/json] Content-Encoding: [gzip] Date: [Wed, 03 Jun 2026 10:11:13 GMT] Server: [uvicorn] Content-Length: [809] Vary: [Origin Accept-Encoding] X-Krakend-Completed: [false]
[GIN] 2026/06/03 - 10:11:13 | 200 | 13.283935ms | 10.212.143.243 | GET "/prefect/work_pools/default_pool"
2026-06-03 10:11:13.672 app INFO [] [] [] [] : X-Forwarded-Port: [443] Forwarded: [for=10.212.143.243;host=<http://prefect-api.apps.os21v02.dctest.slsp.sk;proto=https|prefect-api.apps.os21v02.dctest.slsp.sk;proto=https>] X-Forwarded-For: [10.212.143.243] Accept: [*/*] X-Prefect-Api-Version: [0.8.4] Content-Length: [93] Content-Type: [application/json] X-Prefect-Worker-Name: [ProcessWorker 7b98008c-3cd3-44f2-b05e-1dcb0ead1316] Accept-Encoding: [gzip, deflate] User-Agent: [prefect/3.7.1 (API 0.8.4)] X-Forwarded-Host: [<http://prefect-api.apps.os21v02.dctest.slsp.sk|prefect-api.apps.os21v02.dctest.slsp.sk>] X-Forwarded-Proto: [https]
2026-06-03 10:11:23.673 app ERROR [babe-cafefacedbed] [] [] [] : [ENDPOINT: /prefect/work_pools/:work_pool_name/workers/heartbeat] context deadline exceeded
2026-06-03 10:11:23.673 access [] [] [] [] : 10.212.143.243 - - [03/Jun/2026 10:11:23 +0000] "POST /prefect/work_pools/default_pool/workers/heartbeat HTTP/1.1" 500 -1 10000 ms
2026-06-03 10:11:23.673 app ERROR [] [] [] [] : Error #01: context deadline exceeded\n
2026-06-03 10:11:23.673 app INFO [] [] [] [] : Vary: [Origin] X-Krakend-Completed: [false]
[GIN] 2026/06/03 - 10:11:23 | 500 | 10.000840736s | 10.212.143.243 | POST "/prefect/work_pools/default_pool/workers/heartbeat"
Error #01: context deadline exceeded
m
thought for 123.0 seconds
The key clue is this part:
Copy code
text
POST /prefect/work_pools/default_pool/workers/heartbeat ... 500 ... 10000 ms
Error #01: context deadline exceeded
What it means: the worker started successfully enough to find the work pool (
GET ... 200
), but when it tried to send its heartbeat, the server did not finish processing that request within its timeout window and returned
500
. In Prefect 3.x, that heartbeat endpoint updates the worker record in the backend database. So this is usually not a worker startup problem — it is usually a server/database timeout problem. Most likely reasons 1. Database is slow or blocked - heartbeat writes to the Prefect DB - if the DB is under load, locked, or slow, the request can time out 2. Server DB timeout is too low - the exact
10000 ms
strongly suggests a 10-second backend timeout is being hit 3. Too few DB connections / connection pool exhaustion - common if several workers or API requests hit the DB at once 4. Proxy / gateway timeout behavior - you are behind OpenShift / KrakenD / ingress - even if the proxy is not the root cause, it can expose or worsen backend latency So the likely flow is:
Copy code
text
worker -> proxy/route -> Prefect API -> Prefect database
The GET succeeds quickly, but the POST heartbeat requires a DB write and that write is exceeding the timeout. Why I think this is the reason -
GET "/prefect/work_pools/default_pool"
returns
200
-
POST "/prefect/work_pools/default_pool/workers/heartbeat"
fails after exactly
10000 ms
-
context deadline exceeded
is a classic timeout symptom, typically from backend request/DB execution timing out What to check first 1. Check Prefect server database health - Is Postgres reachable and healthy? - Any slow queries / locks? - High CPU / memory on DB pod? 2. Increase the Prefect server DB timeout On self-hosted Prefect server, try increasing:
Copy code
bash
PREFECT_SERVER_DATABASE_TIMEOUT=30
Then restart the Prefect server. 3. Increase DB pool settings if needed If you have multiple workers:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=10
4. Check OpenShift / ingress / KrakenD timeouts Make sure upstream/proxy timeout is greater than the Prefect processing time. 5. Look at Prefect API server logs and DB logs at the same timestamp Specifically around:
Copy code
text
2026-06-03 10:11:23
Important note This is probably not caused by the work pool itself, because the pool lookup succeeded. The failure happens when the worker heartbeat tries to write/update worker state. Most probable root cause in one sentence: Your worker can reach the Prefect API, but the API cannot complete the heartbeat database operation within 10 seconds, so it returns
context deadline exceeded
. If you want, I can help you with the exact OpenShift env vars / Helm values to increase the timeout for your Prefect server deployment.