https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I got error from process worker, whic...
# marvin-ai
r
@Marvin I got error from process worker, which is connecting to local Prefect server : 141037.271 | ERROR | Flow run 'invaluable-honeybee' - Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation. 141037.279 | ERROR | Flow run 'invaluable-honeybee' - Crash detected! Execution was cancelled by the runtime environment. 141037.313 | INFO | Flow run 'invaluable-honeybee' - Finished in state Crashed('Execution was cancelled by the runtime environment.') 141037.313 | ERROR | Flow run 'invaluable-honeybee' - Engine execution exited with unexpected exception Traceback (most recent call last): File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 1582, in run_flow ret_val = run_flow_sync(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 1425, in run_flow_sync engine.call_flow_fn() File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 818, in call_flow_fn self.handle_success(result) File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 386, in handle_success self.set_state(terminal_state) File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/flow_engine.py", line 337, in set_state state = propose_state_sync( ^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/engine.py", line 475, in propose_state_sync response = set_state_and_handle_waits(set_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/utilities/engine.py", line 461, in set_state_and_handle_waits response = set_state_func() ^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 397, in set_flow_run_state response = self.request( ^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/orchestration/base.py", line 33, in request return self._client.send(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/base.py", line 591, in send response = self._send_with_retry( ^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/prefect/client/base.py", line 511, in _send_with_retry response = send(request, *send_args, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpx/_client.py", line 914, in send response = self._send_handling_auth( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth response = self._send_handling_redirects( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects response = self._send_single_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request response = transport.handle_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request resp = self._pool.handle_request(req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request raise exc from None File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request response = connection.handle_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request raise exc File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request ) = self._receive_response_headers(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers event = self._receive_event(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event data = self._network_stream.read( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/appdata/clag/env_prefect_worker/lib64/python3.12/site-packages/httpcore/_backends/sync.py", line 128, in read return self._sock.recv(max_bytes) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.12/ssl.py", line 1232, in recv return self.read(buflen) ^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.12/ssl.py", line 1105, in read return self._sslobj.read(len) ^^^^^^^^^^^^^^^^^^^^^^ What caused this error ?
m
✅ thought for 282.1 seconds
Short answer: your worker lost its deployment/global concurrency “lease” and Prefect intentionally cancelled the run to avoid exceeding your concurrency limit. Immediately after cancelling, the engine tried to report the “crashed” state to the API but the connection to your local Prefect server was broken, which is why you see the HTTPX SSL read error. What happened under the hood - Prefect enforces concurrency with timed leases. While a run is executing, the worker periodically renews its lease with the API. If renewal fails, Prefect assumes the slot is no longer reserved and cancels the run to prevent over-allocation. - In your logs: - “Concurrency lease renewal failed - slots are no longer reserved. Terminating execution…” → the lease renewal to the server failed. - “Crash detected! Execution was cancelled by the runtime environment.” → the engine intentionally cancels and marks the run as Crashed. - The subsequent HTTPX SSL read error occurs while the worker is attempting to set the final state on the server, but the client/server TLS connection has been interrupted. Common root causes - Local Prefect server was restarted or briefly unavailable while the run was executing (leases can be lost if the API becomes unavailable during renewal). - Network connectivity issue between the process worker and the server (container-to-host “localhost” mismatch, DNS, firewall, proxy). - HTTPS/TLS issues when pointing the worker at a local server behind a reverse proxy or using self‑signed certs (client does not trust the CA; connection resets). - Misconfigured PREFECT_API_URL (e.g., using http://127.0.0.1 from inside a container where 127.0.0.1 points to the container itself, not the host). How to fix / verify 1) Verify the server is stable and reachable from the worker - From the same environment the worker runs in (same container/host), run:
Copy code
curl -v $PREFECT_API_URL/health
Expect HTTP 200. If you’re using the default local server on the same machine, the typical value is:
Copy code
PREFECT_API_URL=<http://127.0.0.1:4200/api>
If the worker runs in Docker and the server runs on the host, 127.0.0.1 will not work; use the host’s IP/hostname or Docker networking (e.g., host.docker.internal on macOS/Windows). 2) Check for recent server restarts or errors - Inspect server logs around the timestamp of the failure for restarts or API errors. Avoid restarting the server while runs are active. 3) If using HTTPS to a local server or reverse proxy - Ensure the worker trusts the certificate chain (self-signed CA bundle installed). See SSL and proxy configuration guidance: - Configure network/SSL for Prefect 4) Turn on more logging and health checks - Set 
PREFECT_LOGGING_LEVEL=DEBUG
on the worker to get detailed lease renewal and client logs. - Use worker healthchecks and connectivity checks: - Worker healthchecks 5) Sanity-check concurrency config - If you recently enabled deployment concurrency, temporarily remove the concurrency limit to confirm whether the failure is isolated to lease renewal vs. general API connectivity. If API connectivity is the issue, you’ll still see errors when setting run state. Relevant docs - Concurrency and leases: Global concurrency limits - Deployment-level concurrency: Deployments – Concurrency limiting - Running local server and client URL: Run a local Prefect server - Network/SSL troubleshooting: Configure network access - Worker overview: Workers If you can share a bit more, I can help pinpoint the exact cause:
- Is the worker running on the same host as the server, or in a container/VM? - What is the exact 
PREFECT_API_URL
value on the worker? - Did the server restart or change certificates around the time of the failure? - Are you using HTTP or HTTPS in front of the local server?
r
@Marvin Worker is not running on same host as server. Concurrency is set to 1 on deployment. Deployment is scheduled to run every 5 minutes and failes with this error 2-3 times per hour. PREFECT_API_URL is set to local OCP cluster where server is running and https is used. Server was not restarted. Network failure is possible, but it is suspicious, that it happens exactly when flow is starting.
m
✅ thought for 120.8 seconds
Thanks — that extra context helps a lot. Given: - Worker runs off-cluster (not on the same host as the server) - Deployment concurrency = 1 - Schedule every 5 minutes - HTTPS to an OpenShift (OCP) route - Server is not restarting - Failures happen right as the flow starts Most likely root cause - This pattern strongly points to the network path (typically the OpenShift router/ingress or an upstream LB) closing or resetting idle TLS connections. When the worker starts a flow, the engine immediately makes API calls (set state, acquire/renew the deployment/global concurrency lease). If the first call reuses a keep-alive connection that the router has already dropped (common with 300s/5m idle timeouts), the request fails with an SSL read/connection reset, the lease renewal fails, and Prefect cancels the run “to prevent over-allocation.” - That’s why it appears “exactly when the flow is starting” and why you see the HTTPX SSL read error right after the cancellation. Why this aligns with your symptoms - Your cadence is every 5 minutes — that neatly matches common router/LB idle timeouts (5m) used by HAProxy/Nginx/OpenShift Routes. Intermittent failures “2–3 times per hour” are typical if some runs end up picking a stale connection from the pool while others establish a fresh one in time. - Server never restarts, but the TLS session/connection may be silently dropped by the router between runs. What to check/fix (in order) 1) Verify from the worker host - From the worker’s runtime environment, continuously probe for a bit longer than 5 minutes to see if you ever get a reset just as activity resumes:
Copy code
while true; do date; curl -sk -o /dev/null -w "%{http_code}\n" $PREFECT_API_URL/health; sleep 5; done
If you observe occasional non-200 or failures right after ~5 minutes of inactivity, that confirms an idle-timeout issue. 2) Increase/align timeouts on the OpenShift route/ingress - Bump the route’s idle/keep-alive timeouts above your schedule interval (e.g., 10–15 minutes) so the worker’s first call at flow start doesn’t hit a dead connection. - On OpenShift, this is typically done via route annotations for HAProxy timeouts. Your platform/SRE team can set the appropriate annotations/ingress config to ensure upstream connections aren’t dropped at 5 min. 3) Ensure the worker fully trusts the TLS chain - If you’re using a custom or internal CA on the OCP route, install the CA bundle on the worker and set:
Copy code
SSL_CERT_FILE=/path/to/ca-bundle.crt
Docs: Configure network access (proxies/SSL) 4) Sanity checks on URL and networking - Make sure 
PREFECT_API_URL
is the externally reachable route (not a cluster-internal DNS name that intermittently fails from the worker’s network). - Avoid IPs behind short-lived NAT mappings. Prefer stable DNS and ensure DNS TTLs aren’t too aggressive for your environment. 5) Turn on debug logs to see the exact lease error - On the worker:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
This will log the exception that caused “Concurrency lease renewal failed …” and can confirm you’re seeing a connection reset/SSL issue on the first lease call. 6) Quick diagnostic toggles (optional) - Temporarily remove the deployment concurrency limit for a few runs to see if failures disappear. If they do, the first failing call is very likely the lease renewal. If not, you’ll probably still see the SSL error on the subsequent state-setting call — still indicating the same network/TLS issue. 7) Longer-term mitigations - Run the worker in the same cluster/namespace as the server where possible to remove external routing. - If allowed, test over plain HTTP temporarily to isolate TLS termination issues (only for troubleshooting, not for production). Relevant docs - Concurrency and leases: Global concurrency limits
- Deployment concurrency: Deployments – Concurrency limiting - Local/self-hosted server and API URL: Run a local Prefect server - Network/SSL configuration: Configure network access - Worker health/connectivity: Worker healthchecks If you can share: - The OpenShift Route/Ingress timeout settings (or annotations) in front of the Prefect API - The exact exception message right after “Concurrency lease renewal failed …” with DEBUG logging I can confirm the hypothesis and suggest the exact timeout values/annotations to set.
r
@Marvin I investigated that whole flow finishes successfully and when result should be send to Prefect server this error is logged : Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation. On server side I see this:
Copy code
"POST /api/v2/concurrency_limits/leases/7fafac45-0830-462b-a6b2-7fe7d8f43074/renew HTTP/1.1" 410 Gone
Flow is very short: < 1s. There is no network drop or some other problems. I am using PostregSQL database and I am not using Redis for backend services. I am no version 3.6.3. Could it be some known bug ?
m
✅ thought for 211.5 seconds
Thank you — this points to deployment concurrency lease renewal failing at the moment your run transitions to Running/Completed, not to a network issue. What the 410 Gone means - The server log
Copy code
POST /api/v2/concurrency_limits/leases/<lease_id>/renew ... 410 Gone
is the server telling the worker “that lease no longer exists (it expired or was revoked).” - When the worker then tries to renew a non-existent lease, the engine cancels the run to avoid over-allocating slots, hence “Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.” Why you’re seeing it at flow start/finish with a 5-minute schedule - In Prefect 3.x, deployment-level concurrency uses an initial “deployment lease” created by the server before the run actually starts. That lease has a default TTL of 300 seconds (5 minutes). - With your schedule every 5 minutes and concurrency=1, there are times when the run starts right as the initial lease lifespan is expiring. If the lease expires (and is reaped) just before the worker’s first renewal, the server returns 410 Gone. Because your flow is very short (<1s), this can look like it “finishes” and then fails when reporting its result — in reality, the background lease task hit the 410 right at start/end. - This is not affected by PostgreSQL; leases are not stored in Postgres. If you’re not using Redis, you are on the in-memory or filesystem lease storage depending on your server config, but the core issue here is lease TTL vs. your 5-minute cadence. Known fixes and improvements after 3.6.3 - There have been robustness fixes to concurrency leasing since 3.6.3 (including improved renewal behavior and retry logic) that reduce flakiness around lease renewal timing. - Recommendation: upgrade both server and workers to the latest 3.6.x (or newer) to pick up these improvements. Latest notes: Release notes How to resolve in your setup - Increase the initial deployment lease duration above your maximum expected queueing/polling/scheduling delay so the lease does not expire just as runs begin: - Global setting (server):
Copy code
PREFECT_SERVER__CONCURRENCY__INITIAL_DEPLOYMENT_LEASE_DURATION=600  # e.g., 10 minutes
- Or set a per-deployment grace period via deployment concurrency options (if you configure deployment YAML), e.g., grace_period 600s. Docs: Deployments – Concurrency limiting - Keep server and worker clocks in sync (NTP). Significant clock skew can make leases appear older to the server than they are. - Optionally, use Redis-based lease storage for strongest guarantees in production (atomic ops, no FS races):
Copy code
pip install prefect-redis
  PREFECT_SERVER__CONCURRENCY__LEASE_STORAGE=prefect_redis.lease_storage
- Enable debug logs on the worker to confirm the exact renewal failure:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
You should see the 410 Gone tied to the first lease renewal attempt at run start. Relevant docs - Concurrency and leases: Global concurrency limits - Deployment concurrency: Deployments – Concurrency limiting - Settings reference (server, concurrency): Settings reference If you can share your current server setting for 
PREFECT_SERVER__CONCURRENCY__INITIAL_DEPLOYMENT_LEASE_DURATION
(or the deployment’s 
concurrency_options.grace_period
), I can suggest exact values. Upgrading from 3.6.3 and increasing the initial lease duration (e.g., to 600–900s) typically eliminates these 410s for 5‑minute schedules.
Open in Slack
PreviousNext
Powered by