Hello everyone. Is anyone else having issues with lease renewals lately? I think at some point concurrency limits started to be strict, so the flow runs are failing with

Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation

I can see three issues related to this https://github.com/PrefectHQ/prefect/issues/19068 https://github.com/PrefectHQ/prefect/issues/18839 https://github.com/PrefectHQ/prefect/issues/19218 but none of them have a possible solution or answer. I might rollback to a previous client version at least Thank you

are you using tag-based concurrency or global concurrency limits directly?

The concurrency I'm using is setting this in the prefect.yaml file

concurrency_limit: { "limit": 30, "collision_strategy": "ENQUEUE" }

And then in the UI I can see the limits as Global Concurrency Limits with the deployment ID as name

gotcha, so its a concurrency limit set directly on a deployment, thanks and all your API instances and background services are configured to talk to redis as a message broker? (ie they all have those env vars set?)

Now that I'm seeing this, I'm not setting this variable

PREFECT_SERVER_EVENTS_CAUSAL_ORDERING

, I think it should be new, because I followed this doc to setup the high availability server in the first place

causal ordering should be unrelated from the info shared here, i think the behavior you're seeing is intentional since 3.4.15 - deployment concurrency strictly terminates flows when lease renewal fails to prevent over-allocation. things to check: - all API instances AND background services must use (the same) redis for lease storage - look for 410 GONE responses in logs - that's the server saying the lease expired or was revoked increasing lease duration helps buffer against transient issues but won't fix persistent renewal failures. if renewals keep failing, i'd guess something is talking to a different lease storage?

Yes, I'm seeing the 410s in both the server and the flows when they fail. I'll make sure that both API and background are using the same Redis (pretty sure they are) And I already increased the lease duration to an hour. I'll keep investigating and let you know if I find something. Thank you!