@Marvin why is my job failing with an error when it can't acquire a lock? I thought it would block until it could. The error is

Finished in state Failed("Flow run encountered an exception. ConcurrencySlotAcquisitionError: Unable to acquire concurrency slots on ['concurrency_key']")

It appears that you're encountering an issue with Prefect's concurrency limits, specifically with acquiring concurrency slots. The

ConcurrencySlotAcquisitionError

means that your flow run failed to acquire the necessary concurrency slots, which are required to enforce concurrency limits. The documentation doesn't explicitly cover the scenario when a job fails to acquire a lock, but it mentions the

slot_decay_per_second

parameter that allows configuration of slot availability. If this parameter is set to a higher value, slots become available quicker, whereas a lower value will slow down their availability. To address this issue, you may need to review your concurrency limit configurations, potentially adjusting the

slot_decay_per_second

parameter. Unfortunately, the provided documentation snippets do not offer a direct answer, but you can learn more about global concurrency limits in Prefect. If your jobs are not blocking as expected until a slot is acquired, this behavior might be due to specific configurations or limitations within Prefect that aren't covered in the information returned. I suggest checking your concurrency limit settings and the Prefect version you're using since you might need a different approach or some further configurations. For further assistance, it would be beneficial to consult Prefect's support or community forums as they may have more detailed information on handling situations where jobs fail to acquire concurrency slots.

I also just noticed this in the logs, a 500 error to the concurrency limits API:

prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/GUID-FOR-ACCOUNT/workspaces/24167a34-57cb-4afc-a47a-1012feda8773/v2/concurrency_limits/increment>'
Response: {'exception_message': 'Internal Server Error'}

hi @Omar Khudeira - would you be able to reach out with your account ID in a DM?

Done.