It seems there might be an issue with failed tasks not releasing their concurrency limits in your Prefect setup. In Prefect, concurrency limits can be managed using task tags or global concurrency limits. When a task fails, its concurrency slot should typically be released.
Here are a few points to consider:
1.
Tag-based Concurrency: If you're using tag-based concurrency limits, ensure that the tags are correctly set up and monitored. Concurrency limits are checked when a task attempts to enter a Running state. More details can be found
here.
2.
Global Concurrency: If you're using global concurrency limits, ensure that your setup is correctly managing slot allocation and release. Prefect's concurrency limits should handle slot release even on task failure. More information is available
here.
If the issue persists, you might want to check if there are any known bugs or updates related to this behavior.