We have started to see a new bug where all our flows with concurrency limits are being cancelled. It gives error messages:

"Deployment concurrency limit reached."

It runs fine and then suddenly all future flows are cancelled. If I filter an all flows I can see nothing is running, i.e. which could block as it has taken the deployment concurrency slot. I am struggling to find out why this is happening. Our setup is quite simple:

concurrency_limit:
      limit: 3
      collision_strategy: "CANCEL_NEW"

Ideally it should cancel the running instance, but that does not seem to be an option in Prefect.

Hey @Tristan Moeller ! If you look at the global concurrency limits in your workspace you should see one that corresponds to your deployment. Do you see any allocated slots for that concurrency limit?

Hi @alex, Yes, there is an active slot assigned for the deployment, even though it is not running.

Hmmm, I wouldn’t expect this to cause new runs to be cancelled since there are two slots still available. Are new runs still being cancelled? The slot is probably taken because the release failed for one of the runs. We’ll be releasing an enhancement for this very soon, but in the meantime you can manually free that slot.

Hi @alex, Ideally we only have one flow running at the a time, I had to increase it to 3 because it would block all future flows. It is not possible for me to constantly monitor our flows and manually release the slots in case the a in an invalid state. What else can be done?

In the next release (3.4.11, which might go out today), deployment concurrency will be updated to use a leasing system where slots are automatically returned if not renewed, so if you upgrade, then you shouldn't have to worry about this anymore!

Hi @alex, Perfect - we are using prefect cloud, self-hosted infrastructure. Since we are using prefect cloud, does it upgrade automatically or is it the workers we are updating?

You'll need to upgrade the

prefect

version you use when running flows because the client updates to maintain concurrency leases haven't been released yet. This is the PR with the updates, if you're interested.

Hi @alex, I have updated my docker images to use prefect 3.4.11, but it still says:

"resource": {
    "prefect.resource.id": "prefect.worker.kubernetes.kubernetesworker-0840632d-446d-4745-a2d7-8b255b2a1d20",
    "prefect.resource.name": "KubernetesWorker 0840632d-446d-4745-a2d7-8b255b2a1d20",
    "prefect.version": "3.3.6",
    "prefect.worker-type": "kubernetes"
  },

Do I have to update the worker as well? They are hosted in Kubernetes, but I cannot find any reference to the prefect version. Requirements file have: prefect==3.4.11 Dockerfile:

FROM python:3.9-slim
...

# Install your shared requirements
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy in your flow files
COPY flows/ /app/flows/

# Make sure /app is on PYTHONPATH so you can do `import flows.flow_a` etc.
WORKDIR /app
ENV PYTHONPATH=/app

I'm pretty sure you don't need to upgrade your worker

prefect

version for concurrency leasing. Just the version in the Docker image you use to run flows should suffice