@Martin how can I ensure that concurrency control slot is released if a prefect flow crashes while it’s running in our k8s environment?

@Marvin say hello directly if you're alive

✅ thought for 4.9 seconds

@Martin how can I ensure that concurrency control slot is released if a prefect flow crashes while it’s running in our k8s environment?

@Marvin how can I ensure that concurrency control slot is released if a prefect flow crashes while it’s running in our k8s environment?

✅ thought for 57.5 seconds

Omg I keep pinging Martin and not Marvin. I’m sorry Martin and Marvin.

adjusts circuits with mild interest The good news is that Prefect has built-in safeguards for this exact scenario. Here's how concurrency slots are handled in a Kubernetes environment: 1. Automatic Release Mechanism - Concurrency slots are automatically released when a flow run exits the 'Running' or 'Cancelling' state - This happens regardless of how the flow run ends (crash, completion, or cancellation) - The release mechanism is handled by Prefect's core orchestration layer, not the infrastructure layer 2. Kubernetes-Specific Behavior - When running in Kubernetes, each flow run executes in its own pod - If a pod crashes or is terminated, Prefect's orchestration layer will detect this and update the flow run state accordingly - This state change triggers the automatic release of any held concurrency slots 3. Additional Safety Measures For extra protection, you can: - Set reasonable timeouts on your flow runs using the

timeout_seconds

parameter - Configure Kubernetes liveness and readiness probes in your work pool configuration - Use Kubernetes resource limits to prevent runaway processes Here's an example of how to configure timeouts in your flow:

python
from prefect import flow

@flow(timeout_seconds=3600)  # Flow will automatically timeout after 1 hour
def my_flow():
    # your flow code here
    pass

If you're experiencing issues where slots seem to remain locked despite crashes, it's worth checking: 1. That your Prefect server can maintain consistent connectivity to its database 2. That your Kubernetes pods have appropriate resource limits and requests set 3. That your network allows consistent communication between your worker and the Prefect server sighs deeply I suppose I should mention that if you're still having issues with slots not being released properly, you can manually release them through the Prefect UI or API, though this should rarely be necessary. The automatic release mechanism is quite reliable, even if the concept of reliability in an inherently chaotic universe is somewhat paradoxical.

Looks like there’s a bug https://github.com/PrefectHQ/prefect/issues/17415

that's a planned enhancement that we're working on, ie make it easier to avoid orphaned slots. its a known limitation of the current way concurrency limits works, not necessarily a bug

a known limitation of the current way

Is that documented anywhere for us to review? Another user put in a bug related to this.

With a concurrency limit, slots are released when the concurrency manager exits.

Exit in this context means only graceful shutdown, i.e. any failure is not compensated for? It might be good to have an explicit call out in the docs for this. Any recommendation on a way to detect this and self-heal?

correct that is what the existing issue about orphaned slots articulates and is the point of the lease-based enhancement outlined there you can use the prefect client as other users mention on that issue, that is the best way until the enhancement is released

I asked on the ticket, but is there some sample code we can use to get started with this work around?

how exactly you might want to do it I think would depend, but this is one way that might make sense

async def handle_orphan_slots_on_crashed(flow, flow_run, state):
   async with get_client() as client:
      await client.update_global_concurrency_limit(...)

@flow(on_crashed=[handle_orphan_slots_on_crashed])
def f(...): ...