When my tasks crash that use concurrency limit slots they do

Question

When my tasks crash that use concurrency limit slots they don t seem to clear out of the concurrency limit queue ever Is there a way to set a timeout on concurrency limit slots Now none of the preceding tasks can run because the slots are full with tasks they think are still running but they are not

Bianca Hoch · Answer

Hey Mike I have a few clarifying q s for you the tasks are entering a `Crashed` state is that correct what version of prefect are you using currently output of `prefect version`

Mike Safruk · Answer

Hey Bianca Thanks for getting back to me The Flow that creates the tasks is in a crashed state The tasks them self get stuck in a Running state indefinitely so they never get removed from the task concurrency queue and the next time the flow runs the task concurrency limit is reached until I delete the concurrency queue and re create it to clear the stuck running tasks in it I am using Prefect version 2 10 4

Bianca Hoch · Answer

Hey Mike this seems related to <https github com PrefectHQ prefect issues 5995|this open issue> From what I understand a fix for this is currently being planned by our engineers

Bianca Hoch · Answer

In the meantime we do have a method to <https docs prefect io latest api ref prefect client orchestration prefect client orchestration PrefectClient reset concurrency limit by tag|reset the concurrency limit> on a specific tag It may be beneficial to create an automation which triggers a helper flow that resets the concurrency limit for the affected tag in response to a flow entering a `CRASHED` state

Mike Safruk · Answer

OK sounds good Bianca Thanks for the information Any line of sight into when that issues fix will be merged Looks like it s been open since July 2022

Ghislain Picard · Answer

FYI I still have plenty of tasks creashing in subflow that do not stop the main flow The flow is in running state why the sub flow is crashed This plays badly with the concurrency limit