Hi Prefect Devs. Id like to discuss about a possible bug regarding Task Concurrency Flag not being cleared up.
Details in thread,
In our prefect dashboard, I noticed that a work queue was "Unhealthy" and had lots of "Late" flow runs. The work queue had 20 concurrency limit and all the task in the flows in that work queue have "Task run concurrency tags" assigned.
So I checked the concurrency tag limit and noticed this one task that has been running for 10h+. The tasks usually finishes in few mins.
I looked at the task and it is Running.
I checked the flow - the flow has already failed.
The flow run has failed with the following log.
Bumping prefect devs again
07/18/2023, 5:28 PM
Hey Decevious, thanks for raising this. We've shared this with other members of the team to get more eyes on the matter.
Are any of the flows with concurrency limited tasks subflows by chance?
07/18/2023, 5:34 PM
My flow calls around 17 sub flows sequentially. Each sub flow has around 50 api calls that are called sequentially. the api calls are task that have been tagged with the limit.
Not sure if that answers ur question.
The flow by itself is not tagged with the task concurrency limit tag.