Channels
#prefect-community
Title
s
Shane Breeze
09/04/2023, 9:43 AMHi everyone, I’m having an issue with pending tasks blocking the task concurrency limit slots. It is quite difficult to replicate as it seems spurious and happens more frequently for long running / complex task dependencies, so unfortunately I haven’t been able to write simple example code to replicate this.
If this issue happens multiple times it can block all slots of a concurrency limit and cause a deadlock as preceding tasks cannot run as all the concurrency limit slots are taken by dependent tasks. Should tasks only occupy the concurrency limit only if they are “running”?
My current workaround is to delete the concurrency limit and recreate it to clear out the pending tasks. But this requires manual intervention
I’m also using prefect 2.12.0, but have seen the same issue on previous versions
b
Bianca Hoch
09/05/2023, 5:12 PMHi Shane, thanks for reporting this. I believe that tasks which are in a transitional state (Cancelling, Running, Pending) do occupy concurrency slots
With that being said, I'm curious to hear about what the tasks are doing. Are you passing large amounts of data between them? How long do they typically hang out in a pending state for?
s
Shane Breeze
09/06/2023, 7:39 AMHi Bianca, they’re mostly data engineering tasks running a mix of database queries (hence the concurrency limit to avoid overloading the database) and pandas. There’s no data transfer between the tasks, these are saved in the database instead. However, there is a large task dependency multiple layers deep.
It seems as though the server attempts to run a task and adds it to the concurrency limit. Then it checks for task dependencies and places it in “pending” whilst it waits for other tasks to complete, but it does not remove it from the concurrency limit.