https://prefect.io logo
Join SlackCommunities
Powered by
Hi - I ran into a problem recently with the Task C...
# ask-community
m
Hi - I ran into a problem recently with the Task Concurrency Limiting feature. I have a task tag that is supposed to throttle the number of tasks that can interact with a particular database. In the past, this tag was only applied to a task that is mapped in a single flow. Yesterday, I tried applying this same tag to a separate mapped task in a different flow that interacts with this same database. At first, it worked as I expected. However then I saw that any task with this tag applied to it would just sit in state of "Queued" and never move into "Running". I made sure that no other tasks were running with tasks using this tag, but the behavior remained the same. I ended up having to delete the tag to get the tasks to run. My questions are: 1. is it appropriate to have a single task tag shared between different flow's tasks that have a common resource dependency limitation like the one I've described? 2. Any idea why all these tasks were getting stuck in a "Queued" state when there were no actively running tasks with that tag?
k
Hi @Mark McDonald, Yes to number 1. For number 2, how long were they stuck in the Queued state?
m
these tasks would have stayed in Queued indefinitely had I not cancelled the flow run. As an example, this morning we had a run where the task stayed in a queued state for ~2.5 hours. I ended up deleting the task 'tag' from the UI, and at that point the task started running. You can see the big white gap in this screenshot is the period during which the task was stuck in the queued state.
k
Ok will bring this up to the team
m
thanks, Kevin
k
Can you give me some example flow urls?
m
so this is really weird. I deleted the tag a couple of hours ago. I added the tag back (through the UI) using the same tag name. It immediately said that there is 1 task running with this tag.
do you know if there is a way to query the tag through graphql to see what this task is that is running? I'm sure that there is no flow currently running that would have a task using this tag
k
I’ll look into it
Not seeing it immediate. Will add this to what i communicate to the team
Try something like this? Change Success to Running
Copy code
query {
  task (where: {task_runs: {state: {_eq: "Success"}}}){
    	flow {
    	  name
    	}
      name
      tags
      task_runs {
        state
      }
    }
  }
m
interesting... that does return a response that indicates that the flow that uses this tag has some tasks in a running state... however this is not what the UI shows. The UI doesn't show this flow running at all. Also, looking at the AWS Fargate side, and I can tell that we definitely don't have this flow running on our cluster currently.
k
Could you give that flow id?
m
82239b9e-d455-4e5d-9f96-f512087cd001
k
Sorry I meant flow run id of the one still RUNNING. Is that what you gave me? Just making sure
m
no that's the flow id - not the run, let me see
this is the flow run id: 7bfa9981-cb28-4a29-b35f-0b3fecaeb6b3
ahh - the flow was cancelled - I guess the mapped tasks aren't in a cancelled state
k
What state are the mapped tasks in?
m
running
k
Thanks for the info! I forwarded all the details to the team and I’ll get back to you
m
I think I understand what's happening here. When I cancelled the flow, I did not click this option:
so the mapped tasks are forever stuck in a "Running" state
and will always block any other tasks with that tag from running
k
Yeah this seems like an issue
m
ya, it's not ideal. Maybe when you cancel a flow, you can include mapped tasks as cancelled by default
k
We are opening an issue for this in our internal repo. This is a bug.
m
sounds good. Thanks for your help, Kevin
33 Views
Open in Slack
PreviousNext
Powered by