Title
m

Mark McDonald

05/21/2021, 1:01 PM
Hi - I ran into a problem recently with the Task Concurrency Limiting feature. I have a task tag that is supposed to throttle the number of tasks that can interact with a particular database. In the past, this tag was only applied to a task that is mapped in a single flow. Yesterday, I tried applying this same tag to a separate mapped task in a different flow that interacts with this same database. At first, it worked as I expected. However then I saw that any task with this tag applied to it would just sit in state of "Queued" and never move into "Running". I made sure that no other tasks were running with tasks using this tag, but the behavior remained the same. I ended up having to delete the tag to get the tasks to run. My questions are: 1. is it appropriate to have a single task tag shared between different flow's tasks that have a common resource dependency limitation like the one I've described? 2. Any idea why all these tasks were getting stuck in a "Queued" state when there were no actively running tasks with that tag?
k

Kevin Kho

05/21/2021, 1:20 PM
Hi @Mark McDonald, Yes to number 1. For number 2, how long were they stuck in the Queued state?
m

Mark McDonald

05/21/2021, 1:35 PM
these tasks would have stayed in Queued indefinitely had I not cancelled the flow run. As an example, this morning we had a run where the task stayed in a queued state for ~2.5 hours. I ended up deleting the task 'tag' from the UI, and at that point the task started running. You can see the big white gap in this screenshot is the period during which the task was stuck in the queued state.
k

Kevin Kho

05/21/2021, 1:37 PM
Ok will bring this up to the team
m

Mark McDonald

05/21/2021, 1:41 PM
thanks, Kevin
k

Kevin Kho

05/21/2021, 2:01 PM
Can you give me some example flow urls?
so this is really weird. I deleted the tag a couple of hours ago. I added the tag back (through the UI) using the same tag name. It immediately said that there is 1 task running with this tag.
do you know if there is a way to query the tag through graphql to see what this task is that is running? I'm sure that there is no flow currently running that would have a task using this tag
k

Kevin Kho

05/21/2021, 2:10 PM
I’ll look into it
Not seeing it immediate. Will add this to what i communicate to the team
Try something like this? Change Success to Running
query {
  task (where: {task_runs: {state: {_eq: "Success"}}}){
    	flow {
    	  name
    	}
      name
      tags
      task_runs {
        state
      }
    }
  }
m

Mark McDonald

05/21/2021, 2:33 PM
interesting... that does return a response that indicates that the flow that uses this tag has some tasks in a running state... however this is not what the UI shows. The UI doesn't show this flow running at all. Also, looking at the AWS Fargate side, and I can tell that we definitely don't have this flow running on our cluster currently.
k

Kevin Kho

05/21/2021, 2:35 PM
Could you give that flow id?
m

Mark McDonald

05/21/2021, 2:35 PM
82239b9e-d455-4e5d-9f96-f512087cd001
k

Kevin Kho

05/21/2021, 2:45 PM
Sorry I meant flow run id of the one still RUNNING. Is that what you gave me? Just making sure
m

Mark McDonald

05/21/2021, 2:48 PM
no that's the flow id - not the run, let me see
this is the flow run id: 7bfa9981-cb28-4a29-b35f-0b3fecaeb6b3
ahh - the flow was cancelled - I guess the mapped tasks aren't in a cancelled state
k

Kevin Kho

05/21/2021, 2:51 PM
What state are the mapped tasks in?
m

Mark McDonald

05/21/2021, 2:51 PM
running
k

Kevin Kho

05/21/2021, 2:52 PM
Thanks for the info! I forwarded all the details to the team and I’ll get back to you
m

Mark McDonald

05/21/2021, 2:52 PM
I think I understand what's happening here. When I cancelled the flow, I did not click this option:
so the mapped tasks are forever stuck in a "Running" state
and will always block any other tasks with that tag from running
k

Kevin Kho

05/21/2021, 2:54 PM
Yeah this seems like an issue
m

Mark McDonald

05/21/2021, 2:55 PM
ya, it's not ideal. Maybe when you cancel a flow, you can include mapped tasks as cancelled by default
k

Kevin Kho

05/21/2021, 4:54 PM
We are opening an issue for this in our internal repo. This is a bug.
m

Mark McDonald

05/21/2021, 6:10 PM
sounds good. Thanks for your help, Kevin