Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
m
Mark McDonald
05/21/2021, 1:01 PMHi - I ran into a problem recently with the Task Concurrency Limiting feature. I have a task tag that is supposed to throttle the number of tasks that can interact with a particular database. In the past, this tag was only applied to a task that is mapped in a single flow. Yesterday, I tried applying this same tag to a separate mapped task in a different flow that interacts with this same database.
At first, it worked as I expected. However then I saw that any task with this tag applied to it would just sit in state of "Queued" and never move into "Running". I made sure that no other tasks were running with tasks using this tag, but the behavior remained the same. I ended up having to delete the tag to get the tasks to run.
My questions are:
1. is it appropriate to have a single task tag shared between different flow's tasks that have a common resource dependency limitation like the one I've described?
2. Any idea why all these tasks were getting stuck in a "Queued" state when there were no actively running tasks with that tag?
k
Kevin Kho
05/21/2021, 1:20 PMHi @Mark McDonald, Yes to number 1. For number 2, how long were they stuck in the Queued state?
m
Mark McDonald
05/21/2021, 1:35 PMthese tasks would have stayed in Queued indefinitely had I not cancelled the flow run. As an example, this morning we had a run where the task stayed in a queued state for ~2.5 hours. I ended up deleting the task 'tag' from the UI, and at that point the task started running. You can see the big white gap in this screenshot is the period during which the task was stuck in the queued state.
k
Kevin Kho
05/21/2021, 1:37 PMOk will bring this up to the team
m
Mark McDonald
05/21/2021, 1:41 PMthanks, Kevin
k
Kevin Kho
05/21/2021, 2:01 PMCan you give me some example flow urls?
m
Mark McDonald
05/21/2021, 2:01 PMso this is really weird. I deleted the tag a couple of hours ago. I added the tag back (through the UI) using the same tag name. It immediately said that there is 1 task running with this tag.
do you know if there is a way to query the tag through graphql to see what this task is that is running? I'm sure that there is no flow currently running that would have a task using this tag
k
Kevin Kho
05/21/2021, 2:10 PMI’ll look into it
Not seeing it immediate. Will add this to what i communicate to the team
Try something like this? Change Success to Running
query {
task (where: {task_runs: {state: {_eq: "Success"}}}){
flow {
name
}
name
tags
task_runs {
state
}
}
}m
Mark McDonald
05/21/2021, 2:33 PMinteresting... that does return a response that indicates that the flow that uses this tag has some tasks in a running state... however this is not what the UI shows. The UI doesn't show this flow running at all. Also, looking at the AWS Fargate side, and I can tell that we definitely don't have this flow running on our cluster currently.
k
Kevin Kho
05/21/2021, 2:35 PMCould you give that flow id?
m
Mark McDonald
05/21/2021, 2:35 PM82239b9e-d455-4e5d-9f96-f512087cd001
k
Kevin Kho
05/21/2021, 2:45 PMSorry I meant flow run id of the one still RUNNING. Is that what you gave me? Just making sure
m
Mark McDonald
05/21/2021, 2:48 PMno that's the flow id - not the run, let me see
this is the flow run id: 7bfa9981-cb28-4a29-b35f-0b3fecaeb6b3
ahh - the flow was cancelled - I guess the mapped tasks aren't in a cancelled state
k
Kevin Kho
05/21/2021, 2:51 PMWhat state are the mapped tasks in?
m
Mark McDonald
05/21/2021, 2:51 PMrunning
k
Kevin Kho
05/21/2021, 2:52 PMThanks for the info! I forwarded all the details to the team and I’ll get back to you
m
Mark McDonald
05/21/2021, 2:52 PMI think I understand what's happening here. When I cancelled the flow, I did not click this option:
so the mapped tasks are forever stuck in a "Running" state
and will always block any other tasks with that tag from running
k
Kevin Kho
05/21/2021, 2:54 PMYeah this seems like an issue
m
Mark McDonald
05/21/2021, 2:55 PMya, it's not ideal. Maybe when you cancel a flow, you can include mapped tasks as cancelled by default
k
Kevin Kho
05/21/2021, 4:54 PMWe are opening an issue for this in our internal repo. This is a bug.
m
Mark McDonald
05/21/2021, 6:10 PMsounds good. Thanks for your help, Kevin