Channels
#prefect-community
Title
# prefect-community
f
Frank Hardisty
05/31/2022, 9:12 PMHello, Frank Hardisty of Replica here, we are heavy Prefect users, thanks for all the great functionality!
Our organization has some Task Concurrency limits, very useful. However, some of them have become inaccurate, for example we have one called
ClickhouseODHourlyk
Kevin Kho
05/31/2022, 9:16 PM
Hey Frank, did you try exploring if there are tasks stuck in a running state with the following query?
Copy code
query {
task_run (where: {state: {_eq: "Running"},
task: {tags: {_contains: "tag"}}}) {
id
flow_run{
name
id
}
task{
name
tags
}
}
}👀 1
f
Frank Hardisty
05/31/2022, 9:18 PMThanks so much Kevin! Just got sucked into a meeting, I'll run that pronto.
Thanks for the tip on how easy it is to run GraphQL from the Prefect console, from the
Interactive APIquery came back clean
Copy code
{
"data": {
"task_run": []
}
}I tried deleting the task concurrency limits and re-creating them, but they still had the "ghost" numbers in them.
Also if you know how we can keep the "ghost" runs from creeping in, please let me know
k
Kevin Kho
05/31/2022, 10:06 PM
The query pulls from the database in a similar way. There are tasks in a running state there. What happens if you take out the tag clause?
Just get runs in a running state
f
Frank Hardisty
05/31/2022, 10:09 PMoh my yes
Huh, so those are task runs that are no longer associated with their parent Flow Runs?
k
Kevin Kho
05/31/2022, 10:12 PM
That’s weird since these don’t have tags. So these are running but you think the flow run is already no longer running?
f
Frank Hardisty
05/31/2022, 10:13 PMchecking....
No... there are many duplicates there
I think that list is accurate
we have two Flow Runs right now, with many tasks.
none of them are on the tagged list though
So, back to the original question, how to clear out the "ghost tasks"? Thanks!
k
Kevin Kho
05/31/2022, 10:18 PM
We can’t clear them out if we can’t find them like this. The goal is to get a bunch of id’s and then we move them to a failed of success state to get them out of running
Can you try using the tag but not a running state?
👍 1
f
Frank Hardisty
05/31/2022, 10:21 PMHere's what I tried:
Copy code
query {
task_run (where: {state: {_neq: "Running"},
task: {tags: {_contains: "tag"}}}) {
id
flow_run{
name
id
}
task{
name
tags
}
}
}first I got an error, then
Copy code
{
"data": {
"task_run": []
}
}I think I have the IDs from the Chrome Console
k
Kevin Kho
05/31/2022, 10:22 PM
You need to change the tag with the tag that is not going down
🙏 1
✅ 1
f
Frank Hardisty
05/31/2022, 10:23 PMbingo!
Untitled.txt
There's 136 of them, sorry to say
There's others that are smaller, if that would be helpful.
k
Kevin Kho
05/31/2022, 10:25 PM
Exactly, we need to find if these are indeed running or what happened to our flow so you can get a flow run id and look at it
f
Frank Hardisty
05/31/2022, 10:26 PMOK, took the first Flow Run id, and tried it in the UI
the Flow Run succeeded ~2.5 hours ago
k
Kevin Kho
05/31/2022, 10:28 PM
Flow run succeeded but are there tasks left in a Running state?
f
Frank Hardisty
05/31/2022, 10:28 PMug no sorry.... I had the query still at
neqWith the corrected query, I got a bunch of Flow Run ids back
checking the first one, UI is empty
I'll try to add "Flow Run is not in state `Running`" to the query
k
Kevin Kho
05/31/2022, 10:33 PM
i think it might just take a while to load, especially if it has tons of tasks?
f
Frank Hardisty
05/31/2022, 10:33 PMit might, thanks for the tip
I'll do a separate query to get the status of that first Flow Run
OK checked the last one on the list, yes the Flow Run concluded on May 25 in state
Failedbut the Task Run is returned by this query:
Copy code
query {
task_run (where: {state: {_eq: "Running"},
task: {tags: {_contains: "ClickhouseODHourly"}}}) {
id
flow_run{
name
id
}
task{
name
tags
}
}
}OK I can see them in the UI in state
Runningk
Kevin Kho
05/31/2022, 10:39 PM
Ah ok. Hard to tell what happened here, but I think the approach is to use Python to get the GraphQL API, get the tasks, and then move them all to a Failed state. I have a conference at like 8:30 PM ET I have to prepare for to fully make the queries until after. I can help with that or you can take a stab. But yes, moving these to Failed will free up your concurrency
f
Frank Hardisty
05/31/2022, 10:40 PMGot it... thanks. Happily this particular task is someone else's responsibility 😄
thanks a ton Kevin, your help was spot on! Kudos!
k
Kevin Kho
05/31/2022, 10:41 PM
Of course!
6 Views