https://prefect.io
Join Slack
Hello, Frank Hardisty of Replica here, we are heavy Prefect users, thanks for all the great function...
# ask-community
f
Hello, Frank Hardisty of Replica here, we are heavy Prefect users, thanks for all the great functionality! Our organization has some Task Concurrency limits, very useful. However, some of them have become inaccurate, for example we have one called 
ClickhouseODHourly
which is permanently stuck at 136 even when nothing is running.
k
Hey Frank, did you try exploring if there are tasks stuck in a running state with the following query?
Copy code
query {
  task_run (where: {state: {_eq: "Running"},
                    task: {tags: {_contains: "tag"}}}) {
    id
    flow_run{
      name
      id
    }
    task{
      name
      tags
    }
  }
}
👀 1
f
Thanks so much Kevin! Just got sucked into a meeting, I'll run that pronto.
Thanks for the tip on how easy it is to run GraphQL from the Prefect console, from the 
Interactive API
tab up top in the UI.
query came back clean
Copy code
{
  "data": {
    "task_run": []
  }
}
I tried deleting the task concurrency limits and re-creating them, but they still had the "ghost" numbers in them.
Also if you know how we can keep the "ghost" runs from creeping in, please let me know
k
The query pulls from the database in a similar way. There are tasks in a running state there. What happens if you take out the tag clause?
Just get runs in a running state
f
oh my yes
Huh, so those are task runs that are no longer associated with their parent Flow Runs?
k
That’s weird since these don’t have tags. So these are running but you think the flow run is already no longer running?
f
checking....
No... there are many duplicates there
I think that list is accurate
we have two Flow Runs right now, with many tasks.
none of them are on the tagged list though
So, back to the original question, how to clear out the "ghost tasks"? Thanks!
k
We can’t clear them out if we can’t find them like this. The goal is to get a bunch of id’s and then we move them to a failed of success state to get them out of running
Can you try using the tag but not a running state?
👍 1
f
Here's what I tried:
Copy code
query {
  task_run (where: {state: {_neq: "Running"},
                    task: {tags: {_contains: "tag"}}}) {
    id
    flow_run{
      name
      id
    }
    task{
      name
      tags
    }
  }
}
first I got an error, then
Copy code
{
  "data": {
    "task_run": []
  }
}
I think I have the IDs from the Chrome Console
k
You need to change the tag with the tag that is not going down
🙏 1
✅ 1
f
bingo!
Untitled.txt
Untitled.txt
There's 136 of them, sorry to say
There's others that are smaller, if that would be helpful.
k
Exactly, we need to find if these are indeed running or what happened to our flow so you can get a flow run id and look at it
f
OK, took the first Flow Run id, and tried it in the UI
the Flow Run succeeded ~2.5 hours ago
k
Flow run succeeded but are there tasks left in a Running state?
f
ug no sorry.... I had the query still at 
neq
... trying again
With the corrected query, I got a bunch of Flow Run ids back
checking the first one, UI is empty
I'll try to add "Flow Run is not in state `Running`" to the query
k
i think it might just take a while to load, especially if it has tons of tasks?
f
it might, thanks for the tip
I'll do a separate query to get the status of that first Flow Run
OK checked the last one on the list, yes the Flow Run concluded on May 25 in state 
Failed
but the Task Run is returned by this query:
Copy code
query {
  task_run (where: {state: {_eq: "Running"},
                    task: {tags: {_contains: "ClickhouseODHourly"}}}) {
    id
    flow_run{
      name
      id
    }
    task{
      name
      tags
    }
  }
}
OK I can see them in the UI in state 
Running
k
Ah ok. Hard to tell what happened here, but I think the approach is to use Python to get the GraphQL API, get the tasks, and then move them all to a Failed state. I have a conference at like 8:30 PM ET I have to prepare for to fully make the queries until after. I can help with that or you can take a stab. But yes, moving these to Failed will free up your concurrency
f
Got it... thanks. Happily this particular task is someone else's responsibility 😄
thanks a ton Kevin, your help was spot on! Kudos!
k
Of course!
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by