Hello Frank Hardisty of Replica here we are heavy Prefect us

Question

Hello Frank Hardisty of Replica here we are heavy Prefect users thanks for all the great functionality Our organization has some Task Concurrency limits very useful However some of them have become inaccurate for example we have one called `ClickhouseODHourly` which is permanently stuck at 136 even when nothing is running

Kevin Kho · Answer

Hey Frank did you try exploring if there are tasks stuck in a running state with the following query ```query task run where state eq Running task tags contains tag id flow run name id task name tags ```

Frank Hardisty · Answer

Thanks so much Kevin Just got sucked into a meeting I ll run that pronto

Frank Hardisty · Answer

Thanks for the tip on how easy it is to run GraphQL from the Prefect console from the `Interactive API` tab up top in the UI

Frank Hardisty · Answer

query came back clean

Frank Hardisty · Answer

``` data task run ```

Frank Hardisty · Answer

I tried deleting the task concurrency limits and re creating them but they still had the ghost numbers in them

Frank Hardisty · Answer

Also if you know how we can keep the ghost runs from creeping in please let me know

Kevin Kho · Answer

The query pulls from the database in a similar way There are tasks in a running state there What happens if you take out the tag clause

Kevin Kho · Answer

Just get runs in a running state

Frank Hardisty · Answer

oh my yes

Frank Hardisty · Answer

Huh so those are task runs that are no longer associated with their parent Flow Runs

Kevin Kho · Answer

That s weird since these don t have tags So these are running but you think the flow run is already no longer running

Frank Hardisty · Answer

checking

Frank Hardisty · Answer

No there are many duplicates there

Frank Hardisty · Answer

I think that list is accurate

Frank Hardisty · Answer

we have two Flow Runs right now with many tasks

Frank Hardisty · Answer

none of them are on the tagged list though

Frank Hardisty · Answer

So back to the original question how to clear out the ghost tasks Thanks

Kevin Kho · Answer

We can t clear them out if we can t find them like this The goal is to get a bunch of id s and then we move them to a failed of success state to get them out of running

Kevin Kho · Answer

Can you try using the tag but not a running state

Frank Hardisty · Answer

Here s what I tried

Frank Hardisty · Answer

```query task run where state neq Running task tags contains tag id flow run name id task name tags ```

Frank Hardisty · Answer

first I got an error then ``` data task run ```

Frank Hardisty · Answer

I think I have the IDs from the Chrome Console

Kevin Kho · Answer

You need to change the tag with the tag that is not going down

Frank Hardisty · Answer

bingo

Frank Hardisty · Answer

Untitled txt

Frank Hardisty · Answer

There s 136 of them sorry to say

Frank Hardisty · Answer

There s others that are smaller if that would be helpful

Kevin Kho · Answer

Exactly we need to find if these are indeed running or what happened to our flow so you can get a flow run id and look at it

Frank Hardisty · Answer

OK took the first Flow Run id and tried it in the UI

Frank Hardisty · Answer

the Flow Run succeeded ~2 5 hours ago

Kevin Kho · Answer

Flow run succeeded but are there tasks left in a Running state

Frank Hardisty · Answer

ug no sorry I had the query still at `neq` trying again

Frank Hardisty · Answer

With the corrected query I got a bunch of Flow Run ids back

Frank Hardisty · Answer

checking the first one UI is empty

Frank Hardisty · Answer

I ll try to add Flow Run is not in state `Running` to the query

Kevin Kho · Answer

i think it might just take a while to load especially if it has tons of tasks

Frank Hardisty · Answer

it might thanks for the tip

Frank Hardisty · Answer

I ll do a separate query to get the status of that first Flow Run

Frank Hardisty · Answer

OK checked the last one on the list yes the Flow Run concluded on May 25 in state `Failed`

Frank Hardisty · Answer

but the Task Run is returned by this query ```query task run where state eq Running task tags contains ClickhouseODHourly id flow run name id task name tags ```

Frank Hardisty · Answer

OK I can see them in the UI in state `Running`

Kevin Kho · Answer

Ah ok Hard to tell what happened here but I think the approach is to use Python to get the GraphQL API get the tasks and then move them all to a Failed state I have a conference at like 8 30 PM ET I have to prepare for to fully make the queries until after I can help with that or you can take a stab But yes moving these to Failed will free up your concurrency

Frank Hardisty · Answer

Got it thanks Happily this particular task is someone else s responsibility smile

Frank Hardisty · Answer

thanks a ton Kevin your help was spot on Kudos

Kevin Kho · Answer

Of course