Frank Hardisty

    Frank Hardisty

    3 months ago
    Hello, Frank Hardisty of Replica here, we are heavy Prefect users, thanks for all the great functionality! Our organization has some Task Concurrency limits, very useful. However, some of them have become inaccurate, for example we have one called
    ClickhouseODHourly
    which is permanently stuck at 136 even when nothing is running.
    Kevin Kho

    Kevin Kho

    3 months ago
    Hey Frank, did you try exploring if there are tasks stuck in a running state with the following query?
    query {
      task_run (where: {state: {_eq: "Running"},
                        task: {tags: {_contains: "tag"}}}) {
        id
        flow_run{
          name
          id
        }
        task{
          name
          tags
        }
      }
    }
    Frank Hardisty

    Frank Hardisty

    3 months ago
    Thanks so much Kevin! Just got sucked into a meeting, I'll run that pronto.
    Thanks for the tip on how easy it is to run GraphQL from the Prefect console, from the
    Interactive API
    tab up top in the UI.
    query came back clean
    {
      "data": {
        "task_run": []
      }
    }
    I tried deleting the task concurrency limits and re-creating them, but they still had the "ghost" numbers in them.
    Also if you know how we can keep the "ghost" runs from creeping in, please let me know
    Kevin Kho

    Kevin Kho

    3 months ago
    The query pulls from the database in a similar way. There are tasks in a running state there. What happens if you take out the tag clause?
    Just get runs in a running state
    Frank Hardisty

    Frank Hardisty

    3 months ago
    oh my yes
    Huh, so those are task runs that are no longer associated with their parent Flow Runs?
    Kevin Kho

    Kevin Kho

    3 months ago
    That’s weird since these don’t have tags. So these are running but you think the flow run is already no longer running?
    Frank Hardisty

    Frank Hardisty

    3 months ago
    checking....
    No... there are many duplicates there
    I think that list is accurate
    we have two Flow Runs right now, with many tasks.
    none of them are on the tagged list though
    So, back to the original question, how to clear out the "ghost tasks"? Thanks!
    Kevin Kho

    Kevin Kho

    3 months ago
    We can’t clear them out if we can’t find them like this. The goal is to get a bunch of id’s and then we move them to a failed of success state to get them out of running
    Can you try using the tag but not a running state?
    Frank Hardisty

    Frank Hardisty

    3 months ago
    Here's what I tried:
    query {
      task_run (where: {state: {_neq: "Running"},
                        task: {tags: {_contains: "tag"}}}) {
        id
        flow_run{
          name
          id
        }
        task{
          name
          tags
        }
      }
    }
    first I got an error, then
    {
      "data": {
        "task_run": []
      }
    }
    I think I have the IDs from the Chrome Console
    Kevin Kho

    Kevin Kho

    3 months ago
    You need to change the tag with the tag that is not going down
    Frank Hardisty

    Frank Hardisty

    3 months ago
    bingo!
    There's 136 of them, sorry to say
    There's others that are smaller, if that would be helpful.
    Kevin Kho

    Kevin Kho

    3 months ago
    Exactly, we need to find if these are indeed running or what happened to our flow so you can get a flow run id and look at it
    Frank Hardisty

    Frank Hardisty

    3 months ago
    OK, took the first Flow Run id, and tried it in the UI
    the Flow Run succeeded ~2.5 hours ago
    Kevin Kho

    Kevin Kho

    3 months ago
    Flow run succeeded but are there tasks left in a Running state?
    Frank Hardisty

    Frank Hardisty

    3 months ago
    ug no sorry.... I had the query still at
    neq
    ... trying again
    With the corrected query, I got a bunch of Flow Run ids back
    checking the first one, UI is empty
    I'll try to add "Flow Run is not in state Running" to the query
    Kevin Kho

    Kevin Kho

    3 months ago
    i think it might just take a while to load, especially if it has tons of tasks?
    Frank Hardisty

    Frank Hardisty

    3 months ago
    it might, thanks for the tip
    I'll do a separate query to get the status of that first Flow Run
    OK checked the last one on the list, yes the Flow Run concluded on May 25 in state
    Failed
    but the Task Run is returned by this query:
    query {
      task_run (where: {state: {_eq: "Running"},
                        task: {tags: {_contains: "ClickhouseODHourly"}}}) {
        id
        flow_run{
          name
          id
        }
        task{
          name
          tags
        }
      }
    }
    OK I can see them in the UI in state
    Running
    Kevin Kho

    Kevin Kho

    3 months ago
    Ah ok. Hard to tell what happened here, but I think the approach is to use Python to get the GraphQL API, get the tasks, and then move them all to a Failed state. I have a conference at like 8:30 PM ET I have to prepare for to fully make the queries until after. I can help with that or you can take a stab. But yes, moving these to Failed will free up your concurrency
    Frank Hardisty

    Frank Hardisty

    3 months ago
    Got it... thanks. Happily this particular task is someone else's responsibility 😄
    thanks a ton Kevin, your help was spot on! Kudos!
    Kevin Kho

    Kevin Kho

    3 months ago
    Of course!