https://prefect.io logo
Join Slack
Powered by
Hello, We currently have no running flows however ...
# prefect-community
t
Hello, We currently have no running flows however when looking at the "Task Concurrency" page, I have a tag for which it says that 2 task currently running 😮
k
Hey Thomas, didn’t get a chance to respond to your other message yet but you can run a query like:
Copy code
query {
  task_run (where: {state: {_eq: "Running"},
                    task: {tags: {_contains: "tag"}}}) {
    id
    flow_run{
      name
      id
    }
    task{
      name
      tags
    }
  }
}
to find the responsible tasks that remain in a running state and marking them as Failed or Completed through the UI
upvote 1
t
Thanks, I'll try that 🙂
It works, I found the 2 tasks. The strange thing is that the flow are in a Failed state, but there is a task inside which stays in a running/blue state. I think those tasks/flows have failed because the pods were OOM by k8s. Here are the flow run ids, if it's of any use: • d2b9e49a-9c09-4c17-8037-266e29dea7fe • 8a7379de-013a-46d0-8c50-f97137362781 My guess is that it is related to my other thread on pod not being terminated after failures...
k
You can just mark them as failed
t
Is it possible to modify your query to only get the "running" tasks that belong to a "failed" flow run ? and then adding a mutation to set those task to failed ?
k
Let me try the first part first
👍 1
Maybe like this:
Copy code
query {
  task_run (where: {state: {_eq: "Running"},
                    task: {tags: {_contains: "tag"}},
                    flow_run: {state: {_eq: "Failed"}}}) {
    id
    flow_run{
      name
      id
			state
    }
    task{
      name
      tags
    }
  }
}
t
Yes great thanks you, I'll work on a mutation to automatically cancel those task with a cronjob
3 Views
Open in Slack
PreviousNext
Powered by