https://prefect.io logo
Join Slack
Powered by
Hi, Is this a known issue that there are prefect ...
# prefect-community
n
Hi, Is this a known issue that there are prefect jobs running, but the "runs in progress" is showing 0? We are running into concurrency limits because some tasks are stuck running for ever, but there is no easy way to find those runs, except digging into each flow one by one. Is there a way I can stop all running flows (even though the dashboard is showing 0 running flows)?
✅ 1
1️⃣ 1
b
Hi Nace! Assuming this is for 1.0, can you try running the following query in the interactive api?
Copy code
{
  task_run(
    where: {
      created: {_lt: "2022-09-29 00:00:00", _gte: "2022-09-01 00:00:00"}, 
      state: {_eq: "Running"},
      }
  ) {
    id
    name
       flow_run_id
    task {
      name
      id
    }
  }
}
This should get you all of the task runs stuck in a 'Running' state, as well as the corresponding flow run IDs. After that, you should be able to cancel those flow runs from the UI.
If you're unable to cancel them from the UI, you can run the following query to cancel the task runs
Copy code
mutation setTaskRunStates($input: [set_task_run_state_input!]!) {
set_task_run_states(input: {states: $input}) {
  states {
  id
  status
  message
  __typename
  }
  __typename
}
}
When running the query above, the query variables will need to be set in the API like so
Copy code
{
  "input": {
    "task_run_id": "task_runid",
    "version": 1,
    "state": {
      "type": "Cancelled",
      "message": "marked task run as Cancelled because \"test\""
    }
  }
}
n
@Bianca Hoch yes we are on prefect v1. Thank you!! Honestly I didn't even know that interactive api exists, this opens a whole new world to me, thanks!!
✨ 1
🚀 1
@Bianca Hoch I apologize for my lack of graphql knowledge, but I can't figure it out why this is not a valid json. It's complaining that it's expecting json, but I can't figure it out how my input is not json. Is there something obvious that I'm doing wrong?
b
The same thing occurred when running the query in the screenshot, however, the task was still able to be set to "Cancelled". Would you mind sharing the error you receive when you run the query?
n
You are right, the query actually worked for most of the cases! However I was unable to cancel 4 task runs, they are all a few months old. I'm getting the error that is in the screenshot. Do you know how could I cancel those as well?
b
Hmmmm...I presume those flows may have been deleted
Can you provide those 4 task run IDs here? We can look into removing them on our end.
n
They are still counting towards concurrency limit though. Yes that would be great if you could delete those (or if there was a way I could delete them). Turns out there are more than 4 in such state, but for some reason just 4 count towards the concurrency limit "0a6c4764-6cba-4061-8151-f6bbb5fe99a6", "68747f03-c8cd-494f-a991-c017fbfb71ba", "5fcb8a72-9572-49cf-ba05-3a60ed65f212", "16019956-cd19-41fa-94c9-e8c04bc032e9", "5e5e8c0f-eff0-4e2f-b111-de0a9c6f139c" and "73bb1530-3455-4400-af2a-8850d37f2e08". They are all version 2
gratitude thank you 1
Thank you for deleting those! Now I don't see any task in "running" state anymore, but the concurrency limit still says that there are 4 running tasks
I see 0 now, thank you so much!!
marvin 1
b
Woohoo!
You got it, Nace.
n
It was all you! 🙌
I have another question related to these forever running tasks. We have a shell task that maps to multiple tasks, and we set the timeout like on the screenshot below. I would think that each mapped task would timeout after 3 hours, but just yesterday some of them were stuck for ever and running 13 hours until I manually cancelled them. 
db_shell_task
on the screenshot is 
ShellTask
Does anything on the screenshot below look obviously wrong? Again, I apologize for my lack of knowledge of Prefect
Rephrasing the same question with the knowledge I just learned. Setting a timeout on 
ShellTask
doesn't seem to do anything. ShellTask doesn't seem to have Duration and the mapped child runs are also not affected by that timeout setting. Is there a way to set timeout on mapped children from 
ShellTask
?
9 Views
Open in Slack
PreviousNext
Powered by