Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
pacc-london-2023
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
n
Nace Plesko
09/29/2022, 5:51 PMHi,
Is this a known issue that there are prefect jobs running, but the "runs in progress" is showing 0? We are running into concurrency limits because some tasks are stuck running for ever, but there is no easy way to find those runs, except digging into each flow one by one.
Is there a way I can stop all running flows (even though the dashboard is showing 0 running flows)?
✅ 1
1️⃣ 1
b
Bianca Hoch
09/29/2022, 7:27 PMHi Nace! Assuming this is for 1.0, can you try running the following query in the interactive api?
{
task_run(
where: {
created: {_lt: "2022-09-29 00:00:00", _gte: "2022-09-01 00:00:00"},
state: {_eq: "Running"},
}
) {
id
name
flow_run_id
task {
name
id
}
}
}This should get you all of the task runs stuck in a 'Running' state, as well as the corresponding flow run IDs. After that, you should be able to cancel those flow runs from the UI.
If you're unable to cancel them from the UI, you can run the following query to cancel the task runs
mutation setTaskRunStates($input: [set_task_run_state_input!]!) {
set_task_run_states(input: {states: $input}) {
states {
id
status
message
__typename
}
__typename
}
}When running the query above, the query variables will need to be set in the API like so
{
"input": {
"task_run_id": "task_runid",
"version": 1,
"state": {
"type": "Cancelled",
"message": "marked task run as Cancelled because \"test\""
}
}
}n
Nace Plesko
09/29/2022, 9:19 PM@Bianca Hoch yes we are on prefect v1. Thank you!! Honestly I didn't even know that interactive api exists, this opens a whole new world to me, thanks!!
✨ 1
🚀 1
@Bianca Hoch I apologize for my lack of graphql knowledge, but I can't figure it out why this is not a valid json. It's complaining that it's expecting json, but I can't figure it out how my input is not json. Is there something obvious that I'm doing wrong?
b
Bianca Hoch
09/30/2022, 2:52 PMThe same thing occurred when running the query in the screenshot, however, the task was still able to be set to "Cancelled". Would you mind sharing the error you receive when you run the query?
n
Nace Plesko
09/30/2022, 3:05 PMYou are right, the query actually worked for most of the cases! However I was unable to cancel 4 task runs, they are all a few months old. I'm getting the error that is in the screenshot. Do you know how could I cancel those as well?
b
Bianca Hoch
09/30/2022, 3:06 PMHmmmm...I presume those flows may have been deleted
Can you provide those 4 task run IDs here? We can look into removing them on our end.
n
Nace Plesko
09/30/2022, 3:13 PMThey are still counting towards concurrency limit though. Yes that would be great if you could delete those (or if there was a way I could delete them).
Turns out there are more than 4 in such state, but for some reason just 4 count towards the concurrency limit
"0a6c4764-6cba-4061-8151-f6bbb5fe99a6", "68747f03-c8cd-494f-a991-c017fbfb71ba", "5fcb8a72-9572-49cf-ba05-3a60ed65f212", "16019956-cd19-41fa-94c9-e8c04bc032e9", "5e5e8c0f-eff0-4e2f-b111-de0a9c6f139c" and "73bb1530-3455-4400-af2a-8850d37f2e08". They are all version 2
:gratitude-thank-you: 1
Thank you for deleting those! Now I don't see any task in "running" state anymore, but the concurrency limit still says that there are 4 running tasks
I see 0 now, thank you so much!!
:marvin: 1
b
Bianca Hoch
09/30/2022, 4:41 PMWoohoo!
You got it, Nace.
n
Nace Plesko
09/30/2022, 4:42 PMIt was all you! 🙌
I have another question related to these forever running tasks. We have a shell task that maps to multiple tasks, and we set the timeout like on the screenshot below. I would think that each mapped task would timeout after 3 hours, but just yesterday some of them were stuck for ever and running 13 hours until I manually cancelled them.
db_shell_taskShellTaskRephrasing the same question with the knowledge I just learned. Setting a timeout on
ShellTaskShellTask