https://prefect.io logo
Title
s

scott

01/04/2023, 11:26 PM
I’ve canceled a flow run, but it’s stuck in state “Cancelling” for at least 10 minutes. I see this issue https://github.com/PrefectHQ/prefect/issues/4490 not sure if it’s related. Any ideas?
m

Mason Menges

01/05/2023, 5:03 PM
Hey scott is this in prefect 1 or prefect 2, also where is the flow run executing, i.e. k8s, docker, ecs, etc.
s

scott

01/05/2023, 5:04 PM
Prefect 2. Flow is running on a GCP VM
And running within docker containers
I’m guessing the issue is a problem with the prefect agent running on the VM not being able to shut down the docker container?
m

Mason Menges

01/05/2023, 5:13 PM
Can you see any of the agent logs in the vm? it might provide some more insight as to what it might be getting stuck on
s

scott

01/05/2023, 5:20 PM
I can’t remember which flow run it was now, but remembering from yesterday I didn’t see any logs prefect related that seemed suspicious. I’ll ping again if it happens again
m

Mason Menges

01/05/2023, 6:24 PM
Sounds good, generally speaking this can sometimes occur if the infrastructure running the flow crashes we have I'd be curious to see if we can track down why the flow got stuck canceling in that context though and why it wasn't reported as a crash in the UI
s

scott

01/05/2023, 6:25 PM
Okay, next time I’ll collect as much info as possible and report back
:gratitude-thank-you: 1