Eric Bolton

09/09/2022, 2:42 PM
Running into a problem with a flow whose latest run gets stuck in the "Running" state if a deployment happens to coincide with the run. • We are using ECS to run the
task that registers and runs the flows • Whenever we make an update through our CI pipeline, a new ECS task is spun up and the old task gets deprovisioned • Even when the old task is marked as "Stopped" in AWS (there is no hardware now running the old
), the old Flow Run still says "Running" in the Prefect UI and hangs. Given our concurrency flags, it blocks all scheduled flow runs until it is manually canceled • Questions: ◦ Is there a way for Prefect to detect that the old
is now stopped and automatically force cancel the flow run? ◦ Should we be managing our deployment process differently?

Rob Freedy

09/12/2022, 1:39 PM
Hey @Eric Bolton!! Have you looked into the Zombie Killer process (I assume this is for 1.0)? Might be worth configuring heartbeats so Prefect can detect when the ECS task has been deprovisioned: