<@ULVA73B9P> i cancelled my prefect flow from ui, ...
# ask-marvin
r
@Marvin i cancelled my prefect flow from ui, this flow runs a set of subflows that run on ecs tasks. now i see that task containers are not getting removed from cluster and are stuck with error`prefect.exceptions.UnfinishedRun: Run is in CANCELLING state, its result is not available.
m
thought for 535.8 seconds
Thanks for the details — I can help you sort this out. What you’re seeing breaks down into two related pieces: 1) Why you see UnfinishedRun - When a run is in CANCELLING (or CANCELLED), Prefect will not return a result for it. If your parent flow tries to await or access the result of a child run at that point, you’ll see:
prefect.exceptions.UnfinishedRun: Run is in CANCELLING state, its result is not available.
- This is expected behavior during cancellation. It doesn’t, by itself, stop any external infrastructure (like ECS tasks). 2) Why ECS tasks/containers are still running - Cancelling a parent flow from the UI does not automatically stop separately-deployed child flow runs that were launched with
run_deployment
. They are independent runs executed by workers (on ECS in your case), so the parent’s cancellation will not “cascade” and stop those ECS tasks automatically. - To stop the underlying containers, the ECS worker must explicitly stop the ECS tasks (AWS StopTask). If the worker is offline, misconfigured, or lacks IAM permission to call
ecs:StopTask
, tasks can continue running even though the parent was cancelled. Immediate steps to clean up - In Prefect UI: open the parent flow run’s page -> Related runs (or filter by parent flow run ID) -> cancel the child flow runs that are still RUNNING. - Or via CLI:
Copy code
prefect flow-run ls --state RUNNING
  prefect flow-run cancel <child_flow_run_id>
You can cancel multiple child runs this way. - In AWS (to force stop immediately): stop the tasks directly in ECS (Console or CLI
aws ecs stop-task --cluster <cluster> --task <taskArn> --reason "Prefect cancellation"
). Prevention and best practices - Ensure the ECS worker can stop tasks on cancel: - Keep your worker online; it needs to receive the cancellation and act on it. - Verify the worker’s IAM role has at least
ecs:StopTask
(and the necessary
ecs:ListTasks
,
ecs:DescribeTasks
) for your cluster. Without
StopTask
, the worker can’t cleanly stop containers when runs are cancelled. - Keep Prefect and the AWS integration up to date to get the latest worker behavior. - ECS Guide: Prefect AWS ECS guide - Be explicit about cancelling child runs: - Because
run_deployment
starts independent flow runs, cancellation of the parent does not guarantee cancellation of the children. If you need “cascade cancel,” keep track of child flow run IDs and cancel them when the parent is cancelled (either manually via UI/CLI or with your own orchestration logic). - Use time limits where appropriate: - Add timeouts on flows that should exit if they exceed a duration. This keeps long-lived ECS tasks from lingering if something upstream aborts. - Expect UnfinishedRun during cancellation: - If your parent code tries to fetch a result while cancelling, handle that case (e.g., don’t await results once a cancel is requested, or catch and ignore
UnfinishedRun
during shutdown). A couple of questions to make sure we guide you precisely: - Which Prefect version are you on? (Assuming 3.x unless you say otherwise.) - How are you launching the children —
run_deployment(..., as_subflow=True)
(default) or otherwise? - Are the child runs showing as RUNNING in the Prefect UI after you cancel the parent? - Is your ECS worker still online, and does it have permissions to call
ecs:StopTask
? Docs you may find helpful: - Cancel workflows in Prefect 3 - ECS worker setup and permissions If you share the above details (version, how you launch subflows, worker status/permissions), I can help pinpoint why the tasks weren’t stopped and suggest exact changes.