Slackbot
09/24/2025, 2:08 PMMarvin
09/24/2025, 2:08 PMMarvin
09/24/2025, 2:13 PMecs:StopTask
on the task that’s running your flow/subflow.
- The error prefect.exceptions.UnfinishedRun: Run is in CANCELLING state, its result is not available.
is expected during cancellation. It shows up when Prefect (or your code) tries to fetch the result of a run that’s not going to finish because it’s being cancelled.
- If your ECS tasks/containers keep running, it usually means the cancellation signal did not reach the worker or the worker could not stop the ECS task (permissions, offline worker, or an old worker/collection version).
Things to check first
1) Worker online and logs
- Was your ECS worker online at the moment you clicked Cancel?
- Check the worker logs around the cancel time — you should see something like “received cancellation” and “stopping ECS task …”. If you don’t see this, the cancel didn’t reach the worker.
2) IAM permissions for the worker’s role
- The role used by the ECS worker must be able to call ecs:StopTask
on your cluster. A minimal policy excerpt (adjust to your environment/resources):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:RunTask",
"ecs:StopTask",
"ecs:DescribeTasks",
"ecs:DescribeTaskDefinition"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:CreateLogGroup"
],
"Resource": "*"
}
]
}
AWS StopTask reference: AWS ECS StopTask API
3) Versions
- Make sure you’re on current Prefect and prefect-aws so the ECS worker has the latest cancellation behavior:
- prefect
(3.x)
- prefect-aws
(latest)
- If you recently upgraded Prefect but your worker task definition still pulls an older image/environment, cancellation features may not match.
About subflows on ECS
- Each subflow is its own flow run. If the parent is cancelled, Prefect marks the parent CANCELLING and will also try to cancel running subflow runs — but those subflow runs must be picked up by a worker that can stop their ECS tasks too. If any worker is offline or lacks ecs:StopTask
, those ECS tasks keep running.
- If you programmatically retrieve subflow results (e.g., await state.result()
from a client call), you may see UnfinishedRun
when cancellation begins. That’s expected; guard result retrieval with a state check if needed.
Manual cleanup for now
- If some tasks are orphaned, you can stop them directly:
aws ecs list-tasks --cluster <cluster> --desired-status RUNNING --family <family>
aws ecs stop-task --cluster <cluster> --task <task-arn> --reason "Cancelled by Prefect"
Good practices to avoid orphans
- Keep workers online during cancellations so they can act on stop requests.
- Ensure the worker’s IAM role includes ecs:StopTask
for your cluster and tasks.
- Keep prefect
and prefect-aws
up to date in the worker environment.
- If you launch subflows via run_deployment
, make sure those runs land on a work pool whose workers also have permissions to stop tasks.
References
- Prefect cancellation overview: Cancel workflows
- Prefect + AWS ECS guide: ECS integration guide
If you can share:
- Prefect and prefect-aws versions
- How you launch the subflows (awaited subflows vs run_deployment)
- Whether the same ECS work pool/worker runs those subflows
- A snippet from your worker logs around the cancel time
…I can pinpoint why the stop requests didn’t remove those ECS tasks and suggest the exact fix.