This message was deleted.
# marvin-ai
s
This message was deleted.
m
thought for 286.4 seconds
Thanks for the details — a few things are going on here. What you’re seeing - When you hit Cancel in the UI, Prefect moves the run into CANCELLING and asks the worker to stop the underlying infrastructure. For ECS, that means calling
ecs:StopTask
on the task that’s running your flow/subflow. - The error
prefect.exceptions.UnfinishedRun: Run is in CANCELLING state, its result is not available.
is expected during cancellation. It shows up when Prefect (or your code) tries to fetch the result of a run that’s not going to finish because it’s being cancelled. - If your ECS tasks/containers keep running, it usually means the cancellation signal did not reach the worker or the worker could not stop the ECS task (permissions, offline worker, or an old worker/collection version). Things to check first 1) Worker online and logs - Was your ECS worker online at the moment you clicked Cancel? - Check the worker logs around the cancel time — you should see something like “received cancellation” and “stopping ECS task …”. If you don’t see this, the cancel didn’t reach the worker. 2) IAM permissions for the worker’s role - The role used by the ECS worker must be able to call
ecs:StopTask
on your cluster. A minimal policy excerpt (adjust to your environment/resources):
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:RunTask",
        "ecs:StopTask",
        "ecs:DescribeTasks",
        "ecs:DescribeTaskDefinition"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:CreateLogGroup"
      ],
      "Resource": "*"
    }
  ]
}
AWS StopTask reference: AWS ECS StopTask API 3) Versions - Make sure you’re on current Prefect and prefect-aws so the ECS worker has the latest cancellation behavior: -
prefect
(3.x) -
prefect-aws
(latest) - If you recently upgraded Prefect but your worker task definition still pulls an older image/environment, cancellation features may not match. About subflows on ECS - Each subflow is its own flow run. If the parent is cancelled, Prefect marks the parent CANCELLING and will also try to cancel running subflow runs — but those subflow runs must be picked up by a worker that can stop their ECS tasks too. If any worker is offline or lacks
ecs:StopTask
, those ECS tasks keep running. - If you programmatically retrieve subflow results (e.g.,
await state.result()
from a client call), you may see
UnfinishedRun
when cancellation begins. That’s expected; guard result retrieval with a state check if needed. Manual cleanup for now - If some tasks are orphaned, you can stop them directly:
Copy code
aws ecs list-tasks --cluster <cluster> --desired-status RUNNING --family <family>
aws ecs stop-task --cluster <cluster> --task <task-arn> --reason "Cancelled by Prefect"
Good practices to avoid orphans - Keep workers online during cancellations so they can act on stop requests. - Ensure the worker’s IAM role includes
ecs:StopTask
for your cluster and tasks. - Keep
prefect
and
prefect-aws
up to date in the worker environment. - If you launch subflows via
run_deployment
, make sure those runs land on a work pool whose workers also have permissions to stop tasks. References - Prefect cancellation overview: Cancel workflows - Prefect + AWS ECS guide: ECS integration guide If you can share: - Prefect and prefect-aws versions - How you launch the subflows (awaited subflows vs run_deployment) - Whether the same ECS work pool/worker runs those subflows - A snippet from your worker logs around the cancel time …I can pinpoint why the stop requests didn’t remove those ECS tasks and suggest the exact fix.