hey everyone! We’re on Prefect Cloud with ECS workers (Prefect 3.6.5, prefect-aws 0.7.1). ECS worke...
j
hey everyone! We’re on Prefect Cloud with ECS workers (Prefect 3.6.5, prefect-aws 0.7.1). ECS worker runs on prefect-ecs-cpu-cluster (Fargate). infrastructure_pid is set on runs (e.g., prefect-ecs-cpu-cluster:arnawsecsus-east-2:.../task/...). IAM for the worker task role includes ecs:StopTask. Region/queue are set (AWS_REGION/AWS_DEFAULT_REGION=us-east-2, PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_NAME=prefect-ecs-events). SQS/EventBridge queues are empty (no backlog). Problem: Cancelling a flow run leaves it in “Cancelling”; the ECS task keeps running. Worker logs show no StopTask or cancel handling. Even with infrastructure_pid present, no stop is issued. Forcing state via UI/CLI doesn’t stop the task. Any known issues or fixes to get Cancel to issue ECS StopTask? Should we tweak anything on the worker side, or is there a known bug in the cancel path?
a
Hey @João Pedro Boufleur! Are you able to share any logs from a hung ECS task around the time that you cancelled the task?
j
thanks for the quick response Alex! Going to DM you the event logs for the flow that I cancelled mid run.
I can see that there's something getting cancelled:
but it just seems to be ignored:
so it basically keeps on running the flow with the status set to
cancelling
the flow got wrapped up, the ECS task is already stopped, but the flow is stuck in
cancelling
image.png
a
Have you customized the start up command in the container? By default it is
prefect flow-run execute
j
@alex sorry for late response, but thanks a billion, that was it! it was using
prefect.engine
instead have a blessed end of week!