Does Prefect 1.1 support retries on

Flows

? Specifically, I’m trying to workaround some delayed scaling issues with ECS using EC2 instances (not ECS with Fargate tasks) Often, this failure is reported back to Prefect like the following error until Capacity Provider scaling has caught up again:

FAIL signal raised: FAIL('a4f09101-0577-41ce-b8b0-31b84f26d855 finished in state <Failed: "Failed to start task for flow run a4f09101-0577-41ce-b8b0-31b84f26d855. Failures: [{\'arn\': \'arn:aws:ecs:us-east-1:<redacted>:container-instance/a8bc98b7c6864874bc6d1138f758e8ea\', \'reason\': \'RESOURCE:CPU\'}]">')

I’m using the following calls to launch the sub-flows (as part of a larger script):

flow_a = create_flow_run(flow_name="A", project_name="myles")
wait_for_flow_a = wait_for_flow_run(flow_a, raise_final_state=True, stream_logs=True)

there are no flow-level retries - to retry a flow run, you would need to create a new flow run - this is something you could do in a state handler or using Automations Are you on Prefect Cloud?

Yep, running on Prefect Cloud.

Sure, understandable! To sort of retry a flow run you could set an SLA using Automations to ensure that e.g. if your flow run failed to move to a Running state after e.g. 10 minutes (e.g. due to some ECS provisioning issue), you could start a new flow run but not sure if this is the right approach here, let me check sth

The behavior I’m seeing is that the Parent Flow starts (and we can usually manage making capacity for this ahead of time via Scheduled Scaling Policy in ASGs) but a Child Flow fails to start due to insufficient resources available. Retrying to launch the Child Flow is what I’m hoping to do, while the Parent Flow keeps running.

I checked the logs of your flow run and the Automations won't help you here because the flow run fails immediately because Prefect cannot deploy the flow run to ECS due to not enough CPU resources on your EC2 instance: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/ecs/agent.py#L349 I hate to say that because it's a bit lazy answer but the best option going forward here is to ensure that either: • you ensure you have enough capacity on your self-managed EC2 data plane e.g. by implementing a bit more aggressive scaling policy • move to Fargate and not have to worry about infrastructure management, but accept the latency of serverless

Yup, this makes sense and gives me something to play with! I’m trying to avoid moving to Fargate for launching due to existing security vendor deployments which we would also need to update. We might at some point, just can’t do it yet.

thank you so much for the detailed description, great write-up!

BTW, I’m currently testing out the latest code from the repo to confirm if PR 5411 resolves this. Overall it does seem to be! I’m running into a related issue to that test, but it’s not the fault of Prefect Server or the ECS Agent.