Hello all, in our development environment currently we are using a ECS Agent and have a flow of flows, that is running a large-ish batch of small pipelines concurrently, however we are getting a mix of Throttle Errors (RegisterTaskDefinition, DeregisterTaskDefinition). My agent has the AWS_RETRY_MODE and AWS_MAX_ATTEMPTS env variables as 'adaptive' and '25' respectively, but I'm still getting this error, I'm up for splitting this up into smaller calls, but I'm curious if this is a situation anyone has come across and how it was solved. How did you implement the wait/backoff logic on the flow, and is there a way to iterate with waits to implement less calls to the aws api?

Hi @Michael Moscater, looks like you followed this already. I think if you can have wait logic on the Flow, I think you can explore flow run concurrency limiting to spread out the API calls? I have seen some people use AWS Batch instead for the big flow triggering many small jobs. Of course, the Batch jobs are not Prefect Flows anymore so you don’t have visibility in the UI. But for stuff like 300+ subflows, I am not sure there is a more efficient way in ECS

Yeah, AWS batch is an option, but like you said it eliminates the monitoring and visibility in a single place. Concurrency limits looks helpful, but unfortunately.. We are not 'allowed' to use Prefect Cloud, to my chagrin.

Ah ok. Maybe you can try splitting the big flow into 2-3 less big flows if possible like you suggested? These would count against the API limits differently I believe

Thanks Kevin, I'll give it a try and let you know what I come up with for future ref

The throttling you see involves the issue with registering flows as ECS tasks, but you can force Prefect to use a preregistered task definition by providing a specific task definition ARN on your ECSRun. This way, Prefect will not attempt to register a new task definition. The downside is that you need to sort of manually handle new flow versions by reregistering new ECS task definitions before registering a new flow version

@Anna Geller - Currently I have a task definition stored in s3 that my flows are referencing. Am i misunderstanding what you are saying?

What I meant was using a preregistered task definitions as in this example https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/s3_ecs_run_preregistered_task_definition.py

ah ok, So I'm using task defintion path, and its using that path to register the task definition at run time on each flow. If i use the task_definition_arn kwarg, then it will use the preregistered, and shouldn't throttle registration, so in theory.. 40 tasks should spawn using that same definition without throttling. Am i tracking?

That's exactly the case! Basically, using a preregistered task definition will avoid the throttling issues that happen during the registration of new ECS tasks

ok, I'll give this a try, and I'm using a custom file to run to register my flows with different run configs based on its directory from our CI/CD tool. It looks like i might be able to handle the registration of new definitions because the cli response gives the arn (accoriding to the aws docs), so i can add this to my build when using different definitions and insert the new arn in the ECSRun args. As always, thank you so much for the assistance.

Great work! And thanks for updating us on that! 🙌