Hello! I have a prefect flow that sends jobs to an aws ecs. this cluster has an autoscaling group th...
a
Hello! I have a prefect flow that sends jobs to an aws ecs. this cluster has an autoscaling group that will spin up ec2 instances for jobs that require more memory than fargate tasks. We have a separate work pool that is configured for ec2. Unfortunately, submitting a job to this work pool does not automatically spin up ec2 instances on the aws side...I have to manually set capacity in the autoscaling group to a desired number some minutes before the requests hit the worker pool and hope everything goes smoothly. Is there a way to correctly configure work pools for ec2 so this is more automatic and less manual? Any help appreciated!
k
I'm not super familiar with EC2 autoscaling groups, but from a look at the AWS docs this is something that's managed through a capacity provider strategy. We recently updated the ECS worker and work pool to allow you to provide your own capacity provider strategy. You should see that field if you create a new ECS work pool
a
would you mind pointing me to the field you're referencing? thanks!
k
Yep! If you're going through the steps to create a new ECS work pool, you'll eventually end up on the Base Job Template page for configuring defaults. You'll find Capacity Provider Strategy just under Launch Type
Note that Launch Type and Capacity Provider strategy is an either/or situation, so Launch Type will be ignored if you provide a strategy