Hi, I setup an ECS Push Work Pool on Prefect Cloud using an Auto Scaling group capacity provider with a MinSize of zero. On attempting to run a simple hello world flow, I saw the following message which leads me to believe perhaps I shouldn't expect the flow to wait for auto scaling to kick in before bailing out. Is that the case or am I missing something?

Flow run could not be submitted to infrastructure: Failed to run ECS task, cluster 'arn:aws:ecs:******' does not appear to have any container instances associated with it. Confirm that you have EC2 container instances available.

I’m wondering if the cold start is too slow for prefect to handle. can you try a min size of 1 and confirm that things work as expected?

Hi @Jamie Zieziula I did try with a min size of 1 but then I ran into a similar error message as in this issue which is now closed so I will need to test again either by installing from main or waiting for the next release. https://github.com/PrefectHQ/prefect-aws/issues/301

Luis, I am seeing a similar problem when setting up a push work pool with GPU instances - do you have any more information or have you resolved the issue?

Hi @Geoffrey Keating and @Jamie Zieziula I have stumbled on a way to autoscale a push pool on ECS after an exchange with AWS support. Unfortunately, the configuration is not supported in the AWS Push Pool itself. I have been pointed out to this article which describes the right approach: https://repost.aws/questions/QUnKdakxvQROuUEjq2UWpa9g/should-ecs-ec2-asgprovider-capacity-provider-be-able-to-scale-up-from-zero-0-1 To use this, you need to replace the "launchStrategy" part of the template with a "capacityProviderStrategy" part similar to the following:

"capacityProviderStrategy": [
        {
          "base": 0,
          "weight": 1,
          "capacityProvider": "OurAutoScalingGroup"
        }
      ]

For instance in my case I put it right after the "taskDefinition" part under "task_run_request". I haven't been able to test though because if I remove the "launchStrategy" part completely, then I get the following error message:

Flow run could not be submitted to infrastructure: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.

So it seems that Fargate is hard coded somewhere in the pool code. If I specify a Launch Strategy of None and keep the "launchStrategy" key in the template then I get the following error:

Flow run could not be submitted to infrastructure: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.

This might have gotten a bit further because I am using an image in our ECR registry. If I let the pool use the default image, then I get the following error as if its defaulting to Fargate again with the None setting.

Flow run could not be submitted to infrastructure: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Tasks using the Fargate launch type do not support GPU resource requirements.

Thank you Luis for the follow up! Perhaps our tasking is best executed via AWS Batch, but unfortunately it seems like using Batch will limit flow observability

I tried AWS Batch some time back and hit the same scaling to zero issue but I didn't know about the "capacityProviderStrategy" settings so I should try again.

I think batch abstracts all of this away? I think I have my autoscaling group configured as posted in the links above

Hmm ok, could you please post back here if that works for you? Crossing fingers!

Not getting anywhere on this one, no. While I don't think this is necessarily prefect related, I would like to know if this is an expected pattern for the ECS Push Worker. I feel like having no capacity at rest for heavy GPU instances is very desirable...

Hi @Luis Arias have you checked out the advanced tab in the work pools configuration? — more info coming, just want to ask before i send over info you already have

Hi @Taylor Curran! How are you? Thanks for your help! 🙌 I have tried to add the relevant JSON in the advanced tab previously, but I get the above errors. I will check out the video, I'm probably doing something wrong. If I'm still stuck I'll post the JSON here.

Thank you for the update Luis, could you create a github issue with this info in the prefect-aws repo? I’ll make sure someone from our integration team takes a look.

Absolutely. I'll do that now and post the link back here. Thanks for your help!

Hi all, I just tried to get an ECS push work pool working and I encountered the same issue. Setting the minimum and desired capacity to 1 on the auto scaling group worked around the issue, but this isn't great for resource-demanding instance types as noted. Just hoping to understand: does Prefect not use my auto scaling group by default and@Luis Arias' proposal to support configuration for capacityProvider will allow it to be plugged in and used?

Hi @James Gatter could you comment on the issue that Luis made explaining that you are also getting this issue? https://github.com/PrefectHQ/prefect-aws/issues/310

Hi @James Gatter, exactly. The current version doesn't support setting the

capacityProviderStrategy

because of some logic requiring a

launchType

when making the task run request. I'm actually working on contributing a PR for this but have had limited capacity to work on it yesterday. Giving it another shot today.