Hello, not sure if this is a correct channel please let me know if it is not. We are experiencing a problem with placement of ECS (flow-run) tasks in private subnets only (AWS). Currently our setup is: 1. We manage the ECS cluster with terrafom it works just fine. 2. Circle ci deploys a docker image and updates ECR service 3. ECR service brings this updated docker image and runs the ECS Worker there (prefect api authentication is correct and working) 4. We have an

pull based work pool setup

, so docker's entry point script configures in prefect cloud the work pool and registers the flows with the necessary blocks

prefect-ecs

registered, and starts a ECS worker 5. We relay on the IAM role assigned to the container which is running the ECS worker, and not setting up nothing additional for AWS credentials (we relay on the default behavior) If we do not specify any subnets for ECS task, all is going correctly, the task is placed on any available subnet from the VPC, and completes successfully (we don't have a default VPC). The problem comes when we define a custom network configuration and explicitly request to place the task in a private subnet. And we get the following error:

Subnets ['subnet-XXX', 'subnet-YYY'] not found within VPC with ID vpc-IDIDID.Please check that VPC is associated with supplied subnets

The subnets are correct, and the VPC is correct, and the subnets are associated with the VPC. The error message comes from github.dev/PrefectHQ/prefect-aws/blob/6865af76b7f0a0555eefc63fce41721fcf827407/prefect_aws/workers/ecs_worker.py#L1382. Seems like the ECS Worker tries to validate custom network configuration against the VPC subnets, and fails. I suspect the problem might come from the fact that the VPC is not a default one and we need to specify the aws-region. If I request the subnets via aws-cli and query in the same way prefect does, without specifying the region the list is empty. Providing the region the list is correct. There is a way to "force" prefect to use a particular region? I tried it via aws-credentials defining only the region and I get:

Client error '422 Unprocessable Entity' for url '<https://api.prefect.cloud/api/accounts/ACCOUT_ID/workspaces/WORKSPACE/work_pools/>

Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'loc': ['body', 'base_job_template'], 'msg': "Validation failed for field 'aws_credentials.aws_access_key_id'. Failure reason: None is not of type 'string'", 'type': 'value_error'}], 1

I'm wondering if can I preserve my EXECUTION_ROLE with default credentials but overriding the region only? O perhaps you can suggest a better way to solve/debug this problem? Thanks in advance. I can provide any additional info. Did my best to resume the problem short.

Hey @Sergio Shevtsov! I recommend setting the

AWS_DEFAULT_REGION

environment variable for your worker. It would also be great if you could open an issue in

prefect-aws

to allow just the region to be set on an

AwsCredentials

block document. I think that’s something we should support.

Hey @alex thanks for such a quick response. This env var is already present in the worker. But I'm unable to confirm if it is really being used. As far as I can see https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/credentials.py#L99 boto3 session is instantiated and it takes precedence over

env vars

or

~/.aws/config

as per boto3 docs https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials

Yeah, but we should be passing in

None

for

region_name

, which is the default value for

boto3.session

and should discover the region via other methods. Do you have

AWS_REGION

set as an env var on your task definition too?

env vars regarding to the region only

AWS_DEFAULT_REGION

is defined. Should I try adding

AWS_REGION

?

I don’t think so, but it’s worth trying to see if it will unblock you. Either way, we should update

AwsCredentials

to accept only a region if needed.

are you suggesting me to open a request to prefect-aws?

If you could open an issue, that would be great!