Jafar A

04/21/2023, 5:44 PM
Hello everyone, I'm getting this error when I run the scheduled prefect job today with ECS Service. the only difference compared to yesterday's successful run was that I needed to recreate the image and replace the old ECR image (kept the repository). The ECR repository is private but ECS task Auto-assign public IP is
so this error shouldn't happen as per AWS I don't need to set up NAT gateway to allow my subnets to talk to ECR through the internet, or have to setup ECR VPC Endpoints. Also, security group inbound and outbound is set for all ports and there is no secret manager used.
Submission failed. prefect_aws.ecs.TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "<>": dial tcp i/o timeout. Please check your task network configuration. See Less
UPDATE: Seems like adhoc run from UI is working fine but when the same job is scheduled in UI, it errors out with unable to pull from ECR. weird!!