Hello we are running ECS Agent with `prefecthq prefect 1 2 1

Question

Hello we are running ECS Agent with `prefecthq prefect 1 2 1 python3 9` image and we are using Prefect Cloud All seems to be fine with the Agent we can see it in the UI and also in Cloudwatch logs whenever we trigger a flow run it says `Deploying flow run ` and `Completed deployment of flow run` But no Flows are getting started it gets stuck on Submitted every time it worked before but we changed image for agent from `prefecthq prefect 0 14 13 python3 8` and also roles in AWS which have the same permissions as original had There are no errors and nothing in logs I m not really sure what to do Thanks for any help Run config looks like this `RUN CONFIG = ECSRun ` ` labels = prod ` ` task role arn = arn aws iam XXX role prefectTaskRole ` ` execution role arn = arn aws iam XXX role prefectTaskExecutionRole ` ` task definition arn = prefect task 4 ` ` run task kwargs =dict ` ` cluster = XXX ` ` ` ` `

Jan Bršťák · Answer

Role permissions are following `prefectTaskExecutionRole` ``` Version 2012 10 17 Statement Effect Allow Action ecr GetAuthorizationToken ecr BatchCheckLayerAvailability ecr GetDownloadUrlForLayer ecr BatchGetImage logs CreateLogStream logs PutLogEvents Resource ``` `prefectTaskRole` ``` Version 2012 10 17 Statement Effect Allow Action ec2 AuthorizeSecurityGroupIngress ec2 CreateSecurityGroup ec2 CreateTags ec2 DescribeNetworkInterfaces ec2 DescribeSecurityGroups ec2 DescribeSubnets ec2 DescribeVpcs ec2 DeleteSecurityGroup ecs CreateCluster ecs DeleteCluster ecs DeregisterTaskDefinition ecs DescribeClusters ecs DescribeTaskDefinition ecs DescribeTasks ecs ListAccountSettings ecs ListClusters ecs ListTaskDefinitions ecs RegisterTaskDefinition ecs RunTask ecs StopTask iam PassRole logs CreateLogStream logs CreateLogGroup logs PutLogEvents logs DescribeLogGroups logs GetLogEvents Resource ``` so PassRole include here an additional S3 bucket policy ``` Version 2012 10 17 Statement Effect Allow Action s3 Resource arn aws s3 janb prefect datasets ```

Anna Geller · Answer

Your setup looks good I wonder whether it just takes time for AWS to provision the actual resources do you see ECS tasks being spun up in the ECS console when you trigger a flow run You said something interesting in case you are migrating your set up from 0 14 13 to 1 2 1 you may need to change e g this new version no longer uses auth tokens and uses API keys instead

Jan Bršťák · Answer

Thank you for the reply ECS tasks are not even being started no log groups in Cloudwatch nothing really weird I ve let it run for a while several times and it just timed out 3 retries by lazarus and end I will take a look at these things that changed between those versions maybe there is something we need to do

Anna Geller · Answer

yes I think it may be that the flow run container cannot start due to authentication issues to Prefect Cloud because the Auth method changed from tokens to API keys keep us posted how it goes thumbsup

Jan Bršťák · Answer

I ve redeployed Agent created new API key new ECS Task completely new deployment And tasks are now starting slightly smiling face Thank you for the help

Anna Geller · Answer

wow amazing work so glad you figured it out well done clap