hey guys, I am running prefect agent ecs start in terminal and getting this error. What is the syntax it wants me to use?

ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`

Hey @eric lumsden, can you give me more details about your run configuration?

sure, I set up a new user under IAM in aws, the I used aws configure, set values for key ID, secret key, default region, and then left default output format as none. Ran the prefect agent ecs start... not sure what/where else it wants something specified

Are you using the default VPC?

I'm not sure. How would I check?

I guess this will give instructors are to check your VPCs in the UI. Do you have UI access? Just check if you only have the default VPCs or if you have additional ones.

I have UI access. I just browsed into "Your VPCs" and it is empty. What should my next step be?

Ok. Just making sure, the aws config is in the right region?

yes it is all set to us-east

When you go to ECS, do you see the default cluster?

oh boy ECS doesn't look like it has been set up at all. There are no clusters. Do y'all have a doc?

Let me look around

oh man I was actually trying this guide but then I saw Fargate was deprecated for y'all

let me pull someone who can help better

ok, sorry. I am very new to aws config

No worries. I pinged someone to help 🙂

thanks for your help. I'll sit on my hands so I don't mess up my config further

Can you try doing something like this?

RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                    run_task_kwargs={
                        "networkConfiguration": {
                            "awsvpcConfiguration": {
                                "assignPublicIp": "ENABLED",
                                "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                "securityGroups": ["sg-123456"],
                            }
                        }
                    },
)

I did not add a new VPC

Inside Python code and then pass this object to your

Flow()

call.

RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                    run_task_kwargs={
                        "networkConfiguration": {
                            "awsvpcConfiguration": {
                                "assignPublicIp": "ENABLED",
                                "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                "securityGroups": ["sg-123456"],
                            }
                        }
                    },
)

with Flow(name="ecs-test", storage=STORAGE, run_config=RUN_CONFIG, executor=EXECUTOR) as flow:
  print_log()
flow.run()

am I missing a dependency?

is there a chance you named your file

flow.py

?

yeah I was about to type up that I shoved it all into a script and was getting an exception about STORAGE being undefined

The

STORAGE

is to point to where your code lives. For ECS it might point to the Docker image location. Something like:

STORAGE = Docker(
    registry_url="<http://1234.dkr.ecr.us-east-2.amazonaws.com|1234.dkr.ecr.us-east-2.amazonaws.com>",
    image_name="ecs-test",
)

the original snippet was me trying to run "prefect agent ecs start"

oh yeah oof! let me give you a YAML file instead that gives the VPC info that we can attach to the command line command. will reply a bit later

thank you for your patience

Regardless though, I think this ECS documentation is definitely something we need to improve on. I will note that.

ok I could use a little help finding the value for networkConfiguration. Getting this error: ValueError: Failed to infer default networkConfiguration, please explicitly configure using

--run-task-kwargs

I'm guessing I need to set the networkMode to an existing vpc. Unfortunately my environment shifted it's hosting region overnight so I am pointed somewhere with no VPCs. How do I point to the new region and select a VPC?

The region is on your CLI config. You start out with a YAML like this:

networkMode: awsvpc
cpu: 256
memory: 512
containerDefinitions:
  - name: flow
requiresCompatibilities:
  - FARGATE
executionRoleArn: arn:aws:iam::xxxxxxxxxx:role/tf-iam-role-prefect-ecs-tasks <--- change to whatever you need

ok I repointed my CLI to the new region, recreated the S3 bucket in the new region, grabbed one of the VPC ids from the new region and put it in the networkMode parameter in YAML... still get the same error complaining about my network configuration

yep i pinged someone on the team to give me a more fleshed out example. will get back to you

ok, thank you. I'm probably missing something basic

ok sorry i mislead you from the python snippet yesterday. just to recap and start fresh, the problem is we need to give the VPC details

ok... new error. I went for the subnet arn on the subnet parameter for the YAML file, hope that was correct over the subnet ID. I went the security group ID for lack of having another option. Now when I run prefect agent ecs start --run-task-kwargs s3locationwithyaml, I get prefect.utilities.exceptions.AuthorizationError: No agent API token provided. I went in to my cloud prefect account and generated an API token. Where do I specify its use?

The orchestration tutorial has snippets how to configure: https://docs.prefect.io/orchestration/tutorial/overview.html#install-prefect

ok using that I was able to authenticate with my generated API token to prefect cloud. I was able to kickoff a flow within prefect cloud that does not do anything with our AWS. When I go back and run

prefect agent ecs start --run-task-kwargs <s3://bucketname>

Yeah I’m looking into it.

I was able to get a basic flow running on the cloud from my local agent. Basically, a hello world. I set my AWS creds in my local aws CLI and then called the prefect agent to start ec2 on my local. That's basically all the tutorial does, so I thought this was sufficient. What do I need to change so the prefect agent will know to use my aws credentials?

Just to clarify, you mean start ECS from the local? or did you mean an EC2 instance?

At this point, I can't honestly say I am sure. Whichever will allow me to use prefect cloud to monitor and orchestrate data flows within my AWS environment

I feel there are too many moving parts to ECS (we still have yet to get to Docker images and Execution/Task Role Permissions) that we might not be productive step-by-step debugging like this. What I’ll do is write an end-to-end tutorial around this (some time in the next week) and ping you when I complete it and that’ll be a much better starting point for us to get you up and running with ECS. Does that work?

that'd be great, thank you Kevin

Thank you for your patience as well 🙂