https://prefect.io logo
Title
e

eric lumsden

04/05/2021, 5:30 PM
hey guys, I am running prefect agent ecs start in terminal and getting this error. What is the syntax it wants me to use?
ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
k

Kevin Kho

04/05/2021, 5:35 PM
Hey @eric lumsden, can you give me more details about your run configuration?
e

eric lumsden

04/05/2021, 5:43 PM
sure, I set up a new user under IAM in aws, the I used aws configure, set values for key ID, secret key, default region, and then left default output format as none. Ran the prefect agent ecs start... not sure what/where else it wants something specified
k

Kevin Kho

04/05/2021, 5:45 PM
Are you using the default VPC?
e

eric lumsden

04/05/2021, 5:47 PM
I'm not sure. How would I check?
k

Kevin Kho

04/05/2021, 5:50 PM
I guess this will give instructors are to check your VPCs in the UI. Do you have UI access? Just check if you only have the default VPCs or if you have additional ones.
e

eric lumsden

04/05/2021, 5:53 PM
I have UI access. I just browsed into "Your VPCs" and it is empty. What should my next step be?
wait no I had to select the region
we have additional VPCs
k

Kevin Kho

04/05/2021, 5:54 PM
Ok. Just making sure, the aws config is in the right region?
e

eric lumsden

04/05/2021, 5:56 PM
yes it is all set to us-east
k

Kevin Kho

04/05/2021, 5:57 PM
When you go to ECS, do you see the default cluster?
Or any cluster provisioned?
e

eric lumsden

04/05/2021, 6:01 PM
oh boy ECS doesn't look like it has been set up at all. There are no clusters. Do y'all have a doc?
k

Kevin Kho

04/05/2021, 6:02 PM
Let me look around
aws ecs create-cluster
try this?
or really just create the default in the UI
e

eric lumsden

04/05/2021, 6:11 PM
oh man I was actually trying this guide but then I saw Fargate was deprecated for y'all
ok cluster is created after the aws create cluster but same error on prefect agent start. Is there work to do with configuring the VPC still?
so that error is "ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`"
k

Kevin Kho

04/05/2021, 6:15 PM
let me pull someone who can help better
e

eric lumsden

04/05/2021, 6:15 PM
ok, sorry. I am very new to aws config
k

Kevin Kho

04/05/2021, 6:17 PM
No worries. I pinged someone to help 🙂
Btw, the fargate agent is deprecated but you can still use the ECS agent to run ECS tasks on Fargate
e

eric lumsden

04/05/2021, 6:25 PM
thanks for your help. I'll sit on my hands so I don't mess up my config further
k

Kevin Kho

04/05/2021, 6:27 PM
Can you try doing something like this?
RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                    run_task_kwargs={
                        "networkConfiguration": {
                            "awsvpcConfiguration": {
                                "assignPublicIp": "ENABLED",
                                "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                "securityGroups": ["sg-123456"],
                            }
                        }
                    },
)
then pass it into your flow with
run_config=RUN_CONFIG
. get the VPC configuration from the default subnets
I assume you didn’t add a VPC when you created the cluster right?
e

eric lumsden

04/05/2021, 6:30 PM
I did not add a new VPC
I'm sorry I don't appear to be getting this. Where am I supposed to run the RUN_CONFIG declaration?
k

Kevin Kho

04/05/2021, 6:44 PM
Inside Python code and then pass this object to your
Flow()
call.
RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                    run_task_kwargs={
                        "networkConfiguration": {
                            "awsvpcConfiguration": {
                                "assignPublicIp": "ENABLED",
                                "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                "securityGroups": ["sg-123456"],
                            }
                        }
                    },
)

with Flow(name="ecs-test", storage=STORAGE, run_config=RUN_CONFIG, executor=EXECUTOR) as flow:
  print_log()
flow.run()
Let me know if you don’t know where do find the subnet and security group.
e

eric lumsden

04/05/2021, 7:42 PM
am I missing a dependency?
k

Kevin Kho

04/05/2021, 7:50 PM
is there a chance you named your file
flow.py
?
I think this might be an import conflict. Or do you have a folder named
prefect
inside?
Actually sorry I just noticed this is interactive mode. I don’t think you’re missing any modules. This error might be because the function inside your flow is undefined
Replace
print_log
with
print("1")
to test this block. You don’t have the STORAGE and EXECUTOR defined though so this will error. Do you have the original snippet that caused the error? Maybe we can take a look at that
e

eric lumsden

04/05/2021, 8:02 PM
yeah I was about to type up that I shoved it all into a script and was getting an exception about STORAGE being undefined
k

Kevin Kho

04/05/2021, 8:04 PM
The
STORAGE
is to point to where your code lives. For ECS it might point to the Docker image location. Something like:
STORAGE = Docker(
    registry_url="<http://1234.dkr.ecr.us-east-2.amazonaws.com|1234.dkr.ecr.us-east-2.amazonaws.com>",
    image_name="ecs-test",
)
e

eric lumsden

04/05/2021, 8:04 PM
the original snippet was me trying to run "prefect agent ecs start"
and I suppose I have to create the docker image location on AWS?
k

Kevin Kho

04/05/2021, 8:05 PM
oh yeah oof! let me give you a YAML file instead that gives the VPC info that we can attach to the command line command. will reply a bit later
e

eric lumsden

04/05/2021, 8:06 PM
thank you for your patience
k

Kevin Kho

04/05/2021, 8:06 PM
Regardless though, I think this ECS documentation is definitely something we need to improve on. I will note that.
and no problem!
This thread shows how to create a YAML file, add the taskRoleArn and the execurionRoleArn.
And then call it like this: https://docs.prefect.io/orchestration/agents/ecs.html#custom-runtime-options . Ultimately, there have been a couple of people who have run into this same issue. I have read that EKS was easier to setup so that may be an option. We have a couple of PR’s around this to make the experience easier and a minimum working example is definitely something we have in our minds
e

eric lumsden

04/06/2021, 3:38 PM
ok I could use a little help finding the value for networkConfiguration. Getting this error: ValueError: Failed to infer default networkConfiguration, please explicitly configure using
--run-task-kwargs
I'm guessing I need to set the networkMode to an existing vpc. Unfortunately my environment shifted it's hosting region overnight so I am pointed somewhere with no VPCs. How do I point to the new region and select a VPC?
k

Kevin Kho

04/06/2021, 3:41 PM
The region is on your CLI config. You start out with a YAML like this:
networkMode: awsvpc
cpu: 256
memory: 512
containerDefinitions:
  - name: flow
requiresCompatibilities:
  - FARGATE
executionRoleArn: arn:aws:iam::xxxxxxxxxx:role/tf-iam-role-prefect-ecs-tasks <--- change to whatever you need
and then you add VPC info (looking for an example)
e

eric lumsden

04/06/2021, 3:58 PM
ok I repointed my CLI to the new region, recreated the S3 bucket in the new region, grabbed one of the VPC ids from the new region and put it in the networkMode parameter in YAML... still get the same error complaining about my network configuration
k

Kevin Kho

04/06/2021, 4:00 PM
yep i pinged someone on the team to give me a more fleshed out example. will get back to you
e

eric lumsden

04/06/2021, 4:00 PM
ok, thank you. I'm probably missing something basic
it looks like networkMode should be awsvpc, I may be missing a parameter to specify the actual vpc
k

Kevin Kho

04/06/2021, 4:21 PM
ok sorry i mislead you from the python snippet yesterday. just to recap and start fresh, the problem is we need to give the VPC details
create a file 
run_task.yaml
 with this:
networkConfiguration:
  awsvpcConfiguration:
    subnets:
      - 'subnet-1234'
    securityGroups:
      - 'sg-1234'
    assignPublicIp: 'ENABLED'
run 
prefect agent ecs start --run-task-kwargs /ecs/run_task.yaml
ideally to simplify, subnet and security groups should be the default ones from your AWS account
ignore the previous YAML. that is when we start to get into tasks
e

eric lumsden

04/06/2021, 4:37 PM
ok... new error. I went for the subnet arn on the subnet parameter for the YAML file, hope that was correct over the subnet ID. I went the security group ID for lack of having another option. Now when I run prefect agent ecs start --run-task-kwargs s3locationwithyaml, I get prefect.utilities.exceptions.AuthorizationError: No agent API token provided. I went in to my cloud prefect account and generated an API token. Where do I specify its use?
k

Kevin Kho

04/06/2021, 4:59 PM
The orchestration tutorial has snippets how to configure: https://docs.prefect.io/orchestration/tutorial/overview.html#install-prefect
I also think you can revisit that tutorial you had now that we fixed the networkConfig 🙂
e

eric lumsden

04/06/2021, 5:12 PM
ok using that I was able to authenticate with my generated API token to prefect cloud. I was able to kickoff a flow within prefect cloud that does not do anything with our AWS. When I go back and run
prefect agent ecs start --run-task-kwargs <s3://bucketname>
I still get the same error
prefect.utilities.exceptions.AuthorizationError: No agent API token provided.
we've been talking a while so maybe let me restate what I want to do so we're on track: trying to spin up ECS on AWS and connect it to prefect cloud so that I can kick off and manage flows in prefect cloud that will run on AWS
so to that end we made a configuration file in YAML for ECS and have it on AWS, but I think I am missing the part where there is actual connection between the two cloud environments. Presumably there should be a listener set up on AWS waiting for inputs from Prefect Cloud that will then use the ECS to run flows. Where is the doc for that listener service/bridge between the two cloud environmnets
k

Kevin Kho

04/06/2021, 5:21 PM
Yeah I’m looking into it.
So I believe the issue here is that the agent needs the RUNNER token. Just to make sure, you’re doing the
prefect agent ecs start
on your local machine and you want the local machine to kick off ECS jobs right?
But you were able to get the basic flow running on your local agent from cloud?
Or are you using an EC2 instance to kick off the ECS job like the tutorial?
The agent is also hold the AWS credentials (like seen in the tutorial)
e

eric lumsden

04/06/2021, 5:53 PM
I was able to get a basic flow running on the cloud from my local agent. Basically, a hello world. I set my AWS creds in my local aws CLI and then called the prefect agent to start ec2 on my local. That's basically all the tutorial does, so I thought this was sufficient. What do I need to change so the prefect agent will know to use my aws credentials?
k

Kevin Kho

04/06/2021, 5:55 PM
Just to clarify, you mean start ECS from the local? or did you mean an EC2 instance?
e

eric lumsden

04/06/2021, 6:02 PM
At this point, I can't honestly say I am sure. Whichever will allow me to use prefect cloud to monitor and orchestrate data flows within my AWS environment
k

Kevin Kho

04/06/2021, 6:08 PM
I feel there are too many moving parts to ECS (we still have yet to get to Docker images and Execution/Task Role Permissions) that we might not be productive step-by-step debugging like this. What I’ll do is write an end-to-end tutorial around this (some time in the next week) and ping you when I complete it and that’ll be a much better starting point for us to get you up and running with ECS. Does that work?
This is a pain point on our end that a lot of people can’t get up and running on ECS independently. Having it in our docs is probably needed.
e

eric lumsden

04/06/2021, 6:30 PM
that'd be great, thank you Kevin
k

Kevin Kho

04/06/2021, 6:36 PM
Thank you for your patience as well 🙂