Thread
#prefect-server
    e

    eric lumsden

    1 year ago
    hey guys, I am running prefect agent ecs start in terminal and getting this error. What is the syntax it wants me to use?
    ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @eric lumsden, can you give me more details about your run configuration?
    e

    eric lumsden

    1 year ago
    sure, I set up a new user under IAM in aws, the I used aws configure, set values for key ID, secret key, default region, and then left default output format as none. Ran the prefect agent ecs start... not sure what/where else it wants something specified
    Kevin Kho

    Kevin Kho

    1 year ago
    Are you using the default VPC?
    e

    eric lumsden

    1 year ago
    I'm not sure. How would I check?
    Kevin Kho

    Kevin Kho

    1 year ago
    I guess this will give instructors are to check your VPCs in the UI. Do you have UI access? Just check if you only have the default VPCs or if you have additional ones.
    e

    eric lumsden

    1 year ago
    I have UI access. I just browsed into "Your VPCs" and it is empty. What should my next step be?
    wait no I had to select the region
    we have additional VPCs
    Kevin Kho

    Kevin Kho

    1 year ago
    Ok. Just making sure, the aws config is in the right region?
    e

    eric lumsden

    1 year ago
    yes it is all set to us-east
    Kevin Kho

    Kevin Kho

    1 year ago
    When you go to ECS, do you see the default cluster?
    Or any cluster provisioned?
    e

    eric lumsden

    1 year ago
    oh boy ECS doesn't look like it has been set up at all. There are no clusters. Do y'all have a doc?
    Kevin Kho

    Kevin Kho

    1 year ago
    Let me look around
    aws ecs create-cluster
    try this?
    or really just create the default in the UI
    e

    eric lumsden

    1 year ago
    oh man I was actually trying this guide but then I saw Fargate was deprecated for y'all
    ok cluster is created after the aws create cluster but same error on prefect agent start. Is there work to do with configuring the VPC still?
    so that error is "ValueError: Failed to infer default networkConfiguration, please explicitly configure using --run-task-kwargs"
    Kevin Kho

    Kevin Kho

    1 year ago
    let me pull someone who can help better
    e

    eric lumsden

    1 year ago
    ok, sorry. I am very new to aws config
    Kevin Kho

    Kevin Kho

    1 year ago
    No worries. I pinged someone to help 🙂
    Btw, the fargate agent is deprecated but you can still use the ECS agent to run ECS tasks on Fargate
    e

    eric lumsden

    1 year ago
    thanks for your help. I'll sit on my hands so I don't mess up my config further
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you try doing something like this?
    RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                        run_task_kwargs={
                            "networkConfiguration": {
                                "awsvpcConfiguration": {
                                    "assignPublicIp": "ENABLED",
                                    "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                    "securityGroups": ["sg-123456"],
                                }
                            }
                        },
    )
    then pass it into your flow with
    run_config=RUN_CONFIG
    . get the VPC configuration from the default subnets
    I assume you didn’t add a VPC when you created the cluster right?
    e

    eric lumsden

    1 year ago
    I did not add a new VPC
    I'm sorry I don't appear to be getting this. Where am I supposed to run the RUN_CONFIG declaration?
    Kevin Kho

    Kevin Kho

    1 year ago
    Inside Python code and then pass this object to your
    Flow()
    call.
    RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                        run_task_kwargs={
                            "networkConfiguration": {
                                "awsvpcConfiguration": {
                                    "assignPublicIp": "ENABLED",
                                    "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                    "securityGroups": ["sg-123456"],
                                }
                            }
                        },
    )
    
    with Flow(name="ecs-test", storage=STORAGE, run_config=RUN_CONFIG, executor=EXECUTOR) as flow:
      print_log()
    flow.run()
    Let me know if you don’t know where do find the subnet and security group.
    e

    eric lumsden

    1 year ago
    am I missing a dependency?
    Kevin Kho

    Kevin Kho

    1 year ago
    is there a chance you named your file
    flow.py
    ?
    I think this might be an import conflict. Or do you have a folder named
    prefect
    inside?
    Actually sorry I just noticed this is interactive mode. I don’t think you’re missing any modules. This error might be because the function inside your flow is undefined
    Replace
    print_log
    with
    print("1")
    to test this block. You don’t have the STORAGE and EXECUTOR defined though so this will error. Do you have the original snippet that caused the error? Maybe we can take a look at that
    e

    eric lumsden

    1 year ago
    yeah I was about to type up that I shoved it all into a script and was getting an exception about STORAGE being undefined
    Kevin Kho

    Kevin Kho

    1 year ago
    The
    STORAGE
    is to point to where your code lives. For ECS it might point to the Docker image location. Something like:
    STORAGE = Docker(
        registry_url="<http://1234.dkr.ecr.us-east-2.amazonaws.com|1234.dkr.ecr.us-east-2.amazonaws.com>",
        image_name="ecs-test",
    )
    e

    eric lumsden

    1 year ago
    the original snippet was me trying to run "prefect agent ecs start"
    and I suppose I have to create the docker image location on AWS?
    Kevin Kho

    Kevin Kho

    1 year ago
    oh yeah oof! let me give you a YAML file instead that gives the VPC info that we can attach to the command line command. will reply a bit later
    e

    eric lumsden

    1 year ago
    thank you for your patience
    Kevin Kho

    Kevin Kho

    1 year ago
    Regardless though, I think this ECS documentation is definitely something we need to improve on. I will note that.
    and no problem!
    This thread shows how to create a YAML file, add the taskRoleArn and the execurionRoleArn.
    And then call it like this: https://docs.prefect.io/orchestration/agents/ecs.html#custom-runtime-options . Ultimately, there have been a couple of people who have run into this same issue. I have read that EKS was easier to setup so that may be an option. We have a couple of PR’s around this to make the experience easier and a minimum working example is definitely something we have in our minds
    e

    eric lumsden

    1 year ago
    ok I could use a little help finding the value for networkConfiguration. Getting this error: ValueError: Failed to infer default networkConfiguration, please explicitly configure using
    --run-task-kwargs
    I'm guessing I need to set the networkMode to an existing vpc. Unfortunately my environment shifted it's hosting region overnight so I am pointed somewhere with no VPCs. How do I point to the new region and select a VPC?
    Kevin Kho

    Kevin Kho

    1 year ago
    The region is on your CLI config. You start out with a YAML like this:
    networkMode: awsvpc
    cpu: 256
    memory: 512
    containerDefinitions:
      - name: flow
    requiresCompatibilities:
      - FARGATE
    executionRoleArn: arn:aws:iam::xxxxxxxxxx:role/tf-iam-role-prefect-ecs-tasks <--- change to whatever you need
    and then you add VPC info (looking for an example)
    e

    eric lumsden

    1 year ago
    ok I repointed my CLI to the new region, recreated the S3 bucket in the new region, grabbed one of the VPC ids from the new region and put it in the networkMode parameter in YAML... still get the same error complaining about my network configuration
    Kevin Kho

    Kevin Kho

    1 year ago
    yep i pinged someone on the team to give me a more fleshed out example. will get back to you
    e

    eric lumsden

    1 year ago
    ok, thank you. I'm probably missing something basic
    it looks like networkMode should be awsvpc, I may be missing a parameter to specify the actual vpc
    Kevin Kho

    Kevin Kho

    1 year ago
    ok sorry i mislead you from the python snippet yesterday. just to recap and start fresh, the problem is we need to give the VPC details
    create a file 
    run_task.yaml
     with this:
    networkConfiguration:
      awsvpcConfiguration:
        subnets:
          - 'subnet-1234'
        securityGroups:
          - 'sg-1234'
        assignPublicIp: 'ENABLED'
    run 
    prefect agent ecs start --run-task-kwargs /ecs/run_task.yaml
    ideally to simplify, subnet and security groups should be the default ones from your AWS account
    ignore the previous YAML. that is when we start to get into tasks
    e

    eric lumsden

    1 year ago
    ok... new error. I went for the subnet arn on the subnet parameter for the YAML file, hope that was correct over the subnet ID. I went the security group ID for lack of having another option. Now when I run prefect agent ecs start --run-task-kwargs s3locationwithyaml, I get prefect.utilities.exceptions.AuthorizationError: No agent API token provided. I went in to my cloud prefect account and generated an API token. Where do I specify its use?
    Kevin Kho

    Kevin Kho

    1 year ago
    The orchestration tutorial has snippets how to configure: https://docs.prefect.io/orchestration/tutorial/overview.html#install-prefect
    I also think you can revisit that tutorial you had now that we fixed the networkConfig 🙂
    e

    eric lumsden

    1 year ago
    ok using that I was able to authenticate with my generated API token to prefect cloud. I was able to kickoff a flow within prefect cloud that does not do anything with our AWS. When I go back and run
    prefect agent ecs start --run-task-kwargs <s3://bucketname>
    I still get the same error
    prefect.utilities.exceptions.AuthorizationError: No agent API token provided.
    we've been talking a while so maybe let me restate what I want to do so we're on track: trying to spin up ECS on AWS and connect it to prefect cloud so that I can kick off and manage flows in prefect cloud that will run on AWS
    so to that end we made a configuration file in YAML for ECS and have it on AWS, but I think I am missing the part where there is actual connection between the two cloud environments. Presumably there should be a listener set up on AWS waiting for inputs from Prefect Cloud that will then use the ECS to run flows. Where is the doc for that listener service/bridge between the two cloud environmnets
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah I’m looking into it.
    So I believe the issue here is that the agent needs the RUNNER token. Just to make sure, you’re doing the
    prefect agent ecs start
    on your local machine and you want the local machine to kick off ECS jobs right?
    But you were able to get the basic flow running on your local agent from cloud?
    Or are you using an EC2 instance to kick off the ECS job like the tutorial?
    The agent is also hold the AWS credentials (like seen in the tutorial)
    e

    eric lumsden

    1 year ago
    I was able to get a basic flow running on the cloud from my local agent. Basically, a hello world. I set my AWS creds in my local aws CLI and then called the prefect agent to start ec2 on my local. That's basically all the tutorial does, so I thought this was sufficient. What do I need to change so the prefect agent will know to use my aws credentials?
    Kevin Kho

    Kevin Kho

    1 year ago
    Just to clarify, you mean start ECS from the local? or did you mean an EC2 instance?
    e

    eric lumsden

    1 year ago
    At this point, I can't honestly say I am sure. Whichever will allow me to use prefect cloud to monitor and orchestrate data flows within my AWS environment
    Kevin Kho

    Kevin Kho

    1 year ago
    I feel there are too many moving parts to ECS (we still have yet to get to Docker images and Execution/Task Role Permissions) that we might not be productive step-by-step debugging like this. What I’ll do is write an end-to-end tutorial around this (some time in the next week) and ping you when I complete it and that’ll be a much better starting point for us to get you up and running with ECS. Does that work?
    This is a pain point on our end that a lot of people can’t get up and running on ECS independently. Having it in our docs is probably needed.
    e

    eric lumsden

    1 year ago
    that'd be great, thank you Kevin
    Kevin Kho

    Kevin Kho

    1 year ago
    Thank you for your patience as well 🙂