Darragh

    Darragh

    2 years ago
    Hey guys, having 2 problems trying to use FargateTaskEnviornment with FargateAgent. The docs page confused me ever so slightly, so I’m not 100% sure I have it all configured correctly. flow.environment is configured as per the following snippet, and builds and registers with my server:
    flow.environment = FargateTaskEnvironment(
            launch_type="FARGATE",
            region="eu-west-1",
            cpu="256",
            memory="512",
            networkConfiguration={
                "awsvpcConfiguration": {
                    "assignPublicIp": "ENABLED",
                    "subnets": ["subnet-X"],
                    "securityGroups": ["sg-Y"],
                }
            },
            family="my_flow",
            taskRoleArn="arn:aws:iam::X:role/Role",
            executionRoleArn="arn:aws:iam::X:role/Role",
            containerDefinitions={
                "name": "my-flow",
                "image": "my-flow",
                "command": [],
                "environment": [],
                "essential": True,
            }
        )
    Fargate Agent:
    echo "export REGION_NAME=${CDK_DEPLOY_REGION}" > .env
    echo "export AWS_DEFAULT_REGION=${CDK_DEPLOY_REGION}" >> .env
    echo "export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" >> .env
    echo "export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}" >> .env
    source .env
    nohup prefect agent start fargate > prefect_fargate.log 2>&1 & disown
    Everything seems fine until I try to run it. There’s no update in the agent logs to show it’s been picked up, and the UI just says:
    Failed to load and execute Flow's environment: AttributeError("'str' object has no attribute 'get'")
    Anything immediate/obvious/stupid stand out? As an aside I’m guessing I’m missing a bunch of config for the agent, but the docs are a little unclear on that one, the top level of the docs [https://docs.prefect.io/orchestration/agents/fargate.html#installation] seem to suggest you only need the handful of vars I added UPDATE: Some progress, it’s now giving a meaningful error,
    Parameter validation failed: Missing required parameter in containerDefinitions[0].logConfiguration: "logDriver
    Fixing it
    j

    josh

    2 years ago
    Oh interesting, what did you fix to alleviate the first error?
    Darragh

    Darragh

    2 years ago
    Killed the duplicate processes that “some nameless idiot” had mistakenly started 🙂
    I had 2 servers and 2 fargate agents running due to a CI/CD womp
    ONTO THE NEXT PROBLEM!!
    The following script returns
    AttributeError: 'tuple' object has no attribute 'start'
    from prefect.agent.fargate import FargateAgent
    
    agent = FargateAgent(
        launch_type="FARGATE",
        aws_access_key_id="KEY",
        aws_secret_access_key="KEY",
        region_name="eu-west-1",
        networkConfiguration={
            "awsvpcConfiguration": {
                "assignPublicIp": "ENABLED",
                "subnets": ["subnet-X"],
                "securityGroups": ["sg-X"],
            }
        },
        cpu="256",
        memory="512",
        platformVersion="1.4.0",
        containerDefinitions=[{
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/my/log/group",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "prefect-flow-runs",
                    "awslogs-create-group": "true",
                },
            },
        }],
    ),
    
    agent.start()
    This seems like a very very stupid problem - am I missing an import/install in the environment or something?
    j

    josh

    2 years ago
    🤔 I have a script here where I call start on a fargate agent and don’t get that tuple issue
    Which version of Prefect are you running?
    Darragh

    Darragh

    2 years ago
    0.11.1 locally, and 0.11.4 on AWS, both give the same error
    j

    josh

    2 years ago
    Looks like you may be defining it in a tuple. Try this:
    agent = FargateAgent(
        launch_type="FARGATE",
        aws_access_key_id="KEY",
        aws_secret_access_key="KEY",
        region_name="eu-west-1",
        networkConfiguration={
            "awsvpcConfiguration": {
                "assignPublicIp": "ENABLED",
                "subnets": ["subnet-X"],
                "securityGroups": ["sg-X"],
            }
        },
        cpu="256",
        memory="512",
        platformVersion="1.4.0",
        containerDefinitions=[
            {
                "logConfiguration": {
                    "logDriver": "awslogs",
                    "options": {
                        "awslogs-group": "/my/log/group",
                        "awslogs-region": "eu-west-1",
                        "awslogs-stream-prefix": "prefect-flow-runs",
                        "awslogs-create-group": "true",
                    },
                },
            }
        ],
    )
    I think that extra comma at the end of your snippet had something to do with it
    Darragh

    Darragh

    2 years ago
    Literally just spotted that as you replied. The shame. THE SHAME.
    j

    josh

    2 years ago
    Hahaha
    Is it running now?
    Darragh

    Darragh

    2 years ago
    Getting there - latest issue:
    Fargate requires task definition to have execution role ARN to support log driver awslogs
    Far as I can tell I’ve added full cloudwatch logs access to the role I’m using
    Output from aws iam command:
    {
    “RoleName”: “ROLE”, “PolicyName”: “ROLE_POLICY, “PolicyDocument”: { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “VisualEditor0", “Effect”: “Allow”, “Action”: [ “cloudwatch😗“, “logs😗“, “ecs😗“, “ecr😗” ], “Resource”: “*” } ] } }
    j

    josh

    2 years ago
    Hey @Darragh unfortunately I am not well versed in AWS issues but I would say that you could attempt using the same boto3 functions that the FargateAgent uses with the same kwargs to see if you could pin down the issue (
    register_task_definition
    and
    run_task
    )
    Jeff Brainerd

    Jeff Brainerd

    2 years ago
    @Darragh not sure if this helps but I just ran into the same
    logConfiguration
    issue after upgrading
    boto3
    , because we weren’t passing it to the agent. We now have it running successfully again. Our IAM role includes the AWS managed policies
    AmazonECS_FullAccess
    and
    AmazonECSTaskExecutionRolePolicy
    , both of which include some cloudwatch actions. You might want to take a look at those…
    Darragh

    Darragh

    2 years ago
    Thanks for those @josh and @Jeff Brainerd between the two i should find an answer. And if not ill be back with more questions ! And if it works ill probably still be back with questions 😂
    j

    josh

    2 years ago
    @Jeff Brainerd just wondering was this observed when a new version of boto3 was released that is within the bounds of what prefect pins it to?
    "aws": ["boto3 >= 1.9, < 2.0"],
    Jeff Brainerd

    Jeff Brainerd

    2 years ago
    🤔 our
    boto3
    library upgraded from
    1.12.x
    to
    1.13.20
    — which looks like it’s the latest version, according to https://boto3.amazonaws.com/v1/documentation/api/latest/index.html# — so maybe you guys are grabbing a different library somehow?
    Darragh

    Darragh

    2 years ago
    Installed a clean version of boto3 there -
    boto3-1.13.22
    So those two policies didn’t do it, which makes me think my Role is just buggered.
    I’m not getting anywhere with the policies, I may have to push the question back out to the wider group - thanks again guys, appreciate the help!