Thread
#prefect-community
    Patrick Tan

    Patrick Tan

    3 months ago
    Hi, I registered flow to S3 storage using AWS SSO credentials.
    with Flow("LiveLots-ETL-Parent", storage=S3(bucket=config_bucket,
                                                client_options={
                                                     "aws_access_key_id":aws_access_key_id,
                                                     "aws_secret_access_key":aws_secret_access_key,
                                                     "aws_session_token":aws_session_token},)) as f:
    Local agent is running on container as ECS task. The ECS task is attached with permission to access all S3 bucket.
    {
      "Statement": [
        {
          "Action": [
            "s3:*"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::*"
        },
        {
          "Action": [
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:s3:::wp-livelots-pt/*",
            "arn:aws:s3:::wp-live-lots/*"
          ]
        }
      ],
      "Version": "2012-10-17"
    }
    Initially flow is running without issue. After one hour, I am getting
    Failed to load and execute flow run: ClientError('An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.')
    Looks like ECS task is using the credentials when I registered flow. Please help
    Kevin Kho

    Kevin Kho

    3 months ago
    That’s a boto3 log so I think your
    aws_session_token
    is really expired. You might not even need it?
    Patrick Tan

    Patrick Tan

    3 months ago
    Correct. My agent does not need token because there is Iam role attached to ECS task. However, seems the agent is using the token since I registered flow with the token. Is there a way to instruct agent to ignore aws_session_token?
    Anna Geller

    Anna Geller

    3 months ago
    where does your ECS agent run - on the same ECS cluster or somewhere else?
    can you provide more info on how you set up your agent? you can check my post here for more guidance on setting up an ECS agent with IAM roles we also have a teraform recipe here https://github.com/PrefectHQ/prefect-recipes/tree/main/aws/prefect/prefect-agent-on-ecs
    I reread your messages and I'm a bit confused - is there any reason you decided to run a local agent on ECS? usually people use the ECS agent we have for that if you follow any of the above tutorials, it will walk you through how to set up an ECS agent if you would like to create one - this way you don't need credentials baked in and can rely on IAM roles
    Patrick Tan

    Patrick Tan

    3 months ago
    Yes I am running as local agent in ECS. I developed in local desktop using local agent, and I thought deploying to ECS as local agent involve minimal change. That’s why I did not spend time to figure out ECS agent. I’ll start looking into your tutorial, thanks.
    Anna Geller

    Anna Geller

    3 months ago
    awesome, that will be helpful I understand your motivation but ECS is a complex service so using the ECS agent will make things easier