https://prefect.io logo
Join Slack
Powered by
Hi, I registered flow to S3 storage using AWS SSO...
# prefect-community
p
Hi, I registered flow to S3 storage using AWS SSO credentials.
Copy code
with Flow("LiveLots-ETL-Parent", storage=S3(bucket=config_bucket,
                                            client_options={
                                                 "aws_access_key_id":aws_access_key_id,
                                                 "aws_secret_access_key":aws_secret_access_key,
                                                 "aws_session_token":aws_session_token},)) as f:
Local agent is running on container as ECS task. The ECS task is attached with permission to access all S3 bucket.
Copy code
{
  "Statement": [
    {
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::wp-livelots-pt/*",
        "arn:aws:s3:::wp-live-lots/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}
Initially flow is running without issue. After one hour, I am getting
Copy code
Failed to load and execute flow run: ClientError('An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.')
Looks like ECS task is using the credentials when I registered flow. Please help
k
That’s a boto3 log so I think your 
aws_session_token
is really expired. You might not even need it?
p
Correct. My agent does not need token because there is Iam role attached to ECS task. However, seems the agent is using the token since I registered flow with the token. Is there a way to instruct agent to ignore aws_session_token?
a
where does your ECS agent run - on the same ECS cluster or somewhere else?
can you provide more info on how you set up your agent? you can check my post here for more guidance on setting up an ECS agent with IAM roles we also have a teraform recipe here https://github.com/PrefectHQ/prefect-recipes/tree/main/aws/prefect/prefect-agent-on-ecs
I reread your messages and I'm a bit confused - is there any reason you decided to run a local agent on ECS? usually people use the ECS agent we have for that if you follow any of the above tutorials, it will walk you through how to set up an ECS agent if you would like to create one - this way you don't need credentials baked in and can rely on IAM roles
p
Yes I am running as local agent in ECS. I developed in local desktop using local agent, and I thought deploying to ECS as local agent involve minimal change. That’s why I did not spend time to figure out ECS agent. I’ll start looking into your tutorial, thanks.
a
awesome, that will be helpful I understand your motivation but ECS is a complex service so using the ECS agent will make things easier
👍 1
7 Views
Open in Slack
PreviousNext
Powered by