Clément Peyrard
01/15/2023, 1:50 PMbotocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
+
PermissionError: Access Denied
when doing
prefect deployment build flow-example.py:hi -q default -n flow-example-aws-d --apply -sb s3/aws-personal-s3
I am using a IAM user key/secret, it works well for uploading files via AWS CLI, and Prefect Cloud Blocks (named .
As I'm discovering both Prefect Blocks and AWS S3/IAM management ; I have a few questions:
• are there constraints regarding the S3 buckets ? (regions, public access ?)
• do I need both AWS Credential and S3 blocks created in ?
• do I need to use manually-created nested blocks ? (but I don't see the point)
• do I need to use IAM roles (which I haven't studied yet 😇 )
boto* libs
- boto3=1.26.50
- botocore=1.29.50
- aiobotocore==2.4.2
- mypy-boto3==1.26.50
- mypy-boto3-s3==1.26.0.post1
prefect* version
prefect==2.7.8+4.ge291d4c18
prefect-aws==0.2.3
Mateo Merlo
01/18/2023, 8:47 PMClément Peyrard
01/18/2023, 8:51 PMMateo Merlo
01/18/2023, 8:54 PMClément Peyrard
01/18/2023, 9:19 PMMateo Merlo
01/18/2023, 9:28 PMClément Peyrard
01/18/2023, 9:34 PMMateo Merlo
01/18/2023, 9:37 PMNate
01/19/2023, 3:20 AMs3/aws-personal-s3
) have an AWSCredentials
block attached to it? I wonder if your machine is using a separate set of aws credentials when trying to upload the deployment files
should I let prefect create it ?sounds like you already have a bucket, but just a heads up - currently prefect will not create infrastructure like s3 buckets on your behalf, blocks like
S3Bucket
are just meant to represent existing resources out in the world (like AWS), and allow you to connect and interact with themClément Peyrard
01/19/2023, 1:38 PMAws Session Token
at block creation ?)
• Is there a nice way to check which authentication is used at upload ?Nate
01/19/2023, 9:23 PMAWSCredentials
stored on the S3Bucket
block to create a boto client, and if there aren't any credentials on the block, we'll defer to boto (which I believe will try to pull from your default credentials in ~/.aws/credentials
)
so while I don't think its strictly necessary to set AWSCredentials
on your S3Bucket
, I think it may be useful to try, if only to eliminate some potential causes of this error