https://prefect.io logo
Title
c

Clément Peyrard

01/15/2023, 1:50 PM
Hello all, Despite reading the various posts about the "Access Denied" S3 error ; I can't get my deployment to upload to S3 from my local laptop 😞.
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
+
PermissionError: Access Denied
when doing
prefect deployment build flow-example.py:hi -q default -n flow-example-aws-d --apply -sb s3/aws-personal-s3
I am using a IAM user key/secret, it works well for uploading files via AWS CLI, and Prefect Cloud Blocks (named . As I'm discovering both Prefect Blocks and AWS S3/IAM management ; I have a few questions: • are there constraints regarding the S3 buckets ? (regions, public access ?) • do I need both AWS Credential and S3 blocks created in ? • do I need to use manually-created nested blocks ? (but I don't see the point) • do I need to use IAM roles (which I haven't studied yet 😇 ) boto* libs
- boto3=1.26.50
- botocore=1.29.50
- aiobotocore==2.4.2
- mypy-boto3==1.26.50
- mypy-boto3-s3==1.26.0.post1
prefect* version
prefect==2.7.8+4.ge291d4c18
prefect-aws==0.2.3
m

Mateo Merlo

01/18/2023, 8:47 PM
Hi @Clément Peyrard, has the IAM user the AmazonS3FullAccess permission?
c

Clément Peyrard

01/18/2023, 8:51 PM
Yes, it belongs to a User Group that has it, and it looks like they flow correctly to the user (PrefectS3BucketPolicy is an attempt to add all kms rights)
m

Mateo Merlo

01/18/2023, 8:54 PM
Did you set up the S3 block from the UI or with a script?
Seems like the permissions are ok
c

Clément Peyrard

01/18/2023, 9:19 PM
From the UI ; tried from a script as well (from AWSCredentials as recommended by @Bianca Hoch in https://prefect-community.slack.com/archives/CM28LL405/p1669831027116889 ) but no success - and I don't understand this recommendation.
m

Mateo Merlo

01/18/2023, 9:28 PM
You have created the bucket, don't you?
c

Clément Peyrard

01/18/2023, 9:34 PM
Yes, I manually created it beforehand - should I let prefect create it ?
m

Mateo Merlo

01/18/2023, 9:37 PM
No, I think is ok
n

Nate

01/19/2023, 3:20 AM
Hi @Clément Peyrard - sorry you've been running into trouble does your s3 bucket block (
s3/aws-personal-s3
) have an
AWSCredentials
block attached to it? I wonder if your machine is using a separate set of aws credentials when trying to upload the deployment files
should I let prefect create it ?
sounds like you already have a bucket, but just a heads up - currently prefect will not create infrastructure like s3 buckets on your behalf, blocks like
S3Bucket
are just meant to represent existing resources out in the world (like AWS), and allow you to connect and interact with them
c

Clément Peyrard

01/19/2023, 1:38 PM
Hi @Nate, thanks for the reply. At first I did not "attached" an AWSCredentials (I thought a S3bucket block could exist on its own but I'm a bit unclear on this given past responses and doc). But I also did a test following the link I shared and I was not successful by creating a manual S3 block. • Is it mandatory to have a AWSCredentials block ? (and if so, is it mandatory to provide a
Aws Session Token
at block creation ?) • Is there a nice way to check which authentication is used at upload ?
n

Nate

01/19/2023, 9:23 PM
I didn't implement this, but from looking here (which is ultimately how we connect you to s3 when you build a deployment that will be stored in s3) it seems that we'll first attempt to use
AWSCredentials
stored on the
S3Bucket
block to create a boto client, and if there aren't any credentials on the block, we'll defer to boto (which I believe will try to pull from your default credentials in
~/.aws/credentials
) so while I don't think its strictly necessary to set
AWSCredentials
on your
S3Bucket
, I think it may be useful to try, if only to eliminate some potential causes of this error