https://prefect.io logo
#prefect-aws
Title
# prefect-aws
s

Santiago Toso

11/11/2022, 4:54 PM
hi all, I'm trying to create a Prefect deployment from a custom Docker image and get this error when running it
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
. It is uploading the file correctly, and even the image is building correctly but it is failing when I run the flow from the UI. From what I understand in the error message, when running the flow from the docker image, it can't access S3 to read the code. Here are a few more details: • Storage block --> S3. It uploads correctly • Docker image --> Custom image retrieved from ECR • Code --> It is the same code I use in other non-dockerized deployment. • The EC2 instances for the Prefect Agent, Prefect Server and S3 Bucket are all in the same AWS account • The EC2 instance for the PrefectAgent has S3 full access (should the EC2 instance for the Prefect Server also have full S3 access?) I leave in the image below with the full error I get in the Prefect UI and in the Agent's terminal. I saw a similar
AccessDenied
error in this channel when trying to upload the code to S3 but this seems different. The problem is when the flow is running in docker and can't retrieve the code. Any ideas why this could be? Thanks a lot for your help!
c

Christopher Boyd

11/11/2022, 4:56 PM
Forbidden means you explicitly do not have the permissions necessary
What kind of IAM permissions are allowed on this flow in s3, and what permissions are being applied? They should be bi-directional as well - that is, the container should be able to explicitly get /list / read that S3 bucket
because this is in the setup where the flow is attempting to be retrieved, you can probably try to list the identity this is running as (sts.getidentity()) and cross-reference that with the IAM permissions
s

Santiago Toso

11/11/2022, 4:58 PM
hi @Christopher Boyd thanks for the quick reply. The IAM permissions of the EC2 where the Agent is running has full access to S3, including the bucket where my code is being uploaded
I'll try that and see what I get. Should I add
sts.getidentity()
to my code? Could you point me to the API reference of this so I can take a deeper look?
what seems strange to me is that this only happens when I try to run the code from a docker container. Is there a way for me to specify the permissions of the docker container specifically? or it takes whatever permissions the EC2 instance where it is running has?
in case the second applies, I'm very confused as of why it is not working 😅
c

Christopher Boyd

11/11/2022, 5:01 PM
I would validate that you can in fact list / retrieve the object / flow from the EC2 instance first; a simple boto3 python script would be more than sufficient (or potentially just the aws cli)
Regarding the docker container piece - do you pass credentials to the docker container
or can you validate they are inherited
s

Santiago Toso

11/11/2022, 5:02 PM
no. We don't have credentials set up in that AWS account. All EC2 instances should be able to talk to all S3 buckets as long as the IAM has the right permissions.
c

Christopher Boyd

11/11/2022, 5:03 PM
if that’s the case, I would start small and just validate that
👍 1
s

Santiago Toso

11/11/2022, 5:03 PM
thanks, I'll take a look at that and report back 🙂
c

Christopher Boyd

11/11/2022, 5:04 PM
something ala:
Copy code
import boto3
s3 = boto3.resource('s3')
s3.Bucket('mybucket').download_file('hello.txt', '/tmp/hello.txt')
👍 1
but with your bucket + flow name from the ec2 instance, just to verify; if that works from ec2, then it should work in theory from the docker container
s

Santiago Toso

11/11/2022, 5:05 PM
thanks, I'm finishing my week now (in EU) but I'll give it a shot next week and see what I get. Thanks a lot Christopher
🙌 1
Copy code
in order to have access to objects within a bucket you need a /*
t

Tobias

11/14/2022, 7:43 AM
Dont know if this is relevant here, but if you are using prefect to upload to S3 with
--upload
there can be issues https://prefect-community.slack.com/archives/C048K0MGHNK/p1667990890478769
s

Santiago Toso

11/14/2022, 8:14 AM
thank you @Tobias, I looked into that thread but it seems that the problem is not when uploading things to S3 (as I can see the code uploaded correctly) but when Prefect tries to fetch the code to run it from a Docker container. I'll try @Christopher Boyd’s suggestions and come back to the thread. Thank you both!
Reporting back: • My permissions policy has all the permissions needed for reading and writing from the bucket where the code is (see image 1). • The code runs correctly when I deploy it as a "normal" prefect deployment. It only shows the
AccessDenied
error when running it as a dockerized deployment. I'll try deploying my image as a Lambda function to see if my Docker file has a problem. If it runs correctly I could only think the problem has to do with Prefect. Does this sound like a right conclusion to you?
c

Christopher Boyd

11/14/2022, 1:08 PM
Not really, it seems like a docker aws issue to me - by default, anything running in docker on EC2 should be able to pull permissions locally via localhost
👍 1
if it’s running in docker on ec2, it just seems like it’s not running as the right user in docker
I’d need to check on that, and I can check with the team as well, but I don’t really think it’s particularly a prefect problem - where would that be the case? Docker spins up, tries to connect to S3 and can’t
How do you have your iam role / profile / instance role set up?
p

Pekka

11/15/2022, 6:58 AM
I was getting
403 Forbidden
when the Prefect Orion server (docker image) was not the correct newest version w.r.t. to the client
s

Santiago Toso

11/15/2022, 3:50 PM
hey @Christopher Boyd my IAM role is set up for the EC2 where the agent is running and where I'm writing the code. I can try to set up the docker image I'm using as a lambda function and see if it works to make sure it is not a problem with the Docker image but the error message doesn't suggest something like that, does it? @Pekka would you mind to expand on that? I'm not sure I understand the "w.r.t" version you mean, or if it is the prefect orion server I should update (?)
c

Christopher Boyd

11/15/2022, 4:11 PM
Hi Santiago, I’ll need to try and reproduce this or test on my end - if I recall right, the issue is specifically when you try to run the flow - a docker container spins up, then fails to retrieve the flow, and exits out right?
s

Santiago Toso

11/15/2022, 5:25 PM
yes, that's exactly right @Christopher Boyd. The same flow works when I run it as a non-dockerized deployment, using the same storage block.
p

Pekka

11/16/2022, 6:11 AM
@Santiago Toso the Docker image had a Prefect Orion server that was 2.4.x (the latest being 2.6.x) I think while the python prefect package I was using was the latest possible version.
gratitude thank you 1
w.r.t. just means 'with regards to'
c

Christopher Boyd

11/16/2022, 2:58 PM
Hi Santiago, if you can find time to confirm the docker container is authenticating and pulling credentials for the right IAM user: https://stackoverflow.com/questions/22409367/fetching-aws-instance-metadata-from-within-docker-container/22411611#22411611
40 Views