hi all, I'm trying to create a Prefect deployment from a custom Docker image and get this error when running it

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

. It is uploading the file correctly, and even the image is building correctly but it is failing when I run the flow from the UI. From what I understand in the error message, when running the flow from the docker image, it can't access S3 to read the code. Here are a few more details: • Storage block --> S3. It uploads correctly • Docker image --> Custom image retrieved from ECR • Code --> It is the same code I use in other non-dockerized deployment. • The EC2 instances for the Prefect Agent, Prefect Server and S3 Bucket are all in the same AWS account • The EC2 instance for the PrefectAgent has S3 full access (should the EC2 instance for the Prefect Server also have full S3 access?) I leave in the image below with the full error I get in the Prefect UI and in the Agent's terminal. I saw a similar

AccessDenied

error in this channel when trying to upload the code to S3 but this seems different. The problem is when the flow is running in docker and can't retrieve the code. Any ideas why this could be? Thanks a lot for your help!

Forbidden means you explicitly do not have the permissions necessary

hi @Christopher Boyd thanks for the quick reply. The IAM permissions of the EC2 where the Agent is running has full access to S3, including the bucket where my code is being uploaded

I would validate that you can in fact list / retrieve the object / flow from the EC2 instance first; a simple boto3 python script would be more than sufficient (or potentially just the aws cli)

no. We don't have credentials set up in that AWS account. All EC2 instances should be able to talk to all S3 buckets as long as the IAM has the right permissions.

if that’s the case, I would start small and just validate that

thanks, I'll take a look at that and report back 🙂

something ala:

import boto3
s3 = boto3.resource('s3')
s3.Bucket('mybucket').download_file('hello.txt', '/tmp/hello.txt')

thanks, I'm finishing my week now (in EU) but I'll give it a shot next week and see what I get. Thanks a lot Christopher

Dont know if this is relevant here, but if you are using prefect to upload to S3 with

--upload

there can be issues https://prefect-community.slack.com/archives/C048K0MGHNK/p1667990890478769

thank you @Tobias, I looked into that thread but it seems that the problem is not when uploading things to S3 (as I can see the code uploaded correctly) but when Prefect tries to fetch the code to run it from a Docker container. I'll try @Christopher Boyd’s suggestions and come back to the thread. Thank you both!

Not really, it seems like a docker aws issue to me - by default, anything running in docker on EC2 should be able to pull permissions locally via localhost

I was getting

403 Forbidden

when the Prefect Orion server (docker image) was not the correct newest version w.r.t. to the client

hey @Christopher Boyd my IAM role is set up for the EC2 where the agent is running and where I'm writing the code. I can try to set up the docker image I'm using as a lambda function and see if it works to make sure it is not a problem with the Docker image but the error message doesn't suggest something like that, does it? @Pekka would you mind to expand on that? I'm not sure I understand the "w.r.t" version you mean, or if it is the prefect orion server I should update (?)

Hi Santiago, I’ll need to try and reproduce this or test on my end - if I recall right, the issue is specifically when you try to run the flow - a docker container spins up, then fails to retrieve the flow, and exits out right?

yes, that's exactly right @Christopher Boyd. The same flow works when I run it as a non-dockerized deployment, using the same storage block.

@Santiago Toso the Docker image had a Prefect Orion server that was 2.4.x (the latest being 2.6.x) I think while the python prefect package I was using was the latest possible version.

Hi Santiago, if you can find time to confirm the docker container is authenticating and pulling credentials for the right IAM user: https://stackoverflow.com/questions/22409367/fetching-aws-instance-metadata-from-within-docker-container/22411611#22411611