Hey everyone! Just started with Prefect, and I am having a great time so far. I’m running into an issue grabbing a flow from S3 and running it in a docker container. I think this should be straightforward; not sure where I am going wrong. Set-up: I am running prefect server, writing flows, and running

DockerAgent()

on my local computer, and I am using

S3

(private bucket) for my flow storage. I am having the flow run with

DockerConfig

, with a custom image. I am able to register the flow, and I see it in S3. Error: When trying to run the Flow, I get

Error downloading Flow from S3: Unable to
locate credentials

What I tried: 1. At first, I thought, oh!, the agent doesn’t have the creds (at first I thought the server might need it as well, but I don’t think that’s true). I shut down the agent,

export AWS_ACCESS_KEY_ID=...

and

export AWS_SECRET_ACCESS_ID=...

, in the shell, and restarted. This didn’t work. 2. I then figured maybe the container needs the creds. I (temporarily) hardcoded them into my docker image. This works! This could be a fine workaround. I could build the image with

--build-arg

to remove the hardcoding, but I don’t really want to bake my secrets in the container environment. I rather pass them in at runtime (via the agent)-- i.e., something like

docker run -e AWS_ACCESS_KEY_ID...

What I don’t understand: What is the best workflow for having docker containers pull flows from S3? I was looking in the prefect

Secrets

, but this seems like it’s for accessing secret within tasks, but getting flows from somewhere. Am I misunderstand how

Secrets

can be used? What would be nice: An example of the best practice. Sorry if this is such an obvious question! Still trying to get my bearings here.

I just ran into something similar, and it turns out that only the local Agent copies all the env variables over verbatim. With anything else, we need to use the

--env

parameter when launching the Agent

Wilson is correct. There are a few different ways to provide the necessary AWS credentials, including the

--env

parameter. If you're only running a docker agent on your local machine that's reasonable. For more complex setups with additional security concerns, the "best practice" offered by Prefect is using the default Secret

AWS_CREDENTIALS

. You can read a bit more about how to configure it here https://docs.prefect.io/core/concepts/secrets.html#default-secrets

Wow thanks for the quick replies! I’ll give this a whirl.