https://prefect.io logo
Title
в

Владислав Богучаров

02/15/2022, 1:31 PM
Hello everyone! I use aws for my work. On my MacOS laptop .aws contains configuration files that are responsible for accessing my work cloud. I want to test prefect with another aws account but I'm afraid something might go wrong. As far as I understand, Prefect uses boto3, and boto3 will take configs from the default path. How to distinguish between a working aws profile and a home one, and what is the most important thing how to say about this to Prefect?
a

Anna Geller

02/15/2022, 1:50 PM
In your ~/.aws/credentials, you may have several profiles as shown in the image. You could then create a boto3 session with a specific profile, or set a default session in your Prefect task:
boto3.setup_default_session(profile_name="dev")
But once you move your flows somewhere to production, you may instead leverage IAM roles on your cluster (e.g. AWS EKS or ECS)
в

Владислав Богучаров

02/15/2022, 1:54 PM
Wow! I will definitely try it, thanks for the quick answer, Anna)
👍 1
r

Ricardo Gaspar

02/18/2022, 9:04 PM
hey @Anna Geller! BTW, is there away to pass the the profile or the
~/.aws/credentials
to my local docker agent? I’m starting a docker agent on my mac which is able to pull a docker image from ECR, but since I’m storing the flow on S3, it’s not able to access it; I guess the docker agent is not grabbing my credentials file. Errors:
prefect.S3	INFO	Downloading flow from <s3://xxxx/example-emr-jobs-success-docker-libs/2022-02-18t17-51-14-668658-00-00>
prefect.S3	ERROR	Error downloading Flow from S3: Unable to locate credentials
execute flow-run	ERROR	Failed to load and execute Flow's environment: NoCredentialsError('Unable to locate credentials')
I’m starting my local docker agent using the command:
prefect agent docker start --name "docker-$(hostname)" --label "local_docker_dev-$(hostname)"
I manged to set the ECR credstore configs on
~/docker/config.json
and using the
docker-credential-helper-ecr
so the agent is able to pull the images correctly.
i’ve tried a similar approach as this one, but still got no luck. https://gist.github.com/cameck/9ce71ba3e447444bf7f799d92c46c171 Different error though:
Failed to load and execute Flow's environment: ClientError('An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.')
a

Anna Geller

02/20/2022, 3:54 PM
You can pass env variables directly when you start your agent:
prefect agent docker start --env KEY=VALUE --env KEY2=VALUE2
You're kind of mixing multiple issues here so I'm not sure which problem do you try to address 🙂 • If you are using S3 storage, you can configure your aws CLI with credentials before starting the agent. • If you try to configure ECR pull permissions, you need to run this before you start your docker agent (this should update your docker config.json automatically I think):
aws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
• and if you need both, then you need to configure both ☝️
and once you deploy your agent somewhere like EC2 instance, you can assign IAM roles to the instance, which makes things much easier anyway. LMK if this is not helpful and you need more help here
r

Ricardo Gaspar

03/04/2022, 7:22 PM
@Anna Geller thanks for your reply. I’ve tried using the env variables, but maybe I’m doing something wrong here. In my AWS EC2 instance the docker agent runs fine because it is using the instance profile. where as in my local machine it seems to be using my
.aws/credentials
file. Can you tell me more about this process of the docker agent loading the credentials? Additionally, while starting the docker agent with the environment variables, it seems to be loading them (I can’t see the values loaded) but is it passed to the docker image? Yes, I forgot to tell you that I am using a custom docker image with custom library code/module - once again works fine on the cloud. the commands I’m using to run the docker agent locally: •
prefect agent docker start --name "docker-$(hostname)" --label "local_docker_dev-$(hostname)" --env AWS_ACCESS_KEY_ID="my_key_id" --env AWS_SECRET_ACCESS_KEY="my_key" --log-level DEBUG
◦ spits out this line, meaning that is creating those environment variables`[2022-03-04 18:33:06,549] DEBUG - docker-MacBook-Pro.local | Environment variables: ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY']` ◦ and outputs this error when I’m running the Flow:
Failed to load and execute Flow's environment: ClientError('An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.')
prefect agent docker start --name "docker-$(hostname)" --label "local_docker_dev-$(hostname)" --log-level DEBUG
◦ Outputs the following error:
Error downloading Flow from S3: Unable to locate credentials
Failed to load and execute Flow's environment: NoCredentialsError('Unable to locate credentials')
@Anna Geller @Kevin Kho whenever you get a chance, could you please help me out understanding how to run a docker agent locally with AWS credentials (see details ☝️ ). Thanks in advance
k

Kevin Kho

03/09/2022, 4:39 PM
First question on how the credentials are retrieved. That is done here for all AWs tasks and S3 storage. So it will look for secrets first and then default to just using the env variables. These env vars from the agent should be passed to the Flow. Are you on Cloud and do you have Secrets configured?
r

Ricardo Gaspar

03/09/2022, 5:00 PM
No I’m not on Prefect Cloud. I’m running prefect core 0.15.x on an EC2 instance (self-hosted). In my description above, I stated that EC2 machines running docker agent work fine by using the instance profile. However, my goal is now to be able to run my flow code locally using a docker agent; but it seems that the docker agent is not able to grab my AWS credentials properly, so I’m trying to understand how to properly pass them, or if there’s any config I need to do before launching the docker agent. Does that mean that I’ve to mount a volume on my local docker with my
~/.aws/.credentials
using the
--volume
https://docs.prefect.io/api/latest/cli/agent.html ?
k

Kevin Kho

03/09/2022, 5:18 PM
Ah ok I think it might be easier to get in as well with using local secrets maybe? This feels like it should work though because I think the docker agent spins a new container and then that container pulls the flow from S3. And the env variables are indeed put in there based on your logs. I think a good way to test is to see if you can instantiate a boto3 client after putting those env vars in the container?
Volume is a good idea though cuz it gets attached to each Flow if it can be pulled by default
r

Ricardo Gaspar

03/09/2022, 7:27 PM
Ok I’ll try with volume. I think my logs already show that the container is not able to pick the credentials from the env vars I’m setting (unless I’m doing it incorrectly). How could I use local secrets then, and rely on
PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS
? https://docs.prefect.io/core/concepts/secrets.html#default-secrets
a

Anna Geller

03/09/2022, 7:33 PM
Yup, that's pretty much how I always do it for local development with Docker - note that you must mount the credentials to the root user, it won't work otherwise:
prefect agent docker start --label AGENT_LABEL --volume ~/.aws:/root/.aws
Here is how I used it in a flow of flows example: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/different_images_per_subflows/parent_docker_different_images_per_subflow.py
🙏 1
b

Brad

04/11/2022, 9:14 PM
Hey @Anna Geller / @Kevin Kho I’m having the same issues running a docker agent on a remote host. Simply mounting my
~/.aws
folder won’t work as this is a remote host. What is the recommended way to set env vars on all flow runs? The docs mention
All agents have a --env flag for configuring environment variables to set on all flow runs managed by that agent.
but this does not appear to be working (I’m setting some AWS variables via
-e AWS_ACCESS_KEY_ID=xxx
)
k

Kevin Kho

04/11/2022, 9:16 PM
That should work. It gets added to the Flow run. Is your problem accessing an AWS resource from a task or are you trying to load a Flow from S3 storage?
b

Brad

04/11/2022, 9:16 PM
using s3 (fsspec / s3fs to be specific) in a task and getting the following:
File "/usr/local/lib/python3.9/site-packages/botocore/auth.py", line 388, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
k

Kevin Kho

04/11/2022, 9:18 PM
Have you tried checking the environment to see if it made it in the container from the agent?
b

Brad

04/11/2022, 9:19 PM
yeah, it doesnt
I can see env vars that I have added from my
DockerRun
config env
but not added to my agent
k

Kevin Kho

04/11/2022, 9:21 PM
That’s pretty weird. Are you on Cloud or Server?
b

Brad

04/11/2022, 9:22 PM
cloud
k

Kevin Kho

04/11/2022, 9:22 PM
You could just store it as a Prefect secret and then use
Secret.get()
to fetch it inside a task and then you don’t need to juggle passing it through the agent
b

Brad

04/11/2022, 9:23 PM
yeah but thats not going to work for things like AWS that are burried deep in tasks
I’ll raise an issue
k

Kevin Kho

04/11/2022, 9:26 PM
I find it hard to believe that wouldn’t work. Maybe you can add a log in the agent code here to see if the env var set makes it to
create_container
right underneath the try?
b

Brad

04/11/2022, 9:28 PM
Yeah I thought the agent at some point logged the env vars it was being set with?
k

Kevin Kho

04/11/2022, 9:31 PM
Oh I don’t think I’ve seen that explicitly logged cuz it could contain secrets. My current thought is there may be some edge case with the merging logic to get the RunConfig and env vars and add them to the container. This logic is always hard to read. But you could just log this variable after the merge to see
b

Brad

04/11/2022, 9:32 PM
Okay interesting - the env vars are being printed
[2022-04-11 21:32:24,092] INFO - agent | Pulling image prefecthq/prefect:1.2.0...
[2022-04-11 21:32:27,072] INFO - agent | Successfully pulled image prefecthq/prefect:1.2.0
ENV_VARS {'PREFECT__CLOUD__API': '<https://api.prefect.io>', 'PREFECT__LOGGING__LEVEL': 'INFO', 'AWS_ACCESS_KEY_ID': 'xxxxx', 'AWS_SECRET_ACCESS_KEY': 'xxxx'
k

Kevin Kho

04/11/2022, 9:35 PM
Uhhh….yeah then that means it’s being handed over to
create_container
right? I’m confused why it wouldn’t end up inside the container as an env var (not even tied to Prefect)
b

Brad

04/11/2022, 9:35 PM
yep
it looks like its after the merge - which means they should flow through
Okay I can see the vars making their way well into docker-py so this isnt a prefect bug ..
k

Kevin Kho

04/11/2022, 11:05 PM
what dockerpy version are you on?
b

Brad

04/11/2022, 11:11 PM
I think this is an issue on my end - on another host it seems to be working