Владислав Богучаров
02/15/2022, 1:31 PMAnna Geller
02/15/2022, 1:50 PMboto3.setup_default_session(profile_name="dev")
But once you move your flows somewhere to production, you may instead leverage IAM roles on your cluster (e.g. AWS EKS or ECS)Владислав Богучаров
02/15/2022, 1:54 PMRicardo Gaspar
02/18/2022, 9:04 PM~/.aws/credentials
to my local docker agent? I’m starting a docker agent on my mac which is able to pull a docker image from ECR, but since I’m storing the flow on S3, it’s not able to access it; I guess the docker agent is not grabbing my credentials file.
Errors:
prefect.S3 INFO Downloading flow from <s3://xxxx/example-emr-jobs-success-docker-libs/2022-02-18t17-51-14-668658-00-00>
prefect.S3 ERROR Error downloading Flow from S3: Unable to locate credentials
execute flow-run ERROR Failed to load and execute Flow's environment: NoCredentialsError('Unable to locate credentials')
I’m starting my local docker agent using the command:
prefect agent docker start --name "docker-$(hostname)" --label "local_docker_dev-$(hostname)"
I manged to set the ECR credstore configs on ~/docker/config.json
and using the docker-credential-helper-ecr
so the agent is able to pull the images correctly.Failed to load and execute Flow's environment: ClientError('An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.')
Anna Geller
02/20/2022, 3:54 PMprefect agent docker start --env KEY=VALUE --env KEY2=VALUE2
You're kind of mixing multiple issues here so I'm not sure which problem do you try to address 🙂
• If you are using S3 storage, you can configure your aws CLI with credentials before starting the agent.
• If you try to configure ECR pull permissions, you need to run this before you start your docker agent (this should update your docker config.json automatically I think):
aws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
• and if you need both, then you need to configure both ☝️Ricardo Gaspar
03/04/2022, 7:22 PM.aws/credentials
file. Can you tell me more about this process of the docker agent loading the credentials?
Additionally, while starting the docker agent with the environment variables, it seems to be loading them (I can’t see the values loaded) but is it passed to the docker image? Yes, I forgot to tell you that I am using a custom docker image with custom library code/module - once again works fine on the cloud.
the commands I’m using to run the docker agent locally:
• prefect agent docker start --name "docker-$(hostname)" --label "local_docker_dev-$(hostname)" --env AWS_ACCESS_KEY_ID="my_key_id" --env AWS_SECRET_ACCESS_KEY="my_key" --log-level DEBUG
◦ spits out this line, meaning that is creating those environment variables`[2022-03-04 18:33:06,549] DEBUG - docker-MacBook-Pro.local | Environment variables: ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY']`
◦ and outputs this error when I’m running the Flow:
Failed to load and execute Flow's environment: ClientError('An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.')
• prefect agent docker start --name "docker-$(hostname)" --label "local_docker_dev-$(hostname)" --log-level DEBUG
◦ Outputs the following error:
Error downloading Flow from S3: Unable to locate credentials
Failed to load and execute Flow's environment: NoCredentialsError('Unable to locate credentials')
Kevin Kho
03/09/2022, 4:39 PMRicardo Gaspar
03/09/2022, 5:00 PM~/.aws/.credentials
using the --volume
https://docs.prefect.io/api/latest/cli/agent.html ?Kevin Kho
03/09/2022, 5:18 PMRicardo Gaspar
03/09/2022, 7:27 PMPREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS
?
https://docs.prefect.io/core/concepts/secrets.html#default-secretsAnna Geller
03/09/2022, 7:33 PMprefect agent docker start --label AGENT_LABEL --volume ~/.aws:/root/.aws
Here is how I used it in a flow of flows example:
https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/different_images_per_subflows/parent_docker_different_images_per_subflow.pyBrad
04/11/2022, 9:14 PM~/.aws
folder won’t work as this is a remote host.
What is the recommended way to set env vars on all flow runs? The docs mention All agents have a --env flag for configuring environment variables to set on all flow runs managed by that agent.
but this does not appear to be working (I’m setting some AWS variables via -e AWS_ACCESS_KEY_ID=xxx
)Kevin Kho
04/11/2022, 9:16 PMBrad
04/11/2022, 9:16 PMFile "/usr/local/lib/python3.9/site-packages/botocore/auth.py", line 388, in add_auth
raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Kevin Kho
04/11/2022, 9:18 PMBrad
04/11/2022, 9:19 PMDockerRun
config envKevin Kho
04/11/2022, 9:21 PMBrad
04/11/2022, 9:22 PMKevin Kho
04/11/2022, 9:22 PMSecret.get()
to fetch it inside a task and then you don’t need to juggle passing it through the agentBrad
04/11/2022, 9:23 PMKevin Kho
04/11/2022, 9:26 PMcreate_container
right underneath the try?Brad
04/11/2022, 9:28 PMKevin Kho
04/11/2022, 9:31 PMBrad
04/11/2022, 9:32 PM[2022-04-11 21:32:24,092] INFO - agent | Pulling image prefecthq/prefect:1.2.0...
[2022-04-11 21:32:27,072] INFO - agent | Successfully pulled image prefecthq/prefect:1.2.0
ENV_VARS {'PREFECT__CLOUD__API': '<https://api.prefect.io>', 'PREFECT__LOGGING__LEVEL': 'INFO', 'AWS_ACCESS_KEY_ID': 'xxxxx', 'AWS_SECRET_ACCESS_KEY': 'xxxx'
Kevin Kho
04/11/2022, 9:35 PMcreate_container
right? I’m confused why it wouldn’t end up inside the container as an env var (not even tied to Prefect)Brad
04/11/2022, 9:35 PMKevin Kho
04/11/2022, 11:05 PMBrad
04/11/2022, 11:11 PM