Hey everyone, Could you advice please how should I provide aws credentials to

KubernetesRun

? I tried to do the following:

run_config=KubernetesRun(
                env={
                    "AWS_ACCESS_KEY_ID": AWS_ACCESS_KEY_ID,
                    "AWS_SECRET_ACCESS_KEY": AWS_SECRET_ACCESS_KEY
                })

but when I run the flow, I get an error saying:

Error downloading Flow from S3: An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.

Any ideas?

@Gabi Pi it looks like you pass the credentials the right way, but those credentials are invalid. You can find more here. I would cross check in the AWS management console whether those credentials are still valid, and when in doubt, create a new IAM user.

@Anna Geller Thanks for the reply! Unfortunately it still doesn't work for me. Is there a way to provide aws credentials to

KubernetesRun

through a secret in the k8s cluster?

@Gabi Pi I haven’t used that myself, but you can try using this keyword argument on `KubernetesRun`: • `image_pull_secrets (list, optional)`: A list of image pull secrets to use for this job. If present, overrides any image pull secrets configured on the agent or in the job template.

Aren't

image_pull_secrets

used only for authentication against the docker repository?

I could ask the team. Do you use Prefect Cloud? If so, you could set those AWS credentials as

PrefectSecret

from the UI. It would look like this: Secret name = secret value AWS_CREDENTIALS=‘{“ACCESS_KEY”: “abcdef”, “SECRET_ACCESS_KEY”: “ghijklmn”}’

I think the credentials need to be set on the agent because the agent pulls the Flow. Do you have these environment variables set on your agent?

but I will cross check on a local K8s cluster to confirm. As another alternative, as Kevin mentioned, you could set those env variables on your agent: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION

@Anna Geller We are currently POCing Prefect and security wise we prefer not to set credentials in the UI. @Kevin Kho I didn't set the credentials on agent start, I thought that passing them through the

env

param in

KubernetesRun

would be enough. How can I set it on agent start? I create the k8s manifest using

prefect agent kubernetes install ... > manifest.yaml

and then run

kubectl apply -f manifest.yaml

@Gabi Pi the Secrets are encrypted and stored in Vault, but I can totally understand if you want it on your infrastructure. I do think that the credentials did work via

KubernetesRun

- they were passed successfully to the Kubernetes job. If you look at the error message that you got, it’s not an error from Prefect but from AWS, telling that it couldn’t download S3 object because those credentials are invalid. Sorry, if I’m annoying by saying it again, but I would really cross check with some DevOps folks whether the IAM user with those credentials 100% do have S3 get object permissions.

You can do

prefect agent kubernetes install --env AWS_ACCESS_KEY_ID=XXXXXX --env AWS_SECRET_ACCESS_KEY=XXXX

. My thinking is that the order is the Agent is the one responsible for downloading the Flow and then it gets run in the RunConfig. Checking the code.

I pass them into the job using a custom job template. The template has the secrets set as environment variables. These secrets are referenced from the K8 secrets stored in the same K8 namespace as the agent.

Ah gotcha. In this case though, it does seem like there is a credential. I think Anna will work on an example

Yeah I agree with Anna as well about IAM being the potential hiccup

@Anna Geller @Kevin Kho @Chris L. Thanks everyone for the help! It indeed was a credentials issue.