https://prefect.io logo
c

Camilo Fernandez

06/01/2022, 1:02 PM
Hi, how should I pass env variables for authentication in AWS S3 bucket to a job created using a custom template? I'm deploying using Terraform with Helm in a EKS Cluster
a

Anna Geller

06/01/2022, 1:06 PM
Is it for Prefect 2.0 or 1.0?
generally speaking, the right way of assigning S3 permissions to any Kubernetes workload on EKS is by using https://eksctl.io/usage/iamserviceaccounts/
c

Camilo Fernandez

06/01/2022, 1:24 PM
I'm using Prefect 1.0 I thought I could try to keep the IaC a bit more independent of AWS by using env vars. Is there a way? The job pod seems is able to pull the flow from the S3 Bucket but still can't access any of my env vars. I attached to it and showed it
PREFECT__CLOUD__USE_LOCAL_SECRETS=false
which I'm struggling to change. It even showed it had AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID set. But still couldn't get them to the KubernetesRun config. Should I use image_pull_secret in the KubernetesRun config? or set agent.env in the Helm chart?
Copy code
aws_secret_key_id = Secret("PREFECT__CONTEXT__SECRETS__AWS_ACCESS_KEY_ID").get()
aws_secret_access_key = Secret("PREFECT__CONTEXT__SECRETS__AWS_SECRET_ACCESS_KEY").get()

#aws_secret_key_id = Secret("AWS_SECRET_ACCESS_KEY").get() #<- This doesn't work

KubernetesRun(
            job_template_path="job-template.yaml",
            image="public.ecr.aws/1235/my-image:dev",
            env=dict(AWS_ACCESS_KEY_ID=aws_secret_key_id, AWS_SECRET_ACCESS_KEY=aws_secret_access_key)
        )
a

Anna Geller

06/01/2022, 1:30 PM
I'd encourage you to use IAM roles since it's more secure and a recommended best practice by AWS if you go the route of env vars, check out this page https://discourse.prefect.io/t/how-to-set-secrets-e-g-github-access-token-on-server/70
c

Camilo Fernandez

06/01/2022, 1:34 PM
Thank you a lot Anna, I will do that for production at least 😄
👍 1
5 Views