Hi, how should I pass env variables for authentication in AWS S3 bucket to a job created using a custom template? I'm deploying using Terraform with Helm in a EKS Cluster

Is it for Prefect 2.0 or 1.0?

I'm using Prefect 1.0 I thought I could try to keep the IaC a bit more independent of AWS by using env vars. Is there a way? The job pod seems is able to pull the flow from the S3 Bucket but still can't access any of my env vars. I attached to it and showed it

PREFECT__CLOUD__USE_LOCAL_SECRETS=false

which I'm struggling to change. It even showed it had AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID set. But still couldn't get them to the KubernetesRun config. Should I use image_pull_secret in the KubernetesRun config? or set agent.env in the Helm chart?

aws_secret_key_id = Secret("PREFECT__CONTEXT__SECRETS__AWS_ACCESS_KEY_ID").get()
aws_secret_access_key = Secret("PREFECT__CONTEXT__SECRETS__AWS_SECRET_ACCESS_KEY").get()

#aws_secret_key_id = Secret("AWS_SECRET_ACCESS_KEY").get() #<- This doesn't work

KubernetesRun(
            job_template_path="job-template.yaml",
            image="public.ecr.aws/1235/my-image:dev",
            env=dict(AWS_ACCESS_KEY_ID=aws_secret_key_id, AWS_SECRET_ACCESS_KEY=aws_secret_access_key)
        )

I'd encourage you to use IAM roles since it's more secure and a recommended best practice by AWS if you go the route of env vars, check out this page https://discourse.prefect.io/t/how-to-set-secrets-e-g-github-access-token-on-server/70

Thank you a lot Anna, I will do that for production at least 😄