This is a bit weird - so I'm using an

EnvVarSecret()

and running on Kubernetes, using a Kubernetes Agent. I'm using a customized agent deployement with the following env vars added (populated via Kube secrets):

- name: PREFECT__CLOUD__USE_LOCAL_SECRETS
          value: 'true'
        - name: AWS_ACCESS_KEY_ID
          valueFrom: 
            secretKeyRef:
              name: aws-access-key
              key: aws_access_key
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-secret-key
              key: aws_secret_key

The keys end up on the agent no problem - I can see them being passed through in the deployment and pod config. I even started a shell session on the agent pod and I can see the env vars there no problem! However, when I try to reference them in a flow running on that agent, I get errors like this:

Task 'AWS_SECRET_ACCESS_KEY': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/secrets/env_var.py", line 50, in run
    raise ValueError("Environment variable not set: {}".format(name))
ValueError: Environment variable not set: AWS_SECRET_ACCESS_KEY

These flows work locally with the Env Vars in question set, and they work if I set them manually in the "Run" area of the UI's Kubernetes Run Configuration under "Environment Variables". Shouldn't flows running on an agent pick up the env vars from the agent by default?

Are you using dask executor? Is that the case this is your problem. When you run in dask the task is looking for the env on the worker not in the agent. They are a kubernetessecret that we are using for our use case

Check out this thread - the solution is to use IAM Roles for Service Accounts. This is the AWS-recommended way of setting credentials to interact with AWS services within Kubernetes workloads. This applies to flow run pods but also to Dask nodes deployed within the same cluster as those permissions can be applied on the cluster level.

Thank @Anna I didn't know about that. Great to know that kubernetes service account permission can extend IAM

@davzucky It fails with the local executor as well. Thanks @Anna Geller - I can check that out, but any thoughts on why Env Vars aren't being picked up? Should I use a job template that references the Kube secret?

Can you share how you use those in your flow? I can share a Discourse topic showing how to set Secrets for Server but IAM roles for service accounts is the recommended way and I would definitely encourage you to try that

For sure. They're being called like so:

access_key = EnvVarSecret(name='AWS_ACCESS_KEY_ID', raise_if_missing=True)
secret_access_key = EnvVarSecret(name='AWS_SECRET_ACCESS_KEY', raise_if_missing=True)

My concern with the IAM roles path is that we'd like something that works locally and when deployed on EKS.

That's quite easy though - locally you can just authenticate your terminal using "aws configure" Nothing suspicious in your config per se as long as you call that within your flow rather than on a module level

Yep, those calls are w/in the flow and they work locally just fine - so in theory, an Agent using a local executor should be able to see those variables when they're set on the Agent, right?

not even that, when you trigger a local flow run, Prefect uses a local process (no agent required), and the run_config and storage are ignored - the executor is not that relevant in this context To trigger a local flow run, you can do:

flow.run()

or:

prefect run -p yourflow.py

@Anna Geller Yeah, so the environment variables are picked up just fine locally. It's just when they run on the agent that they aren't getting picked up for some reason, even when using a local executor...but the environment variables are absolutely available on the agent:

Sure, happy to help! I'd suggest for local debugging and locally testing flows to not use Prefect Server but only use Prefect Core, i.e. do:

flow.run()

or:

prefect run -p yourflow.py

This way you are using just a local process and the agents don't matter. And for your production Helm deployment use IAM roles for service accounts. Environment variables are hard because they are many components here and you would need to be careful that they are set everywhere they are needed - I wouldn't go that route, especially given that for local run you really don't need any environment variables whatsoever - just configure your terminal with

aws configure

and you are good to go locally, and use

eksctl

for server-side IAM roles setup on Kubernetes

@Anna Geller - yeah, we're using Prefect Core for local debugging/testing, not server. The Prefect Server here is deployed to AWS EKS. When you say "configure your terminal with

aws configure

do you mean using the AWS CLI to setup the

~/.aws/credentials

file so that the default credentials are set? The issue there is that we need multiple sets of AWS credentials, sometimes to services outside the AWS environment we're running in.

Yes, exactly! and to manage multiple environments, you can set multiple AWS profiles. For other services you need to figure out some solution that fits your use case, hard to help here really - it's a matter of personal preference, some people use AWS secrets manager or parameter store for that and this is better than setting credentials in environment variables A job template would set the env variables for the relevant flow run pod but you probably realized that I'm really trying to persuade you NOT to use environment variables for that 😄

Yeah, we're considering Hashi Vault for longer-term secrets mgmt. I did test out the Kube Secret task but it seemed a bit janky.

it's an ambitious but often not a worthwhile goal from my experience 😄 but I can understand that

Yep, so the issue is these jobs don't actually run on the agent - because it's a

KubernetesRun

, they actually run as a Kube Job w/ an ephemal Pod spun up. Makes sense now, thanks!

I'm no DevOps expert, but if you still have any questions about it, I may try to help but generally with credentials using Hashicorp Vault or some secrets manager or Kubernetes Secrets, or IAM roles is definitely a better approach than manually injecting env variables

Thanks @Anna Geller - we're not manually injecting Env Vars though - they're coming from Kube Secrets. But we'll probably move to Vault soon. 🙂