for prefect core, is it possible to set context variables for flow runs in kubernetes agent? I tried using the --env flag but whenever I try and run my flow I get this error

May 26th 2020 at 9:00:18am | agent
ERROR 
(400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '9f273015-9326-4806-8497-6847f42b700b', 'Content-Type': 'application/json', 'Date': 'Tue, 26 May 2020 13:00:18 GMT', 'Content-Length': '549'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Job in version \"v1\" cannot be handled as a Job: v1.Job.Spec: v1.JobSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects \" or n, but found {, error found in #10 byte of ...|\"value\": {\"context\":|..., bigger context ...|.CloudTaskRunner\"}, {\"name\": \"prefect\", \"value\": {\"context\": {\"aws_secret\": |...","reason":"BadRequest","code":400}

I don't have this problem when running on docker agent

Hi @Will Milner, this should definitely be possible. Let me open an issue for you and we'll take a look!

Hey @Will Milner could I see an example of what you’re passing in for the

--env

flag?

this is the full command i'm running

prefect agent install --env PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS='"{\"aws_access_key\": \"ACCESS_KEY\",\"aws_secret\": \"SECRET_KEYi\"}" kubernetes --rbac --backend server --api <http://10.0.0.61:4200> --image-pull-secrets my-secret | kubectl apply -f -

Ah I don’t thing that would work (due to yaml parsing / k8s / etc…). You can pass many

env

options to that command and each should be a string. So it would look more like this:

prefect agent install --env PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS__AWS_ACCESS_KEY=YOUR_KEY --env PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS__AWS_SECRET=YOURSECRET

i also tried that and still got the same error

prefect agent install --env PREFECT__CONTEXT__AWS_CREDENTIALS__AWS_ACCESS_KEY="ACCESS_KEY" --env PREFECT__CONTEXT__AWS_CREDENTIALS__AWS_SECRET="SECRET_KEY" kubernetes --rbac --backend server --api <http://10.0.0.61:4200> --image-pull-secrets my-secret | kubectl apply -f -

🤔 Interesting, some stuff recently changed with our env parsing so it could be that. Could you open an issue with some more info?

yup

Thanks! It looks like the agent env vars set under a heading of

PREFECT__CLOUD__AGENT__ENV_VARS__

So I’m now thinking that there’s just some weird stuff happening when also trying to set context that deep.

posted an issue, linking here in case anyone else runs into this issue and wants to track - https://github.com/PrefectHQ/prefect/issues/2652

Awesome! Will look into the behavior, thanks for finding the bug haha

quick follow up, when I pass in an environment variable to kubernetes in this way, how do I go about accessing it in my flows since it won't be in the context

You can grab it using something like

os.getenv("YOUR_KEY")

ah turns out the keys get set to lower case inside of the kubernetes job

Oh interesting 🤔 I’ll make note of that