Brett Naul
09/18/2019, 9:00 PMRemoteEnvironment
and my existing cluster already loads GOOGLE_APPLICATION_CREDENTIALS
from a k8s secret; for a DaskKubernetesEnvironment
I guess I need to pull it from a prefect Secret
instead. do I just need to add os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = Secret(...).get()
everywhere or is there an easier way?josh
09/18/2019, 9:09 PMGOOGLE_APPLICATION_CREDENTIALS
from an environment variable or are you able to provide it as an initialization argument?get
it inside your tasks https://docs.prefect.io/cloud/concepts/secrets.html#using-a-secretBrett Naul
09/18/2019, 9:13 PMgoogle.cloud.bigquery
library. what I wrote was slightly wrong: that env variable actually points to the path /var/secrets/google/key.json
, the file is what gets loaded from the k8s secretjosh
09/18/2019, 9:16 PMDaskKubernetesEnvironment
you want the path of the application credentials to be loaded on to each worker so the tasks that run on them can use the bigquery library?Brett Naul
09/18/2019, 9:18 PMspec:
template:
spec:
volumes:
- name: google-cloud-key
secret:
secretName: svc-acct-big-query-key
...
containers:
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
josh
09/18/2019, 9:20 PMGOOGLE_APPLICATION_CREDENTIALS
is set as an env var on each of the workers and the file is mounted then you shouldn’t need to also set a secret for it to be found in the tasks as it could just be retrieved through os.getenv('GOOGLE_APPLICATION_CREDENTIALS')
if neededBrett Naul
09/18/2019, 9:21 PMjosh
09/18/2019, 9:24 PMDaskKubernetesEnvironment
doesn’t allow for a custom worker manifest (only the min/max # of workers) but we have kicked around the idea of creating a more custom version of the environment which allows for granular control over the worker/scheduler yaml. In it’s current iteration, and without updating the yaml file, you will have to set the environment variable os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = Secret(...).get()
in your custom taskBrett Naul
09/18/2019, 9:32 PMdistributed.Client.register_worker_callbacks
that makes sure a given setup function runs on each worker before any tasks execute? can probably hack my own but that seems like a common patternJeremiah
09/18/2019, 9:45 PMjosh
09/20/2019, 9:18 PMversion 0.6.5
Marvin
09/27/2019, 1:07 PM