is there a simple way to use a secret in a custom task? so far I’ve been using a

RemoteEnvironment

and my existing cluster already loads

GOOGLE_APPLICATION_CREDENTIALS

from a k8s secret; for a

DaskKubernetesEnvironment

I guess I need to pull it from a prefect

Secret

instead. do I just need to add

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = Secret(...).get()

everywhere or is there an easier way?

Are you using a library that only looks for

GOOGLE_APPLICATION_CREDENTIALS

from an environment variable or are you able to provide it as an initialization argument?

this is just the

google.cloud.bigquery

library. what I wrote was slightly wrong: that env variable actually points to the path

/var/secrets/google/key.json

, the file is what gets loaded from the k8s secret

Okay I think I understand. So in using the

DaskKubernetesEnvironment

you want the path of the application credentials to be loaded on to each worker so the tasks that run on them can use the bigquery library?

yeah, our daskworker chart looks like

spec:
  template:
    spec:
      volumes:
      - name: google-cloud-key
        secret:
          secretName: svc-acct-big-query-key
...
      containers:
          env:
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /var/secrets/google/key.json

If the

GOOGLE_APPLICATION_CREDENTIALS

is set as an env var on each of the workers and the file is mounted then you shouldn’t need to also set a secret for it to be found in the tasks as it could just be retrieved through

os.getenv('GOOGLE_APPLICATION_CREDENTIALS')

if needed

right, but how do I get that file mounted w/o modifying https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/execution/dask/job.yaml ?

Ah yes, Currently the

DaskKubernetesEnvironment

doesn’t allow for a custom worker manifest (only the min/max # of workers) but we have kicked around the idea of creating a more custom version of the environment which allows for granular control over the worker/scheduler yaml. In it’s current iteration, and without updating the yaml file, you will have to set the environment variable

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = Secret(...).get()

in your custom task

cool, makes sense. one more follow-up: is there an equivalent of

distributed.Client.register_worker_callbacks

that makes sure a given setup function runs on each worker before any tasks execute? can probably hack my own but that seems like a common pattern

Hey Brett — over the weekend @Chris White and I were just talking about exposing the full dask worker lifecycle through Prefect, but right now no, we don’t have a hook for that

Customizable `DaskKubernetesEnvironment`s were added in https://github.com/PrefectHQ/prefect/pull/1534 and released as part of

version 0.6.5