https://prefect.io logo
Title
b

Brett Naul

09/18/2019, 9:00 PM
is there a simple way to use a secret in a custom task? so far I’ve been using a
RemoteEnvironment
and my existing cluster already loads
GOOGLE_APPLICATION_CREDENTIALS
from a k8s secret; for a
DaskKubernetesEnvironment
I guess I need to pull it from a prefect
Secret
instead. do I just need to add
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = Secret(...).get()
everywhere or is there an easier way?
đź‘€ 1
j

josh

09/18/2019, 9:09 PM
Are you using a library that only looks for
GOOGLE_APPLICATION_CREDENTIALS
from an environment variable or are you able to provide it as an initialization argument?
If you are able to provide it as an argument to your library then you can set a Prefect Secret and
get
it inside your tasks https://docs.prefect.io/cloud/concepts/secrets.html#using-a-secret
b

Brett Naul

09/18/2019, 9:13 PM
this is just the
google.cloud.bigquery
library. what I wrote was slightly wrong: that env variable actually points to the path
/var/secrets/google/key.json
, the file is what gets loaded from the k8s secret
but this code is pretty deep in our library, it isn’t really feasible to inject a bunch of Prefect-specific logic that far down
j

josh

09/18/2019, 9:16 PM
Okay I think I understand. So in using the
DaskKubernetesEnvironment
you want the path of the application credentials to be loaded on to each worker so the tasks that run on them can use the bigquery library?
b

Brett Naul

09/18/2019, 9:18 PM
yeah, our daskworker chart looks like
spec:
  template:
    spec:
      volumes:
      - name: google-cloud-key
        secret:
          secretName: svc-acct-big-query-key
...
      containers:
          env:
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /var/secrets/google/key.json
j

josh

09/18/2019, 9:20 PM
If the
GOOGLE_APPLICATION_CREDENTIALS
is set as an env var on each of the workers and the file is mounted then you shouldn’t need to also set a secret for it to be found in the tasks as it could just be retrieved through
os.getenv('GOOGLE_APPLICATION_CREDENTIALS')
if needed
b

Brett Naul

09/18/2019, 9:21 PM
j

josh

09/18/2019, 9:24 PM
Ah yes, Currently the
DaskKubernetesEnvironment
doesn’t allow for a custom worker manifest (only the min/max # of workers) but we have kicked around the idea of creating a more custom version of the environment which allows for granular control over the worker/scheduler yaml. In it’s current iteration, and without updating the yaml file, you will have to set the environment variable
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = Secret(...).get()
in your custom task
b

Brett Naul

09/18/2019, 9:32 PM
cool, makes sense. one more follow-up: is there an equivalent of
distributed.Client.register_worker_callbacks
that makes sure a given setup function runs on each worker before any tasks execute? can probably hack my own but that seems like a common pattern
j

Jeremiah

09/18/2019, 9:45 PM
Hey Brett — over the weekend @Chris White and I were just talking about exposing the full dask worker lifecycle through Prefect, but right now no, we don’t have a hook for that
đź‘Ś 1
j

josh

09/20/2019, 9:18 PM
Customizable `DaskKubernetesEnvironment`s were added in https://github.com/PrefectHQ/prefect/pull/1534 and released as part of
version 0.6.5
@Marvin archive “Custom DaskKubernetesEnvironment scheduler and worker specifications”