Hi wonder if anyone could help me with a problem I have working with the Dask KubeCluster? So the issue I am having is that various secrets that I have mounted to the usual flow jobs don't get carried over to the pods that are started by dask - there is an added complexity that I am using two images a dev one and a non dev one tied to two different prefect projects - I am able to do something like this to switch the image;

DEV_TAG = os.environ.get("DEV", "") != ""

JOB_IMAGE_NAME = f"blah/flows{':dev' if DEV_TAG else ''}"

And then in each flow I ref the

JOB_IMAGE_NAME

- this just changes the image but otherwise uses the job template I have defined on the agent;

apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
        - name: flow
          imagePullPolicy: Always
          env:
            - name: SOME_ENV
              valueFrom:
                secretKeyRef:
                  name: secret-env-vars
                  key: some_env
                  optional: false

Now when I specify the dask setup I do the following;

executor=DaskExecutor(
        cluster_class=lambda: KubeCluster(make_pod_spec(image=JOB_IMAGE_NAME)),
        adapt_kwargs={"minimum": 2, "maximum": 3},
    )

But this is obviously missing the env part of my default template - I would like to not have to respecify it (its much bigger then the above snippet) - is it possible to grab a handle on the default template and just override the image name?

are you on Prefect Cloud or Server? if you are on Cloud, you could leverage Prefect Secrets which would make the process much easier as you could set those directly from the Prefect Cloud UI

Hmm yeah unfortunately not allowed to store these there 😞

why? we are SOC-2 compliant

We just have a policy that all secrets must stay under our control - would be lots of bureaucracy to convince otherwise

thanks for sharing - so your service account points to this environment variable?

No so two things; 1. Env vars are set on the pod created by the agent - since this pod creates the dask pods can just insert 2.

service_account_name

is needed so that I am able to access GCP resources -

KubeCluster

seems to default to using the

default

service account

I see, thanks for confirming that!