Joshua Greenhalgh
05/21/2022, 4:22 PMDEV_TAG = os.environ.get("DEV", "") != ""
JOB_IMAGE_NAME = f"blah/flows{':dev' if DEV_TAG else ''}"
And then in each flow I ref the JOB_IMAGE_NAME
- this just changes the image but otherwise uses the job template I have defined on the agent;
apiVersion: batch/v1
kind: Job
spec:
template:
spec:
containers:
- name: flow
imagePullPolicy: Always
env:
- name: SOME_ENV
valueFrom:
secretKeyRef:
name: secret-env-vars
key: some_env
optional: false
Now when I specify the dask setup I do the following;
executor=DaskExecutor(
cluster_class=lambda: KubeCluster(make_pod_spec(image=JOB_IMAGE_NAME)),
adapt_kwargs={"minimum": 2, "maximum": 3},
)
But this is obviously missing the env part of my default template - I would like to not have to respecify it (its much bigger then the above snippet) - is it possible to grab a handle on the default template and just override the image name?Anna Geller
Joshua Greenhalgh
05/21/2022, 5:52 PMAnna Geller
Joshua Greenhalgh
05/22/2022, 1:12 PMDASK_POD_SPEC = make_pod_spec(
image=JOB_IMAGE_NAME,
env={
"SECRET_ENV_VAR": os.environ['SECRET_ENV_VAR'],
},
)
DASK_POD_SPEC.spec.service_account_name = "flow-user"
Since the make_pod_spec
doesn't allow to set the service account you want to run asAnna Geller
Joshua Greenhalgh
05/23/2022, 8:06 AMservice_account_name
is needed so that I am able to access GCP resources - KubeCluster
seems to default to using the default
service accountAnna Geller