Lukas N.
12/16/2020, 9:55 AM--env FOO=BAR
option. I'm able to spawn "regular" flows as jobs just fine and it will also pass the env variables (as seen here). Now I would like to test out the DaskKubernetesEnvironment and access the FOO
env variable
with Flow(
"whatever",
environment=DaskKubernetesEnvironment(min_workers=1, max_workers=3),
) as flow:
secret = EnvVarSecret('FOO', cast=str, raise_if_missing=True)
and run it using the Kubernetes agent.
Now the process is bit different, the flow run is picked by the k8s agent, it spawns a job (all the same as previously) and the job executes the DaskKubernetesEnvironment that creates another dask job. The problem I have is the FOO
variable is lost in the last step. Kubernetes agent --[env variables]--> Kubernetes job --[no env vars]--> dask job
. Checking the code there is indeed nothing passing env vars.
How should I populate the env vars for dask jobs?Vladimir Zoubritsky
12/16/2020, 1:17 PMexecutor=DaskExecutor(
cluster_class=KubeCluster,
cluster_kwargs={
'pod_template': make_pod_spec(env={'HELLO': 'world'})
}