When setting environment variables from the Run pa...
# prefect-ui
a
When setting environment variables from the Run page, which environments exactly will receive this environment variable? I use a local agent, which then kicks off a Dask cluster on ECS to run the flow. Will the environment variable of the Dask cluster be set with this, or just the environment variables of the local agent?
z
Hi @Andrew Hannigan! Local Agents should pass your all environment variables to the Dask cluster on execution. For the Local Agent, this will include • variables in your local environment • variables passed via the
prefect agent local start
command • variables passed via the Run page or the flow's RunConfig
a
Hmm I’m not seeing them show up at the moment on the dask cluster - do you have any suggestions on how to quickly inspect the environment variables of a dask cluster? Is that available via that dask dashboard?
z
Hmmm I'm not familiar with Dask unfortunately. Do you see any environment variables come through?
This could be a bug, I'll take a closer look at the code here and keep you updated
a
I think it might be. I just tried specifying the environment variables via the actual dask class that creates the cluster and my process that was failing now works.
👍 1
Using the
environment
argument here in the init for ECSCluster I passed the environment variable I need directly to the cluster: https://cloudprovider.dask.org/en/latest/aws.html#elastic-container-service-ecs
Possibly the CloudFlowRunner could update the
environment
field of the ECSCluster object dynamically with the new variables from the cloud UI before it calls the method to actually create the dask cluster
z
Yep I think you're spot on here. Thanks for bringing this up! There are some complications around this (for example, not all dask clients support environment variables in the same way). I'll open an issue for discussion.
👍 1
@Marvin open "Environment variables not passed to dask clusters created by Prefect"