https://prefect.io
Join Slack
Hey everyone! I am trying to override an environment variable when calling `run_deployment` as shown...
# ask-community
b
Hey everyone! I am trying to override an environment variable when calling 
run_deployment
as shown in screen shot. This doesn't seem to work in my testing. I think maybe its the same issue as here. How are people overriding a single environment variable (not all of them) when creating subflows like this?
r
I always pass environment variable additions/changes/overrides as a dictionary, e.g.
Copy code
{
    "env" : {
        "PREFECT_LOGGING_EXTRA_LOGGERS" : 'blah blah bla'
    }
}
upvote 1
🙌 1
which seems to be the consensus on the issue you linked to too
b
so if I follow your pattern above, will it work as an addition? and any other existing env vars will be fine?
r
should be yep
Without knowing your setup I can't guarantee it but we have quite a few pre-populated environment variables on the images we use and they've never been overwritten
👍 1
b
thanks so much
Ah, although I am not worried about existing ones baked onto an image. I'm worried about those on the work pool / deployment. I will test, but hopefully all good
okay, this doesn't work 😞
other env vars defined in my work pool are lost
as described in this open issue (which doesn't mention 
run_deployment
but I suppose its all the same under the hood)
@Nate: do you happen to know of a workaround for this situation? it makes 
run_deployment
almost unusable for me
n
lemme make sure i have the problem right you want to send extra 
env
to a flow run from a deployment without clearing any 
env
on the work pool so like basically 
env = work_pool_env | run_deployment_env
b
Well, in this case, I am actually overriding one specific env var (GIT_BRANCH) already defined on the workpool. But yes, I essentially want to overwrite or add without having to replace the entire thing
n
Copy code
In [1]: from prefect import get_client

In [2]: async with get_client() as client:
   ...:     wp = await client.read_work_pool(work_pool_name="local")
   ...:

In [3]: work_pool_env = wp.base_job_template.get('variables').get('properties').get('env').get('default')

In [4]: new_env = work_pool_env | {"GIT_BRANCH": "foo"}

In [5]: new_env
Out[5]: {'bar': 42, 'GIT_BRANCH': 'foo'}
so you could write a function like 
get_current_work_pool_env
and then pass 
new_env
to 
run_deployment
as the total 
env
for that run right?
b
So this will work when my subflow called with 
run_deployment
uses the same work pool (and their deployments don't have different env var overrides)
but I guess this won't cover the general case of different work pools?
n
you could just as easily read the 
env
from the deployment itself if it has overrides, since the overrides are accessible on the deployment object so there's likely many ways to do this, but whats most convenient for your use case im not sure
b
okay, fair enough. Thanks for the pointers - I will play around
Actuallly, to take one more shot at this: Is there a way to access the env vars that 
run_deployment
would *use (*If I didn't pass any)? I don't want to assume anything about the deployment in question e.g. I don't want to assume it shares a work-pool with the parent flow
a
We don't have a utility exposed to do this, but the code that workers run to assemble the final set of job variables looks like this: https://github.com/PrefectHQ/prefect/blob/4191a50041ca14b6fe9b27475b37af3cbc3f6261/src/prefect/workers/base.py#L979-L990 That may not be useful to you though
b
ah, okay, nice. I think I can work with that 👍
@Andrew Brookins: the code you've linked here quite clearly displays the bug:
Copy code
job_variables = {**deployment_vars, **flow_run_vars}
this merge should instead be a recursive merge. This actually seems like it could be fixed pretty easily. Do you think there's a chance it will get worked on soon?
a
I agree that a recursive merge would be more useful and easier to use. We’re currently focused on a release candidate, so we probably won’t have capacity to look at this in the next week or so. I’ll huddle with our product folks on scheduling and add any updates to the GH issue when we have them (I’m thinking of #11041).
👍 2
42 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by