Question about environment variables and context. When I want to trigger a flow from the UI I can set environment variables for the run, when I want to trigger the same flow from the CLI can I also pass those environment variables?

Do you want to modify Prefect-specific env variables or your custom ones? When you use UI, you are modifying the env variables of the specific run configuration. When triggering flow run from the CLI, there is currently no option to modify run configuration from prefect run CLI. But you also asked about context, and key-value pairs can be passed to Prefect context using the

--context

flag.

That answers my question. 🙂 (custom ones that I set defaults to in the run_config) A follow-up then: If my flow/tasks use a 12factor.net library dependency we would need to set environment variables to administer those libraries. How would set those with

context

f.e. from the CLI? Also with environment variables I can provide defaults in the run_config which the user can then override. Can I also provide a default context dictionary?

@Alexander van Eck what is your set up in terms of agents, storage and run configs? I wonder whether context is the right place to set this information. Do you need to know those library dependencies at build time or at runtime? Because the context gets populated at runtime and the dependencies are typically installed at build time (when you deploy/register your flows)

We have a task that calls out to a REST service to start some compute and waits for it to complete the computation. The amount of GPUs this compute uses can be configured with an environment variable e.g. COMPUTE_GPUS: 1 is the default in UI but if the compute is heavy we’ll scale this up to 8 or more. From the UI this is no problem 🙌 with environment variables but with the CLI I cannot do this. So perhaps switching over to context variables instead makes it possible to do this. I can provide those in UI & CLI. However; 2 problems. 1. The context in the UI is always empty, can it be set to a default? otherwise our users would have to guess the context keys. 2. How do I take a context variable and then set it in the flow as an environment variable? since it would still need to be an environment variable for the REST call.

You can’t edit the RunConfig with the CLI but it might not be too hard to PR in if you are interested. What do you think of having this as an env variable on the agent side? You can then add a label on the flow run side to hit the label with the right env variable.

Are you making your cluster on the fly?

yes

If it needs to be an environment variable for the REST call, you can still temporarily set it with a task with 

os.environ['key'] = value

 right?

yes that’s what I was thinking.

What do you think of having this as an env variable on the agent side?

Sure but then we’d need an agent for every possible GPU configruation. (1gpu, 2gpus, 3gpus) depending on the users need. 🤔

You can’t edit the RunConfig with the CLI but it might not be too hard to PR in if you are interested.

I think this would be very useful as it’s possible through the UI already. 🙂 How do I contribute to this?

Ok I thought about this more, and here are my thoughts: 1. Passing the entire RunConfig through CLI might be hard to work with (it seems doable coding-wise, but you're gonna end up typing a lot as a user). It might be doable, but I think we're reaching for the hard way here. 2. I agree you the multiple agent s is pretty bad. 3. I think you should either use the context like Anna suggested or a Parameter and use a task to set that. This seems to be the easiest thing. The Parameter seems to be the more organic thing as you get defaults in the UI. Any reason you are opting not to use Parameters here?

The parameters is interesting 🙂 I’ll have a think about that.