Hi, I have a question that is a follow-up from one question I asked on April 19th. I managed to deploy a prefect server and to run flows with docker storage and a DaskKubernetes environment. The only hurdle I have left (I hope!) is that an important part of our Task depend on some common configuration set on

prefect.context

. This includes some secrets and some URLs and object ids on some internal rest services. Before we used the prefect server, we had a small cli that would populate the

prefect.context

and use it when running the flow:

with prefect.context(**our_custom_vars):
    flow_state = flow.run(parameters=flow_parameters, ...)

My ultimate objective is to have a flow with a default context that I can run from the UI or schedule it with that context when deploying it. In my question of April 19th, @Jeremiah pointed out that there is no way to set these contexts at the moment and you will discuss it internally later… is there any update on this front? Otherwise, I am looking for alternatives or workarounds: One workaround would be to understand where the context would be set on an agent, worker, or runner (I am not sure which one). Jeremiah also pointed our that the FlowRunner requests the context from the server, but I can’t find where or even if a flow has its context saved anywhere (there does not seem to be a field named context on the

flow_by_pk

query). Another workaround would be to populate environment variables or change the default

config.toml

of the agent, worker, or runner (I am not sure which one) so that the

prefect.context

is populated with these values. I am not sure if this would work. Another workaround may be to override the

__setstate__

and

__getstate__

method, so that the flow can retrieve the context when unpickled. I am not sure if this would work either. Any ideas on which of these workarounds may be the best bet here?

Hey David, first up I have a quick workaround for you which is a more flexible version of your second idea. (And I apologize, I totally forgot we support what I’m about to describe or I would have mentioned it sooner). You can do exactly what you want via env var:

$ PREFECT__CONTEXT__CUSTOM__VALUE=5 python -c "import prefect; print(prefect.context['custom']['value'])"

5

No need to apologize Jeremiah, you are all already doing a great job…

For a more “first class” approach at this, we are exploring introducing runtime callables to allow flows (and clocks!) to declare default contexts and environments, similar to how clocks can declare parameter defaults today.

Well, we do agree and we are not using context as a mutable object, I’ll tell you later our use cases though…

Hm, I think I may have to tag @josh in to help with setting env vars from the agent

My agent is a copy/paste of your src/prefect/agent/kubernetes/deployment.yaml with the following change:

env:
   ...
   - name: PREFECT__CONTEXT__FOO
     value: 'fofofofofofofofof'

Agents accept a dictionary of environment variables at runtime:

- env_vars (dict, optional): a dictionary of environment variables and values that will be set on each flow run that this agent submits for execution

However I don’t think that would satisfy the use case of an installed agent being able to set env vars on all runs. This would be a great enhancement 🙂

Ok let met try that. In my case, fortunately, I can live with the same context for the sole agent that we have.

In the meantime as a workaround you could edit your agent deployment yaml command to add the environment variables:

args: ["prefect agent start kubernetes"]

to

args: ["prefect agent start kubernetes -e MY_VAR=test -e MY_VAR2=test2"]

Ok thanks for the info Josh. I tried a deployment with

args: ["prefect agent start -v kubernetes -e PREFECT__CONTEXT__FOO=bar123"

and I can see that the k8s job does have this as a environment variable when I do

kubectl describe job …

...
    Environment:
      PREFECT__CLOUD__API:                          <http://apollo:4200>
      PREFECT__CLOUD__AUTH_TOKEN:
      PREFECT__CONTEXT__FLOW_RUN_ID:                98c1491c-d0dc-4885-90df-f84cb702d36b
      PREFECT__CONTEXT__NAMESPACE:                  default
      PREFECT__CLOUD__AGENT__LABELS:                []
      PREFECT__LOGGING__LOG_TO_CLOUD:               true
      PREFECT__CLOUD__USE_LOCAL_SECRETS:            false
      PREFECT__LOGGING__LEVEL:                      DEBUG
      PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS:  prefect.engine.cloud.CloudFlowRunner
      PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS:  prefect.engine.cloud.CloudTaskRunner
      PREFECT__CONTEXT__FOO:                        bar123
...

However, the dask pods do not inherit these environment variables, so the context is still missing the

foo

variable

Haha the lowest tech approach always wins