David Ojeda04/27/2020, 2:18 PM
. This includes some secrets and some URLs and object ids on some internal rest services. Before we used the prefect server, we had a small cli that would populate the
and use it when running the flow:
My ultimate objective is to have a flow with a default context that I can run from the UI or schedule it with that context when deploying it. In my question of April 19th, @Jeremiah pointed out that there is no way to set these contexts at the moment and you will discuss it internally later… is there any update on this front? Otherwise, I am looking for alternatives or workarounds: One workaround would be to understand where the context would be set on an agent, worker, or runner (I am not sure which one). Jeremiah also pointed our that the FlowRunner requests the context from the server, but I can’t find where or even if a flow has its context saved anywhere (there does not seem to be a field named context on the
with prefect.context(**our_custom_vars): flow_state = flow.run(parameters=flow_parameters, ...)
query). Another workaround would be to populate environment variables or change the default
of the agent, worker, or runner (I am not sure which one) so that the
is populated with these values. I am not sure if this would work. Another workaround may be to override the
method, so that the flow can retrieve the context when unpickled. I am not sure if this would work either. Any ideas on which of these workarounds may be the best bet here?
$ PREFECT__CONTEXT__CUSTOM__VALUE=5 python -c "import prefect; print(prefect.context['custom']['value'])" 5
it will be parsed into your config and from there into context (including any nesting you indicate with other
separators, and a best effort to parse ints, floats, and bools
David Ojeda04/27/2020, 2:27 PM
David Ojeda04/27/2020, 2:30 PM
David Ojeda04/27/2020, 2:32 PM
env: ... - name: PREFECT__CONTEXT__FOO value: 'fofofofofofofofof'
josh04/27/2020, 2:36 PM
However I don’t think that would satisfy the use case of an installed agent being able to set env vars on all runs. This would be a great enhancement 🙂
- env_vars (dict, optional): a dictionary of environment variables and values that will be set on each flow run that this agent submits for execution
David Ojeda04/27/2020, 2:36 PM
josh04/27/2020, 2:38 PM
args: ["prefect agent start kubernetes"]
args: ["prefect agent start kubernetes -e MY_VAR=test -e MY_VAR2=test2"]
David Ojeda04/27/2020, 2:51 PM
and I can see that the k8s job does have this as a environment variable when I do
args: ["prefect agent start -v kubernetes -e PREFECT__CONTEXT__FOO=bar123"
kubectl describe job …
However, the dask pods do not inherit these environment variables, so the context is still missing the
... Environment: PREFECT__CLOUD__API: <http://apollo:4200> PREFECT__CLOUD__AUTH_TOKEN: PREFECT__CONTEXT__FLOW_RUN_ID: 98c1491c-d0dc-4885-90df-f84cb702d36b PREFECT__CONTEXT__NAMESPACE: default PREFECT__CLOUD__AGENT__LABELS:  PREFECT__LOGGING__LOG_TO_CLOUD: true PREFECT__CLOUD__USE_LOCAL_SECRETS: false PREFECT__LOGGING__LEVEL: DEBUG PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudFlowRunner PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudTaskRunner PREFECT__CONTEXT__FOO: bar123 ...
storage for my flow