What is the best way to pass several environment v...
# prefect-community
p
What is the best way to pass several environment variables to a docker agent? I have them in a
.env
file but I don't see a way to pass this file in the cli. Alternatively, how do I tell the agent to use my
config.toml
? I have one at
~/.prefect/config.toml
that is used when using
flow.run()
Can I tell the agent running on the same machine to use this config file?
n
Hi @Pedro Machado ! You can pass environment variables to the agent from the CLI using the
-e
or
--env
flags as detailed here in the docs.
p
I have 12 of them. I was wondering if there was a way to pass them via a file. What about the context approach? How can I pass context to the docker agent?
n
Hm I don't think they're passable via a file but perhaps context is a good way to go because you can read the variables you'd like into the context at build time
p
Is it possible to set the context in a config.toml for the agent to use?
n
So I think it's important to note the agent's purpose in Prefect, which is as an orchestrator/manager for disparate, independent workflows. As such, it usually doesn't make sense to pass flow-specific context to an agent, but instead to define them on the flow's context or environment directly. For example, you can build a flow with docker storage that contains and
env_vars
dict in which you can pass your desired environment variables from the build context
p
I see what you are saying. I could pass
env_vars
at build time for flow-specific configuration. What do you recommend for things like db credentials? I am currently passing them to the agent as env variables and reading them via
EnvVarSecret
. Is this the optimal way if we don't have access to a secret store? Do you have any recommended patterns to set up the code so that it runs locally with
flow.run()
and in a docker container? Unfortunately, we'll have to use the local file system for results and output and the paths are different than in the local environment. I got it to work using env variables like
PREFECT_BASE_DIR
which points to the directory where the flows to read some sql templates and
PREFECT__CONTEXT__LOCAL_RESULTS_DIR
to specify where the results will be written. Does this sound reasonable or can you think of a better approach?
n
Definitely @Pedro Machado -
EnvVarSecret
is the way to go in that case. You can always set environment variables during your docker's build step as well.
flow.run
will run in your local process so for that to happen you'd need to call that from within the container itself. Otherwise that sounds like a viable approach