Hello all, we are using a .env file for secret management with prefect instead of the config.toml and are running into issues with injecting updated .env values at runtime with LocalRun(env=env_dict). The only way that the flow picks up the new .env contents is upon registration of the flow, which is not ideal as we need the .env file to be written to during runtime and the new values read from the same file. Any pointers on how to have Prefect pick up the new values? The flow creates a container and injects env values into it with

enviornment=env_vars

in the CreateContainer() function but we need to have it inject updated values at runtime. I have had success with

flow.run_config = DockerRun(env={f"PREFECT__CONTEXT__SECRETS__{k}": v for k, v in config.context.secrets.items()})

but we are not utilizing prefect's config.toml for this project as well as the flow not being executed within a docker container

Hey @Matthew Blau -- sorry about the delay, it's been a busy day for me. I'm not sure there's a straight-forward way to load the environment at runtime like this. We could consider making a change such that the

env

can be a callable that the agent will run at execution time but that has a lot of other implications. Have you considered setting the secrets on the agent instead of the run config? I think you could also probably modify an agent to load your .env file (ie refresh the environment) each time it deploys a flow but I don't think we'd want to introduce something like that to the mainstream -- generally the environment is loaded once at process start only.

Hey @Zanie I understand how those days go! Talking internally we have decided that we need a better way overall of managing secrets and have turned to look at Hashicorp Vault as a solution, which a cursory glance indicates that this should solve our specific use case

I don't know if that help. I added recently a kubernetessecret task https://docs.prefect.io/api/latest/tasks/kubernetes.html#replacenamespacedservice that allow to retrieve on kuberbetes the secret at run time when you run your own perfect server instance. This could work at well with vault which can generate kuberbetes server object