Hi, I wonder whether it is possible to set dynamic context / environment variables during flow runs? (This context is only known at run time)
k
Kevin Kho
09/30/2021, 2:04 PM
Hey @Dan Zhao, how big is it? Maybe you can use the KV Store to store it? There is a 10 KB limit, maybe you can hold it there. Are parameters not a good use for this?
d
Dan Zhao
09/30/2021, 2:15 PM
it's just some strings and integers - but it's related to the executing environment (e.g. available RAM)
k
Kevin Kho
09/30/2021, 2:20 PM
Do you have access to it in the Python process? And you want to load them and then add them to context?
d
Dan Zhao
09/30/2021, 2:21 PM
yes to both questions.
k
Kevin Kho
09/30/2021, 2:23 PM
Gotcha so not 100% this will work but maybe you can store the Flow as a script, and then at the top, do
Copy code
get_env_vars_somehow()
os.environ = new_env_vars #using the context
import prefect
so that when prefect gets imported and the context gets created, the env vars will exist. does that make sense? kind of a hack though and I think the KV Store might be the cleaner approach
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.