Hi everyone. I am working on a flow on prefect 1.4 and i am struggling to understand why my environment variables are not passing from one task to another. The flow is as follow: • In an upstream task, I add a couple of environment variable to the environment (os.environ object). I logged the object at the end of the task. and I can see that my new environment variables are loaded • Then in a downstream task, I use one of the environment variable and I get a key error. when logging the os.environ object, none of my new environment variable are there. I am missing something? I will add a screenshot of the schematic in the thread.

Hey @Arthur Jacquemart I think the KV store might actually fit your use case https://docs-v1.prefect.io/orchestration/concepts/kv_store.html#ui

Hi Mason. Thank you for getting back to me. We do not wish to use kv store: we do not want to share our secrets with prefect cloud

Hmm, are these variables being determined dynamically in the task, if not you might try setting them at the Flow Level rather than the task level and see if that works better or you could set them on the agent running the flow in you local execution environment https://docs-v1.prefect.io/orchestration/getting-started/flow-configs.html#configure-environment-variables To be perfectly transparent I'm not 100% sure on why the environment variables would work sometimes and not others, instinctually I think it's tied to the fact that the task run context for the different tasks are technically separate but it seems odd it would work sometimes in that case, either way though I think defining these at the flow level or ideally prior to the flow actually running in your execution environment would be more stable.

yes the variables are determined in the task. With your method i would need to split my flow into two flows I believe. I will store my variables in a dictionary to pass it from one task to an other. Its a weird behaviour