https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Adam Roderick

07/10/2020, 12:49 PM
Any idea why my environment variables are missing during a pipeline execution? I'm using Docker storage, and locally I can spin up a container and confirm that the env vars are indeed there
n

nicholas

07/10/2020, 1:24 PM
Hi @Adam Roderick - can you give a little more info about your setup? Are these Prefect-prefixed environment variables?
a

Adam Roderick

07/10/2020, 2:06 PM
No, they are the dictionary I pass into Docker storage
n

nicholas

07/10/2020, 2:48 PM
Got it @Adam Roderick - I've not tried it personally, but I'd imagine those variables still need to be prefixed with
PREFECT__
to be read into the pipeline.
a

Adam Roderick

07/10/2020, 2:50 PM
No, these are variables that are injected into the docker image. It was working until this morning when I redeployed, and I'm not sure what is different
Trying to track it down. The build step even shows that they are applied,
Step 21/22 : ENV DBT_EXECUTABLE=/datateer/venv/dbt/bin/dbt DBT_PROFILES_DIR=...
, and when I spin up a container, I can see them all there when I run
printenv
. But the pipeline doesn't see them when I trigger it from cloud
n

nicholas

07/10/2020, 2:52 PM
Interesting, let me ping someone on this
a

Adam Roderick

07/10/2020, 2:56 PM
Thanks, I'll keep looking too
👍 1
j

josh

07/10/2020, 3:00 PM
@Adam Roderick how are you attempting to access the env vars during your flow run?
a

Adam Roderick

07/10/2020, 3:02 PM
Actually I'm accessing them at build time... that might be the issue.
os.environ['MELTANO_EXECUTABLE']
j

josh

07/10/2020, 3:03 PM
Yeah there’s a chance they aren’t set during build time. Where exactly during the build are you attempting to retrieve them?
a

Adam Roderick

07/10/2020, 3:04 PM
in the constructor of a custom Task
I think it's an order of operations thing. Thanks for checking on it with me
👍 1
2 Views