https://prefect.io logo
r

Rajdeep Rao

05/04/2021, 11:18 PM
Hey y'all! I have kind of a niche question. I was wondering if there's any way I could have prefect run my flows as
/bin/sh -c ./bash-script.sh prefect execute flow-run
? I would like to run this bash script that basically exports all my env vars that are in a file. Since we use dot-env to run our services, my base docker image already has all my env vars I just need to export them while my tasks spin up. I tried overriding them with run-task-kwargs, in the ECSRunconfig() and by passing in a custom task_def, but for some reason my tasks don't respect these
k

Kevin Kho

05/05/2021, 1:27 AM
Hi @Rajdeep Rao , maybe try passing them to
env
in the ECSRunConfig . But also, you can make your python script register the flow and invoke it immediately with the
Client
. Then you can do
python flow.py
. If you don’t need to register, you can just have the client start it. Look for create_flow_run .
r

Rajdeep Rao

05/05/2021, 1:36 AM
The thing is it's a file with a lot of env vars. I wanted to avoid making them all individual env vars and feeding them to the flow in the
env
section. I know AWS allows passing env files. Something like that would've helped my case.
We also kinda wanted to avoid putting them up on the secrets tab in the cloud because if these values ever change, we'd need to remember to fix them in multiple places. Driving these values through env vars is kind of nice pattern for us for that reason
k

Kevin Kho

05/05/2021, 1:47 AM
I see maybe having the python script start the flow would help you
r

Rajdeep Rao

05/05/2021, 1:53 AM
I'm sorry I don't exactly follow. How would I pass the env vars?
k

Kevin Kho

05/05/2021, 2:00 AM
I was thinking of using
load_dotenv()
inside your Flow script since the .env file is in the Docker container. Then you could maybe pass them as parameters, but I haven’t used dotenv so I’m not entirely sure if this would work.
Or you can do it inside a Task maybe.
If you use DockerStorage, you might need
stored_as_script=True
r

Rajdeep Rao

05/05/2021, 2:36 AM
Ah, that was sorta my last resort
I wanted to avoid making them all individual env vars and feeding them to the flow in the 
env
 section
Just because it'd be a cleaner experience for the folks that build these jobs. And because this would be something that would be needed to be done before every flow/task which is a little unideal. Esp since they're going to be the same across all of them
fwiw, if there's any interest in having flows consume an env file, I'm more than happy to help contribute in that space
😃 1
k

Kevin Kho

05/05/2021, 2:55 AM
For sure, if you open an issue for it, the core team can see it and comment on it.
👌 1
2 Views