Rajdeep Rao

    Rajdeep Rao

    1 year ago
    Hey y'all! I have kind of a niche question. I was wondering if there's any way I could have prefect run my flows as
    /bin/sh -c ./bash-script.sh prefect execute flow-run
    ? I would like to run this bash script that basically exports all my env vars that are in a file. Since we use dot-env to run our services, my base docker image already has all my env vars I just need to export them while my tasks spin up. I tried overriding them with run-task-kwargs, in the ECSRunconfig() and by passing in a custom task_def, but for some reason my tasks don't respect these
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Rajdeep Rao , maybe try passing them to
    env
    in the ECSRunConfig . But also, you can make your python script register the flow and invoke it immediately with the
    Client
    . Then you can do
    python flow.py
    . If you don’t need to register, you can just have the client start it. Look for create_flow_run .
    Rajdeep Rao

    Rajdeep Rao

    1 year ago
    The thing is it's a file with a lot of env vars. I wanted to avoid making them all individual env vars and feeding them to the flow in the
    env
    section. I know AWS allows passing env files. Something like that would've helped my case.
    We also kinda wanted to avoid putting them up on the secrets tab in the cloud because if these values ever change, we'd need to remember to fix them in multiple places. Driving these values through env vars is kind of nice pattern for us for that reason
    Kevin Kho

    Kevin Kho

    1 year ago
    I see maybe having the python script start the flow would help you
    Rajdeep Rao

    Rajdeep Rao

    1 year ago
    I'm sorry I don't exactly follow. How would I pass the env vars?
    Kevin Kho

    Kevin Kho

    1 year ago
    I was thinking of using
    load_dotenv()
    inside your Flow script since the .env file is in the Docker container. Then you could maybe pass them as parameters, but I haven’t used dotenv so I’m not entirely sure if this would work.
    Or you can do it inside a Task maybe.
    If you use DockerStorage, you might need
    stored_as_script=True
    Rajdeep Rao

    Rajdeep Rao

    1 year ago
    Ah, that was sorta my last resort
    I wanted to avoid making them all individual env vars and feeding them to the flow in the 
    env
     section
    Just because it'd be a cleaner experience for the folks that build these jobs. And because this would be something that would be needed to be done before every flow/task which is a little unideal. Esp since they're going to be the same across all of them
    fwiw, if there's any interest in having flows consume an env file, I'm more than happy to help contribute in that space
    Kevin Kho

    Kevin Kho

    1 year ago
    For sure, if you open an issue for it, the core team can see it and comment on it.