Thread
#prefect-community
    b

    Brian Mesick

    1 year ago
    Hi folks. We’re still trying to move to github storage from Docker, but have run into a new snag. It seems like our toml config files are no longer being picked up. We’ve always configured them at registration time as
    PREFECT__USER_CONFIG_PATH=prod.toml
    but
    prefect.config
    no longer seems to have the values from the toml file. Any idea how we can work around this?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Brian Mesick, did you store the toml inside the Docker image?
    b

    Brian Mesick

    1 year ago
    It’s unclear to me where we’d do that. We’ve been using the default Prefect images not building our own dockerfiles. Is it not possible to pull it in alongside the flow script?
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh I was just asking where the tomy config files live. Are they on the agent already?
    b

    Brian Mesick

    1 year ago
    The live in github alongside the flow python files.
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @Brian Mesick -- I think current best practice here would be to • Set all the config values in the
    RunConfig
    via environment variables or • Copy your config into a docker image based on the prefect base image then specify that as your image via a
    DockerRun
    config
    We're exploring support for copying things alongside your flow script in the
    Git
    storage class (distinct from
    GitHub
    ) but we're not sure what the best way to make that work with all of the different execution environments is
    b

    Brian Mesick

    1 year ago
    Yeah, that’s pretty rough since we went to this specifically to avoid having to make the image changes so frequently and this loses the ability to configure things per-environment without re-baking the image. On the upside we’re not currently doing that anyway so we can probably set these things as env vars in the
    KubernetesRun
    Michael Adkins

    Michael Adkins

    1 year ago
    Hmm. I will set a reminder for myself to dig into this later today if I have time or tomorrow and I'll see if there's a good way to handle this.
    Re the run config env var method though: you can probably generate the environment variables from your prod.toml file pretty easily as well so you can still spec them out that way.
    b

    Brian Mesick

    1 year ago
    Thanks Michael
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey Brian, I haven't had time to look into this yet and I'm going to open an issue on Github so it doesn't get forgotten.
    Internally, we use environment variables so I think that's probably the most feasible way for now. @Marvin open "Document recommended pattern to pass Prefect config via RunConfig"
    Marvin

    Marvin

    1 year ago