https://prefect.io logo
Join Slack
Channels
Powered by
Hey Team, i'm having an issue where my jobs are fa...
# ask-community
c
Hey Team, i'm having an issue where my jobs are failing to pick up "PREFECT__USER_CONFIG_PATH" environment variable that i've preset on a base docker image. But when i bash into the docker container while running, it works fine with flow.run() .. Can you help me understand what happens when a flow is deployed? We're currently using a modified gitlab storage that recursively downloads all files into the home directory, one of which is .prefect/config.toml. For example, whats the order of operations? My hypothesis is that the flow tries to run globals first, and then run the gitlab storage to get the flows, then follow tasks to execute? For more details, i'm trying to make the run_config assigned to the flow configurable through the config.toml file 
# Registration Configs
flow.run_config = NomadRun(**prefect.config.run_config)
flow.storage = GitLab(path='flow.py', **prefect.config.storage)
k
Hey @Charles Leung, is this a duplicate of the server post? No need to double post, we’ll see it 🙂. Could you delete that one? The agent polls our database, loads the flow from storage, and then loads the Docker container and runs the Flow on top. Could you explain the custom storage? I don’t think we would respect a custom storage because we won’t know how to load the flow.
Thanks for deleting btw
c
ah yes i realized i posted it in the wrong place since i was using cloud.
the custom storage is pretty much the same gitlab storage, but i changed it instead of just fetching the flow file, to recursively loop through the gitlab tree and pull every file. I didn't PR it though since from what i read on Issues is that it's not the intended way prefect wants to use moving forward. Problem is we have alot of relative imports in some of our code so it was an adhoc solution
In a test i just did, it actually does recognize the PREFECT__USER_CONFIG_PATH environment variable, but for some reason the prefect.config isn't loading it
may I ask when in the process is the prefect.config loaded?
k
Still thinking through this, but I’ll answer that first. When your flow is loaded from storage and ran, the 
import prefect…
is the one that instantiates the config and it’s only loaded once.
I know for sure environment variables present in the container with ENV are respected. When you say 
prefect.config
isn’t loading it, are you referring to the contents of the file or the env variable itself? Because I’m expecting the env variable to be respected, but for the file to not be present
c
the contents, i'm getting an error for example with a vault_url key i have in my config.toml. I can see the file exists, and the env is respected. but the config in python code is failing
k
A bit confused how you know the file exists. Is that by checking the container after the Flow fails? Or are you able to open it and can the contents inside a Flow run?
c
oh i edited my flow to print out the file contents
oddly enough, when i manually overwrite the config (just literally copying and pasting w/ minor changes), it works.:
Copy code
if 'vault_url' not in prefect.config.keys():
    import re
    print('Env:\n')
    print(os.getenv('PREFECT__USER_CONFIG_PATH'))
    print('\nConfig:\n')
    print(prefect.config)
    print('\nFile:\n')
    print(open(os.getenv('PREFECT__USER_CONFIG_PATH')).read())


    # load prefect configuration
    DEFAULT_CONFIG = os.path.join(os.path.dirname(prefect.configuration.__file__), "config.toml")
    USER_CONFIG = os.getenv("PREFECT__USER_CONFIG_PATH", "~/.prefect/config.toml")
    BACKEND_CONFIG = os.getenv("PREFECT__BACKEND_CONFIG_PATH", "~/.prefect/backend.toml")
    ENV_VAR_PREFIX = "PREFECT"
    INTERPOLATION_REGEX = re.compile(r"\${(.[^${}]*)}")

    config = prefect.configuration.load_configuration(
        path=DEFAULT_CONFIG,
        user_config_path=USER_CONFIG,
        backend_config_path=BACKEND_CONFIG,
        env_var_prefix=ENV_VAR_PREFIX,
    )

    config = prefect.configuration.process_task_defaults(config)

    print('\nReloaded:\n')
    print(config)
    prefect.config = config
for reference, i'm on prefect 0.14.11 - i can't really update it since i'm using a custom RunConfig as well. will be making a PR in the near future regarding this.
k
Ok. I guess that works. You good now?
c
yep! i guess so though it's not the best behavior 🙂 but it'll do
đź‘Ť 1
Open in Slack
PreviousNext
Powered by