https://prefect.io logo
Join Slack
Powered by
Hi everyone, I keep running into issues trying to ...
# ask-community
r
Hi everyone, I keep running into issues trying to load my own 
config.toml
 . I've tried many things already so I'd your input since I might be overlooking something. To keep everything together, I prefer to have the 
config.toml
 in the same git folder as my flows ( (
base/src/config.toml
 ). In 
base/.env
  there are a bunch of environment variables, among others 
PREFECT__USER_CONFIG_PATH
 . Initially this worked on my local machine and the things defined in the config file were used in the flow, both on Server and when I switched to Cloud. But now I cloned the repo to a different machine and it fails to import the config.toml. Strangely enough, when I output the value of  
PREFECT__USER_CONFIG_PATH
 to the log, it is correct. More info in the thread. Is there a better way to do this, or am I forgetting something?
file structure
Copy code
base/
--.env
--src/
----api.py
----config.toml
----run_agent.py
base/.env
Copy code
PREFECT__USER_CONFIG_PATH=config.toml
...
base/src/run_agent.py
Copy code
import socket
from pathlib import Path
from dotenv import dotenv_values
from prefect.agent.local import LocalAgent

hosts = {
    '***': r'C:\users\***\base\src'
}
hostpath = Path(hosts.get(socket.gethostname(), '.'))
env_vars = dotenv_values()
env_vars['PREFECT__USER_CONFIG_PATH'] = str(hostpath / 'config.toml')
LocalAgent(labels=['agenttest'],
           env_vars=env_vars,
           import_paths=paths,
           show_flow_logs=True
           ).start()
base/src/api.py
Copy code
import os
import prefect
from prefect import task, Flow
from prefect import config as pc

@task(name='Check last date of entries in database')
def check_last_date():
    logger = prefect.context.get('logger')
    <http://logger.info|logger.info>(f'config: {os.environ.get("PREFECT__USER_CONFIG_PATH")}')
    if pc.db_to_use != 'SQLite':
	    ...

with Flow(name='Fetch data from API',
          run_config=UniversalRun(labels=['agenttest'])
          ) as api_flow:
    last_date = check_last_date()
base/src/config.toml
Copy code
db_to_use = 'MSSQL'
...
Running on Windows 10 with Python 3.9. I know officially there's no support yet for 3.9, but that can't be it right? Also, I've reregistered the flow many times. Every time the flow fails because 
db_to_use
is not found in config.toml
k
Hey @Rinze, I can’t see anything wrong immediately here and this looks like the way to go, and it shouldn’t be Python 3.9. I know the environment variable is being set, but could you try printing 
prefect.config.user_config_path
to see if that gives the same value?
You have this part of the agent:
Copy code
hosts = {
    '***': r'C:\users\***\base\src'
}
I am assuming you changed that right? Since it seems you loaded the correct env variable.
What gets logged? 
str(hostpath / 'config.toml')
or 
config.toml
?
e
I think setting the env var value within python, and after importing anything from prefect can be the issue here. This issue feels very familiar 😅
👍 1
r
Hi Kevin, thanks for the quick reply. Yes the hosts dictionary was edited for privacy. I tried switching the logging of the environment variable to 
prefect.config.user_config_path
, but it's the same output
The full path is logged
Thanks emre, did you experience similar issues? Is it because of the parsing at runtime perhaps?
One solution might be to use the standard 
~/.prefect/config.toml
, but that will take it out of VCS
k
Maybe you can try populating that standard 
config.toml
with different values and seeing if your Flow loaded the content of it?
r
OK, I thought that would work, too. However it doesn't and I get the same error.
Copy code
File "C:\Users\rjcdo\Documents\Werk\datafuture\code\pipeline\venv\lib\site-packages\box\box.py", line 501, in __getattr__
    raise BoxKeyError(str(err)) from _exception_cause(err)
box.exceptions.BoxKeyError: "'Config' object has no attribute 'db_to_use'"
e
@Rinze not me, but someone in slack had a similar issue. The thread is lost to slack retention sadly. The issue was that prefect attempts to read from 
PREFECT__USER_CONFIG_PATH
exactly once in the runtime (when importing 
prefect.configuration
) https://github.com/PrefectHQ/prefect/blob/64acf232d2ffe733c975797b4144bd8237fb57e9/src/prefect/configuration.py#L14 What this means is that changing the env var value after importing 
prefect.configuration
, or 
import prefect
, has no effect, prefect has already tried to read 
PREFECT__USER_CONFIG_PATH
, couldn't find a value, and used the default config.toml
👀 1
don't know if this helps, but this snippet reminded me of the issue i just mentioned
Copy code
env_vars = dotenv_values()
env_vars['PREFECT__USER_CONFIG_PATH'] = str(hostpath / 'config.toml')
r
Funnily enough I did have it working before when I loaded the env dictionary at the definition of the flow. However, when that failed somewhere along the struggle I tried setting importing the env vars straight into the starting of the agent. I would think that in that case the environment variables should be available even before prefect starts looking at the flow at all?
I did run into issues similar like in the link you provided, when I tried to load with 
load_dotenv()
at the top of the file before. Although always a bit of a mind bender, I am now familiar with the runtime concept
e
yeah, your old local machine having the correct env var set up all along would explain your issue imo.
👍 2
marvin 1
r
OK, very strange. So I copied the 
src/config.toml
to 
~/.prefect
. Like I said before it didn't work, but now I added some text to the logging message and reregistered. Now 
db_to_use
is read. I then removed 
~/.prefect/config.toml
and it still works 🤯
Really not sure why though
k
That is strange. Maybe it was already loaded by the agent? Did you restart the agent even if you reregistered?
upvote 1
r
ok scrap that. Must have been a cache thing or something because now it only works with the file in 
~/.prefect
(as expected)
So is the conclusion that it's just not really possible to load a config.toml by reference to an environment variable?.
Could the problem also be the way I import the values from the config?
Copy code
from prefect import config as pc

if pc.db_to_use etc etc
k
Yeah I think it could be that. I feel that 
import config as pc
might be checking environment variables, but the environment variables loaded by dotenv are only in the Python process . I Ignore this.
I’ll have to check with the team to get a better understanding of this and get back to you later today.
r
Many thanks for your help either way @Kevin Kho 👏
e
I' m trying to get a better understanding of the situation Can you try restarting your agent in the following way, and see if your problem is fixed? On a clean terminal (no value for env var 
PREFECT__USER_CONFIG_PATH
, change your run_agent.py as follows:
Copy code
import socket
from pathlib import Path
from dotenv import dotenv_values

hosts = {
    '***': r'C:\users\***\base\src'
}
hostpath = Path(hosts.get(socket.gethostname(), '.'))
env_vars = dotenv_values()
env_vars['PREFECT__USER_CONFIG_PATH'] = str(hostpath / 'config.toml')

from prefect.agent.local import LocalAgent

LocalAgent(labels=['agenttest'],
           env_vars=env_vars,
           import_paths=paths,
           show_flow_logs=True
           ).start()
I moved the prefect import LocalAgent part below your envvar setup.
using 
os.environ
instead of 
dotenv_values
helped me overwrite environment variables, this snippet seems to start the localagent with the correct 
PREFECT__USER_CONFIG_PATH
, but the dotenv_values version didn't have any effect. Probably 
dotenv_values
returns a static copy of yout 
.env
state, instead of the actual effective envvars.
Copy code
import os

os.environ["PREFECT__USER_CONFIG_PATH"] = str("config.toml")
from prefect.agent.local import LocalAgent

LocalAgent(labels=["agenttest"]).start()
r
Thanks @emre. I've tried your solutions and a few other options as well. This is what I've found so far: • moving the import alone (your option 1): no • using os.environ just for `PREFECT__USER_CONFIG_PATH`: yes! • using 
PREFECT__USER_CONFIG_PATH
in os.environ + 
env_vars
in agent definition: no • using 
load_dotenv()
(instead of 
dotenv_values()
) and `env_vars`: no • 
dotenv_values()
and put it in `os.environ`: no • combination of previous and 
PREFECT__USER_CONFIG_PATH
in `os.environ`: no I've come to the conclusion that any time you load anything in 
env_vars
, the normal environment is disregarded.
The odd thing is, when I only load it in the general environment, it still isn't fully used. So now I'll just go with the one thing that works and save the remaining things from the 
.env
file in Cloud. Still interested if anyone finds a working solution!
Copy code
SESSIONNAME: Console
PREFECT__USER_CONFIG_PATH: config.toml
PREFECT__LOGGING__LEVEL: DEBUG
GIT_TOKEN: ***
PREFECT__CONTEXT__SECRETS__***: ***
PREFECT__CONTEXT__SECRETS__***: ***
PREFECT__CONTEXT__SECRETS__***: ***
PREFECT__CONTEXT__SECRETS__***: ***
PREFECT__CONTEXT__SECRETS__***: ***
[2021-07-21 19:48:31,393] INFO - agent | Registering agent...
[2021-07-21 19:48:31,573] INFO - agent | Registration successful!
And still unable to load the 
config.toml
9 Views
Open in Slack
PreviousNext
Powered by