https://prefect.io logo
Join Slack
Powered by
Prefect profile: changing profile in one virtual e...
# prefect-community
m
Prefect profile: changing profile in one virtual environment is reflected in other virtual environments on the same machine. I'm confused about using profiles. Details in thread.
✅ 1
I maintain virtual environments for my prefect projects, as I develop them for packaging elsewhere. I'm confused because changes are spanning across shell virtual environments on my local machine: open a terminal
Copy code
cd /path/to/prefect-project-A
poetry shell    # (activates the virtual env)
prefect profile use 'dev'
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│               prod  │
│             *  dev  │
└─────────────────────┘
now open another terminal
Copy code
cd /path/to/prefect-project-B
poetry shell   # (activate this different virtual env)
prefect profile use 'prod'
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│             * prod  │
│                dev  │
└─────────────────────┘
but now go back the first terminal window, and look at the active profile.
Copy code
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│             * prod  │
│                dev  │
└─────────────────────┘
So changes to the active profile in any session or virtual environment are global. Are there steps I can take to isolate these?
j
Hi merlin, a prefect profile is just an api url and api key that can be set in your 
profiles.toml
file within the .prefect directory. When you set your profile the 
active
profile within the 
profiles.toml
file also changes
It wouldn't be specific to your virtual env, as it is a config set outside of it
🙏 1
z
Profiles can contain any settings, not just API key and URL — they are global based on the user though.
You can change the profile per virtual environment by setting the 
PREFECT_PROFILE
environment variable when entering an environment.
upvote 2
snap point 2
m
Thanks James! ahhh exactly the detail I was missing. that explains it. So... posting a relevant doc page for others coming to this, an environment variable will override the profile settings of the same name. In my case its gotta look like:
Copy code
poetry shell
export PREFECT_PROFILE='prod'

# now when I come back to this terminal I should have the same active Prefect profile, despite the top of my `profiles.toml` showing `active = "dev"`
One day I need to write a blog post about these little things I've learned.
Hmm... Its still a little confusing when I try it. The desired effect is there, the session environment remains in prod even if i set the prefect profile differently in another project. However,
Copy code
echo $PREFECT_PROFILE
prod
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│              * prod │
│              debug │
└─────────────────────┘
Now I'm staying in my current project, we can forget about the project-B problem now. But I want to change to my debugging profile in project-A.
Copy code
prefect profile use debug
⠋ Connecting...
Connected to Prefect Orion using profile 'debug'
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│              * prod │
│              debug │
└─────────────────────┘
echo $PREFECT_PROFILE
prod
So it looks like I'm connected to Prefect Orion in the 'debug' profile, but I think not:
Copy code
prefect config view
PREFECT_PROFILE='prod'
PREFECT_LOGGING_LEVEL='INFO'
I was expecting 
PREFECT_LOGGING_LEVEL='DEBUG'
```
z
Environment variables will always take precedence over the profiles file* so 
prefect profile use …
won’t switch if the environment variable is there.
m
OK. Maybe this is very theoretical, but I'm already finding myself developing different projects concurrently -- seems like getting the local development environment organized is important. I can see myself working all day and deploying in the wrong environment. In any case each project would have different profiles with distinct {DB URL, PREFECT_ORION_API_PORT} pairs for each project. Solution 1: PREFECT_PROFILE • set active profile with env, 
PREFECT_PROFILE=[prod|dev|proj-B-prod|etc]
. • Manage the attributes of the different profiles as usual, but remember not to do 
prefect profile use
because its overridden. • To change profiles, you must set the environment variable. • The desired separation will happen for each project. ✅ Solution 2: PREFECT_HOME
Copy code
# set these in virtual env for each of  project-A/B
PREFECT_HOME='~/.prefect_project-A'
PREFECT_HOME='~/.prefect_project-B'
• Per project, PREFECT_HOME env variable does not need to change. Can be activated automatically when the environment is activated, (python-dotenv, 
env $(cat .env) poetry shell
, env-cmd, etc.) then forget about it. • Separate DB, 
profile.toml
and all happens naturally. • Do not need to set DB URL • Set PREFECT_ORION_API_PORT, must be distinct for each project • 
prefect profile use
behaves as expected.
z
👍 we basically use 
prefect profile use
to switch between Cloud staging/dev/prod but usually we can use the same local Python environment.
I can see your use-case being more common for our users though!
snap point 1
5 Views
Open in Slack
PreviousNext
Powered by