Prefect profile changing profile in one virtual environment

Question

Prefect profile changing profile in one virtual environment is reflected in other virtual environments on the same machine I m confused about using profiles Details in thread

merlin · Answer

I maintain virtual environments for my prefect projects as I develop them for packaging elsewhere I m confused because changes are spanning across shell virtual environments on my local machine open a terminal ```cd path to prefect project A poetry shell activates the virtual env prefect profile use dev prefect profile ls Available Profiles default prod dev ``` now open another terminal ```cd path to prefect project B poetry shell activate this different virtual env prefect profile use prod prefect profile ls Available Profiles default prod dev ``` but now go back the first terminal window and look at the active profile ```prefect profile ls Available Profiles default prod dev ``` So changes to the active profile in any session or virtual environment are global Are there steps I can take to isolate these

James Sopkin · Answer

Hi merlin a prefect profile is just an api url and api key that can be set in your `profiles toml` file within the prefect directory When you set your profile the `active` profile within the `profiles toml` file also changes

James Sopkin · Answer

It wouldn t be specific to your virtual env as it is a config set outside of it

Zanie · Answer

Profiles can contain any settings not just API key and URL they are global based on the user though

Zanie · Answer

You can change the profile per virtual environment by setting the `PREFECT PROFILE` environment variable when entering an environment

merlin · Answer

Thanks James ahhh exactly the detail I was missing that explains it So <https docs prefect io concepts settings overriding defaults with environment variables|posting a relevant doc page for others coming to this > an environment variable will override the profile settings of the same name In my case its gotta look like ```poetry shell export PREFECT PROFILE= prod now when I come back to this terminal I should have the same active Prefect profile despite the top of my `profiles toml` showing `active = dev ```` One day I need to write a blog post about these little things I ve learned

merlin · Answer

Hmm Its still a little confusing when I try it The desired effect is there the session environment remains in prod even if i set the prefect profile differently in another project However ```echo $PREFECT PROFILE prod prefect profile ls Available Profiles default prod debug ``` Now I m staying in my current project we can forget about the project B problem now But I want to change to my debugging profile in project A ```prefect profile use debug Connecting Connected to Prefect Orion using profile debug prefect profile ls Available Profiles default prod debug echo $PREFECT PROFILE prod``` So it looks like I m connected to Prefect Orion in the debug profile but I think not ```prefect config view PREFECT PROFILE= prod PREFECT LOGGING LEVEL= INFO ``` I was expecting `PREFECT LOGGING LEVEL= DEBUG ` ```

Zanie · Answer

Environment variables will always take precedence over the profiles file so `prefect profile use ` won t switch if the environment variable is there

merlin · Answer

OK Maybe this is very theoretical but I m already finding myself developing different projects concurrently seems like getting the local development environment organized is important I can see myself working all day and deploying in the wrong environment In any case each project would have different profiles with distinct DB URL PREFECT ORION API PORT pairs for each project Solution 1 PREFECT PROFILE set active profile with env `PREFECT PROFILE= prod|dev|proj B prod|etc ` Manage the attributes of the different profiles as usual but remember not to do `prefect profile use` because its overridden To change profiles you must set the environment variable The desired separation will happen for each project white check mark Solution 2 PREFECT HOME ``` set these in virtual env for each of project A B PREFECT HOME= ~ prefect project A PREFECT HOME= ~ prefect project B ``` Per project PREFECT HOME env variable does not need to change Can be activated automatically when the environment is activated python dotenv `env $ cat env poetry shell` env cmd etc then forget about it Separate DB `profile toml` and all happens naturally Do not need to set DB URL Set PREFECT ORION API PORT must be distinct for each project `prefect profile use` behaves as expected

Zanie · Answer

+1 we basically use `prefect profile use` to switch between Cloud staging dev prod but usually we can use the same local Python environment

Zanie · Answer

I can see your use case being more common for our users though