merlin
12/08/2022, 6:22 PMcd /path/to/prefect-project-A
poetry shell # (activates the virtual env)
prefect profile use 'dev'
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│ default │
│ prod │
│ * dev │
└─────────────────────┘
now open another terminal
cd /path/to/prefect-project-B
poetry shell # (activate this different virtual env)
prefect profile use 'prod'
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│ default │
│ * prod │
│ dev │
└─────────────────────┘
but now go back the first terminal window, and look at the active profile.
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│ default │
│ * prod │
│ dev │
└─────────────────────┘
So changes to the active profile in any session or virtual environment are global.
Are there steps I can take to isolate these?James Sopkin
12/08/2022, 6:32 PMprofiles.toml
file within the .prefect directory. When you set your profile the active
profile within the profiles.toml
file also changesZanie
12/08/2022, 6:54 PMPREFECT_PROFILE
environment variable when entering an environment.merlin
12/09/2022, 12:46 AMpoetry shell
export PREFECT_PROFILE='prod'
# now when I come back to this terminal I should have the same active Prefect profile, despite the top of my `profiles.toml` showing `active = "dev"`
One day I need to write a blog post about these little things I've learned.echo $PREFECT_PROFILE
prod
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│ default │
│ * prod │
│ debug │
└─────────────────────┘
Now I'm staying in my current project, we can forget about the project-B problem now.
But I want to change to my debugging profile in project-A.
prefect profile use debug
⠋ Connecting...
Connected to Prefect Orion using profile 'debug'
prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│ default │
│ * prod │
│ debug │
└─────────────────────┘
echo $PREFECT_PROFILE
prod
So it looks like I'm connected to Prefect Orion in the 'debug' profile, but I think not:
prefect config view
PREFECT_PROFILE='prod'
PREFECT_LOGGING_LEVEL='INFO'
I was expecting PREFECT_LOGGING_LEVEL='DEBUG'
```Zanie
12/09/2022, 1:43 AMprefect profile use …
won’t switch if the environment variable is there.merlin
12/09/2022, 4:54 AMPREFECT_PROFILE=[prod|dev|proj-B-prod|etc]
.
• Manage the attributes of the different profiles as usual, but remember not to do prefect profile use
because its overridden.
• To change profiles, you must set the environment variable.
• The desired separation will happen for each project.
✅ Solution 2: PREFECT_HOME
# set these in virtual env for each of project-A/B
PREFECT_HOME='~/.prefect_project-A'
PREFECT_HOME='~/.prefect_project-B'
• Per project, PREFECT_HOME env variable does not need to change. Can be activated automatically when the environment is activated, (python-dotenv, env $(cat .env) poetry shell
, env-cmd, etc.) then forget about it.
• Separate DB, profile.toml
and all happens naturally.
• Do not need to set DB URL
• Set PREFECT_ORION_API_PORT, must be distinct for each project
• prefect profile use
behaves as expected.Zanie
12/09/2022, 3:40 PMprefect profile use
to switch between Cloud staging/dev/prod but usually we can use the same local Python environment.