https://prefect.io logo
Join Slack
Powered by
Hey Team - what is the best way to pass in the pre...
# ask-community
m
Hey Team - what is the best way to pass in the prefect workspace and key to a docker image when frequently switching between workspaces? We would like to create one image and easily switch between them, however our code runs using the PyCharm interpreter and thus we need to be able to pass the key/workspace in easily. Do you have any recommendations?
✅ 1
j
Hi there, The first thing that comes to mind is profiles: https://docs.prefect.io/2.10.13/concepts/settings/?h=prof#creating-and-removing-profiles you can add a profile file in your docker image with two profiles, one with each api key/url. Although I guess it depends how/when you need to switch
example:
Copy code
$ cat ~/.prefect/profiles.toml                                                                                                                                                                      
active = "workspace1"


[profiles.workspace1]
PREFECT_API_KEY = "<api key for workspace1>"
PREFECT_API_URL = "<workspace1 url>"

[profiles.workspace2]
PREFECT_API_KEY = "<api key for workspace2>"
PREFECT_API_URL = "<workspace2 url>"
m
Thanks @Jake Kaplan! I'll give it a try.
🙌 1
In my image, I can't seem to find the .prefect directory where I would then have the toml. I have the prefect executable at 
/usr/local/bin/prefect
Currently I'm logging into the workspace at the end of the docker file using 
RUN /bin/sh -c "prefect cloud login --key <creds> --<workspace>"
I think I can pass the args into the dockerfile and use them as such
Copy code
RUN prefect profile create 'dev'
RUN prefect --profile "dev" config set PREFECT_API_KEY==PREFECT_API_KEY_DEV PREFECT_API_URL=PREFECT_API_URL_DEV
RUN prefect profile create 'prod'
RUN prefect --profile "prod" config set PREFECT_API_KEY==PREFECT_API_KEY_PROD PREFECT_API_URL=PREFECT_API_URL_PROD
Then default the container to use dev 
RUN prefect use dev
I would then be able to run in my deploy section if I wanted to use the other workspace
Copy code
def deploy():
    import os
    os.system('prefect use prod')
    deployment = Deployment.build_from_flow(...)
Using this syntax, I would not need to run the 
prefect cloud login
command at the end of my dockerfile? 
RUN /bin/sh -c "prefect cloud login --key <creds> --<workspace>"
What is the difference between using a profile with the api key and URL (
prefect use
) vs using 
prefect cloud login
?
Also, found the file: 
./usr/local/lib/python3.10/site-packages/prefect/profiles.toml
j
prefect cloud login ...
is effectively a nice interactive wrapper around setting 
PREFECT_API_KEY
and 
PREFECT_API_URL
m
Hey Jake - thanks again for the help. I am having a very weird issue where I am switching profiles using 
os.system('prefect profile use prod')
. When switching, everything is showing up as prod, however when running the deployment, it is only deploying to dev. 
prefect profile inspect
and 
prefect cloud workspace ls
shows only the prod workspace and is active. I think it may have something to do with having two separate service roles, one for each workspace. Again, everything in the configuration is showing I am in the proper workspace, but when running any prefect commands it is somehow using the other workspace/profile.
j
Are you running 
os.system('prefect profile use prod')
once you're already inside a prefect entrypoint? a prefect cli command or flow run basically?
yess 1
or do you have an example?
m
Yes. In my dockerfile I create and assign the credentials to both of the profiles. I then default in the dockerfile to 
prefect use profile dev
. When I would like to run in production, I add os.system('prefect profile use prod') which successfully shows me getting into the production workspace and proper profile, however when I run a command it still uploads to the other workspace. Could you elaborate on the entrypoint? I do not see an entrypoints in the .prefect folder in the container, however both of my profiles look good in the profiles.toml
I'll be honest it is quite the conundrum!
j
Once you've imported prefect in python, you'll need to use a context manager to switch profiles. Heres an example of something that might work for you
Copy code
from prefect.context import use_profile
from prefect import flow
from prefect.settings import load_current_profile


@flow
def my_flow():
    print(load_current_profile())


if __name__ == '__main__':
    import os
    profile = os.environ.get("PROFILE")
    with use_profile(profile):
        my_flow()
m
Ooo let me give it a shot.
j
Copy code
# profiles.toml

active_profile = "dev_profile"

[profiles.dev_profile]
PREFECT_API_KEY = "dev_api_key"

[profiles.prod_profile]
PREFECT_API_KEY = "prod_api_key"
✅ 1
Copy code
(py311) ➜  prefect git:(disable-events) ✗ export PROFILE=prod_profile
(py311) ➜  prefect git:(disable-events) ✗ python my_flow.py          
17:03:19.353 | INFO    | prefect.engine - Created flow run 'ultramarine-jackal' for flow 'my-flow'
name='prod_profile' settings={<PREFECT_API_KEY: str>: 'prod_api_key'} source=PosixPath('/Users/jakekaplan/.prefect/profiles.toml')
17:03:19.533 | INFO    | Flow run 'ultramarine-jackal' - Finished in state Completed()
(py311) ➜  prefect git:(disable-events) ✗ export PROFILE=dev_profile
(py311) ➜  prefect git:(disable-events) ✗ python my_flow.py         
17:03:35.346 | INFO    | prefect.engine - Created flow run 'stereotyped-agama' for flow 'my-flow'
name='dev_profile' settings={<PREFECT_API_KEY: str>: 'dev_api_key'} source=PosixPath('/Users/jakekaplan/.prefect/profiles.toml')
17:03:35.633 | INFO    | Flow run 'stereotyped-agama' - Finished in state Completed()
m
That works! I guess there must be more prefect configurations which are changed when using the context manager vs just doing 
prefect profile use
?
j
Once you start running your python code that uses prefect you're ALREADY inside of settings/profile context, you just normally don't even know it or care. So running the cli command directly won't refresh the currently loaded profile that's already in memory
🙂 1
m
Got it. Thanks so much for the help! It is very appreciated
j
np, happy it's working!
m
Would one be able to run in one profile for a majority of the flow, then run another profile for just one of the tasks?
j
FWIW most users don't tend to change their settings/profiles dynamically within their flows. The most usually is calling things with 
--profile
in the CLI (doing something for 1 flow run) that being said I think, you should be able to do that within/around a single task, but I would recommend testing to make sure it's working as expected
m
Agreed - not the best practice. I might have a certain case where it could be useful.
👍 1
It works 😎
j
Actually thinking it through a little more, flow runs are workspace scoped. So while you can change profiles, a task run may not know what to do if all the sudden it needs to get created in a workspace where it's parent flow run does not exist, if that makes sense
m
I think that makes sense. It's more for testing purposes, not full flow runs.
j
ah gotcha 👍
m
It actually is not working as expected now - nested things suck as a String.load don't seem to be picking up the block from the other workspace. E.g.
Copy code
#using profile 'z'
with use_profile('x'):
    y = String.load('y') #Still prints string from 'z'
Would it be bad practice to have a service role with access to both workspaces?
j
A service account can have access to both workspaces no problem! Like I said yesterday, I don't think this is great practice. Something is likely to break or not work how you'd expect. Prefect expects to operate within a single workspace during the context of a run
😎 1
11 Views
Open in Slack
PreviousNext
Powered by