https://prefect.io logo
r

Ronald Sam

08/17/2023, 6:20 PM
Any help would be greatly appreciated! Looking for help on how to run my deployment/flow in a virtual environment that's deployed to the cloud.
j

Jacob Bedard

08/17/2023, 6:28 PM
What are you running your python in?
Anaconda?
r

Ronald Sam

08/17/2023, 6:29 PM
no just through normal python -m venv
j

Jacob Bedard

08/17/2023, 6:29 PM
Best way to do this on Mac or PC is to use Anaconda Navigator, make a new virtualenv using a recent version of python, then open that virtualenv and do your pip install of prefect there
Which OS are you using here?
r

Ronald Sam

08/17/2023, 6:30 PM
wouldn't that have just one virtual environment for all prefect flows?
windows
j

Jacob Bedard

08/17/2023, 6:30 PM
Oh yeah I'd use anaconda
r

Ronald Sam

08/17/2023, 6:30 PM
I was hoping to have a virtual environment for each flow
j

Jacob Bedard

08/17/2023, 6:31 PM
That's a lot of virtual environments, potentially. Do you have conflicting python libraries needed for each flow?
That may not be a handy setup unless you have a known issue like that
j

Jake Kaplan

08/17/2023, 6:31 PM
If you want to do this you can update your command on your deployment to use the python executable from your env
1 sec i can get you an example
j

Jacob Bedard

08/17/2023, 6:31 PM
Ah a proper prefect expert is here. I'll step back now
😂 1
r

Ronald Sam

08/17/2023, 6:32 PM
not at the moment, but as we build more flows I'd like to have the option for a newer flow to use the latest version of a module while not needing to test/upgrade a previous flow that's using an older version
1
thanks for helping @Jacob Bedard!
j

Jake Kaplan

08/17/2023, 6:34 PM
The default command on a deployment is
python -m prefect.engine
which is what will kick off your flow run, get the deployment, execute the pull steps, etc. you'll need to use the python from your existing virtualenv. For example you can change the command to
/Users/jakekaplan/opt/anaconda3/envs/py310/bin/python3 -m prefect.engine
r

Ronald Sam

08/17/2023, 6:35 PM
thanks Jake, where do I enter this override command
is it in the deployment? Any way I can specify this override in the prefect.yaml?
is it here?
oh nvm I think it's on the work-pool
If I were to do that it will still only be one virtual environment per work-pool
but I'd like to have a virtual environment per deployment/flow
j

Jake Kaplan

08/17/2023, 6:37 PM
so it can be both. the work pool generates a template. if you want to hard code the
command
on all deployments that use that work pool, put it on the work pool. if you want to do it on the deployment, you can put in infra overrides
command: ...
r

Ronald Sam

08/17/2023, 6:38 PM
oh cool, I'll try it on the deployment
j

Jake Kaplan

08/17/2023, 6:38 PM
by default
command
is a temptable value (this is a random work pool, just to illustrate)
r

Ronald Sam

08/17/2023, 6:39 PM
do you have an example of what I should put in the deployment infra override in the UI?
j

Jake Kaplan

08/17/2023, 6:50 PM
sure it would look like
Copy code
{
  "command": "/Users/jakekaplan/opt/anaconda3/envs/py310/bin/python3 -m prefect.engine"
}
except replace it with the path to the python executable from the virutal env you would like to use
Screenshot 2023-08-17 at 2.50.08 PM.png
r

Ronald Sam

08/17/2023, 6:53 PM
thank you!! I'll give it a try and let you and the community know how it goes
it works!! Thank you so much 🙂
🙌 1
One more question.. I promise. How would I do this in the prefect.yaml file?
image.png
this didn't work
sorry @Jake Kaplan, any luck?
I've tried all combinations and nothing is working. I'm just trying to set infra overrides from prefect.yaml but nothing is working