Hi wave slightly smiling face I dont know if this is bad pra

Question

Hi wave slightly smiling face I dont know if this is bad practice but I am starting out with Prefect and I am having trouble executing my flows in a virtual environment when I use Prefect Cloud to deploy I made <https discourse prefect io t unable to activate a virtual environment for execution of a job using cloud as deployment 3345|this> post on discourse and I am wondering if anyone has a solution to the problem

Jake Kaplan · Answer

wave hi there As you said in your github issue prefect is already executing in a a python environment when those steps are run so it s not possible to move out of that env mid flow run and into your new env Does each flow run need to be executing specific dependencies in it s own isolated virtual environment Additionally what worker type are you using

morten · Answer

Hi thanks for the reply slightly smiling face let me answer your questions gt Does each flow run need to be executing specific dependencies in it s own isolated virtual environment Well strictly speaking no not at the moment but as my of flows are going to grow I will likely need some way to ensure that if one requires package A with version 1 0 0 and another requires version 2 0 0 its all possible gt Additionally what worker type are you using I am starting my worker with `prefect worker start pool my process pool` where the pool is a `Process` type Not sure if that is what you are asking

Jake Kaplan · Answer

Got it I have not tried this but you might be able to change the `command` on your work pool The default command is `python m prefect engine` but you may be able to change it to `source venv amp amp python m prefect engine` This should also be overridable per deployment that uses that work pool Alternatively It s not poetry but one thing you can use is the <https docs prefect io 2 11 0 concepts deployments ux utility steps> `pip install requirements` pull step This will install into your current working environment though while the flow run is running not a new one For managing different environments the prefect way to do this is with container based worker work pools Aside from the `Process` type pretty much all the other worker work pool types involve transient containers This gives you the benefit of having an environment specifically for that 1 run of a flow You can either use the pull step above for each deployment of a flow that has different dependencies or even bake them into your own images if you prefer

morten · Answer

awesome I ll try that Speaking of work pool types do you know where I can find documentation on that In the <https docs prefect io 2 11 0 concepts work pools worker types|Work Pools Workers amp Agents> section I can only find worker types not work pool types

morten · Answer

Thanks for the answer its so nice to get some help with this

Jake Kaplan · Answer

Workers and Work Pools types are 1 and the same or 1 1 I guess So for example a `CloudRunWorker` can only read from a `CloudRunWorkPool` Both have type ` cloud run `

morten · Answer

arh got it Thanks

Jake Kaplan · Answer

and np happy to help

morten · Answer

< Jake Kaplan> Thanks for your help again I tried the docker solution and got some errors I am debugging now But do I not need a public repository for the image or is there a way to keeping it private assuming I dont want to pay for a private repo that is

Jake Kaplan · Answer

It doesn t have to be public necessarily For example if you used AWS ECR <https docs aws amazon com AmazonECR latest userguide docker pull ecr image html> you d just make sure the machine you run your worker on has access to pull images from there

morten · Answer

nice I ll try it out I liked the virtual environment solution as that would be one less tool I would have to maintain But maybe I ll create a worker for each project then I can do a requirement txt install where it will only be used in the one project

Jake Kaplan · Answer

gt as that would be one less tool I would have to maintain I definitely can relate to that feeling sweat smile were you able to try changing the command to include activating the environment

morten · Answer

no I will try it tomorrow I suspect that it will only work if I also install prefect in the virtual environment And then I might need to login again But I will write here what happens if it works I can also update my post about the solution

morten · Answer

Tried to change the command Hardcoded it for the sake for testing it

morten · Answer

but got this really weird error ```06 30 36 449 | INFO | prefect flow runs worker Opening process 06 30 36 454 | ERROR | prefect flow runs worker Failed to submit flow run 55bf52ef 755e 4a60 8245 21f34fc16b5b to infrastructure Traceback most recent call last File usr local lib python3 11 site packages prefect workers base py line 834 in submit run and capture errors result = await self run ^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect workers process py line 176 in run process = await run process ^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect utilities processutils py line 258 in run process async with open process File usr local lib python3 11 contextlib py line 204 in aenter return await anext self gen ^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect utilities processutils py line 202 in open process process = await anyio open process command kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages anyio core subprocesses py line 126 in open process return await get asynclib open process ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages anyio backends asyncio py line 1041 in open process process = await asyncio create subprocess exec ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 asyncio subprocess py line 221 in create subprocess exec transport protocol = await loop subprocess exec ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 asyncio base events py line 1694 in subprocess exec transport = await self make subprocess transport ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 asyncio unix events py line 207 in make subprocess transport transp = UnixSubprocessTransport self protocol args shell ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 asyncio base subprocess py line 36 in init self start args=args shell=shell stdin=stdin stdout=stdout File usr local lib python3 11 asyncio unix events py line 818 in start self proc = subprocess Popen ^^^^^^^^^^^^^^^^^ File usr local lib python3 11 subprocess py line 1026 in init self execute child args executable preexec fn close fds File usr local lib python3 11 subprocess py line 1950 in execute child raise child exception type errno num err msg err filename FileNotFoundError Errno 2 No such file or directory source 06 30 36 481 | INFO | prefect flow runs worker Completed submission of flow run 55bf52ef 755e 4a60 8245 21f34fc16b5b ``` I have no idea why it tries to think if source as a file or directory

Jake Kaplan · Answer

took a little digging here apparently you can t call `source` from inside of a process popopen

Jake Kaplan · Answer

I was able to get this to work by invoking my python executable from the other env directly

Jake Kaplan · Answer

so instead of the command being `python m prefect engine` you would make it use python executable of the virtualenv you want like ` Users jakekaplan opt anaconda3 envs py310 bin python3 m prefect engine`

Jake Kaplan · Answer

but that will run flow run w the given virtualenv I should have realized that sooner sweat smile

morten · Answer

smile np always good to try things

morten · Answer

I might do a write up of my workflow I am getting to atm it is rather simple and somewhat limited by the fact that I use bitbucket and not github but limits the things I need to maintain and minimizes the products I have to use I may write a personal chat to you and ask if you think it is worth sharing at some point But still need iron out some things is that okay

Jake Kaplan · Answer

for sure Would definitely encourage you to share it in < CL09KTZPX|show us what you got>