https://prefect.io logo
Join Slack
Powered by
Hi :wave: :slightly_smiling_face: I dont know if t...
# prefect-getting-started
m
Hi šŸ‘‹ šŸ™‚ I dont know if this is bad practice, but I am starting out with Prefect, and I am having trouble executing my flows in a virtual environment, when I use Prefect Cloud to deploy. I made this post on discourse and I am wondering if anyone has a solution to the problem?
āœ… 1
j
šŸ‘‹ hi there As you said in your github issue, prefect is already executing in a a python environment when those steps are run, so it's not possible to move out of that env mid flow run and into your new env. Does each flow run need to be executing specific dependencies in it's own isolated virtual environment? Additionally what worker type are you using?
m
Hi, thanks for the reply šŸ™‚ let me answer your questions:
Does each flow run need to be executing specific dependencies in it's own isolated virtual environment?
Well, strictly speaking no, not at the moment. but as my of flows are going to grow, I will likely need some way to ensure that if one requires package A with version 1.0.0 and another requires version 2.0.0, its all possible.
Additionally what worker type are you using?
I am starting my worker with 
prefect worker start --pool my-process-pool
where the pool is a 
Process
type. Not sure if that is what you are asking(?)
j
Got it! I have not tried this but you might be able to change the 
command
on your work pool? The default command is 
python -m prefect.engine
but you may be able to change it to 
source /{venv}/ && python -m prefect.engine
This should also be overridable per deployment that uses that work pool. Alternatively: ā€¢ It's not poetry, but one thing you can use is the https://docs.prefect.io/2.11.0/concepts/deployments-ux/#utility-steps 
pip_install_requirements
pull step. This will install into your current working environment though while the flow run is running (not a new one). ā€¢ For managing different environments, the "prefect" way to do this is with container based worker/work pools. Aside from the 
Process
type pretty much all the other worker/work pool types involve transient containers. This gives you the benefit of having an environment specifically for that 1 run of a flow. You can either use the pull step above for each deployment of a flow that has different dependencies or even bake them into your own images if you prefer.
m
awesome - I'll try that. Speaking of work pool types, do you know where I can find documentation on that? In the Work Pools, Workers & Agents section I can only find "worker" types, not "work pool" types.
Thanks for the answer, its so nice to get some help with this
j
Workers and Work Pools types are 1 and the same (or 1:1 I guess). So for example a 
CloudRunWorker
can only read from a 
CloudRunWorkPool
. Both have type 
"cloud-run"
m
arh got it. Thanks !
j
and np! happy to help
m
@Jake Kaplan Thanks for your help again, I tried the docker solution and got some errors I am debugging now. But do I not need a public repository for the image? or is there a way to keeping it private, assuming I dont want to pay for a private repo that is
j
It doesn't have to be public necessarily? For example if you used AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html you'd just make sure the machine you run your worker on has access to pull images from there
m
nice - I'll try it out. I liked the virtual environment solution, as that would be one less tool I would have to maintain. But maybe I'll create a worker for each project, then I can do a requirement.txt install where it will only be used in the one project.
j
as that would be one less tool I would have to maintain.
I definitely can relate to that feeling šŸ˜… were you able to try changing the command to include activating the environment?
m
no - I will try it tomorrow. I suspect that it will only work if I also install prefect in the virtual environment. And then I might need to login again. But I will write here what happens, if it works I can also update my post about the solution.
Tried to change the command. Hardcoded it for the sake for testing it...
but got this really weird error:
Copy code
06:30:36.449 | INFO    | prefect.flow_runs.worker - Opening process...
06:30:36.454 | ERROR   | prefect.flow_runs.worker - Failed to submit flow run '55bf52ef-755e-4a60-8245-21f34fc16b5b' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 834, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/process.py", line 176, in run
    process = await run_process(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/processutils.py", line 258, in run_process
    async with open_process(
  File "/usr/local/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/processutils.py", line 202, in open_process
    process = await anyio.open_process(command, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_subprocesses.py", line 126, in open_process
    return await get_asynclib().open_process(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1041, in open_process
    process = await asyncio.create_subprocess_exec(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/subprocess.py", line 221, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1694, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 207, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 818, in _start
    self._proc = subprocess.Popen(
                 ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.11/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'source'
06:30:36.481 | INFO    | prefect.flow_runs.worker - Completed submission of flow run '55bf52ef-755e-4a60-8245-21f34fc16b5b'
I have no idea why it tries to think if source as a file or directory
j
took a little digging here, apparently you can't call 
source
from inside of a process.popopen
I was able to get this to work by invoking my python executable from the other env directly
so instead of the command being 
python -m prefect.engine
you would make it use python executable of the virtualenv you want like 
/Users/jakekaplan/opt/anaconda3/envs/py310/bin/python3 -m prefect.engine
but that will run flow run w/ the given virtualenv, I should have realized that sooner šŸ˜…
m
šŸ˜„ np - always good to try things.
I might do a write up of my workflow I am getting to atm. it is rather simple, and somewhat limited by the fact that I use bitbucket, and not github, but limits the things I need to maintain and minimizes the products I have to use. I may write a personal chat to you, and ask if you think it is worth sharing at some point. But still need iron out some things. is that okay?
j
for sure. Would definitely encourage you to share it in #CL09KTZPX
Open in Slack
PreviousNext
Powered by