https://prefect.io logo
m

morten

07/25/2023, 10:04 AM
Hi šŸ‘‹ šŸ™‚ I dont know if this is bad practice, but I am starting out with Prefect, and I am having trouble executing my flows in a virtual environment, when I use Prefect Cloud to deploy. I made this post on discourse and I am wondering if anyone has a solution to the problem?
āœ… 1
j

Jake Kaplan

07/25/2023, 10:48 AM
šŸ‘‹ hi there As you said in your github issue, prefect is already executing in a a python environment when those steps are run, so it's not possible to move out of that env mid flow run and into your new env. Does each flow run need to be executing specific dependencies in it's own isolated virtual environment? Additionally what worker type are you using?
m

morten

07/25/2023, 10:53 AM
Hi, thanks for the reply šŸ™‚ let me answer your questions:
Does each flow run need to be executing specific dependencies in it's own isolated virtual environment?
Well, strictly speaking no, not at the moment. but as my of flows are going to grow, I will likely need some way to ensure that if one requires package A with version 1.0.0 and another requires version 2.0.0, its all possible.
Additionally what worker type are you using?
I am starting my worker with
prefect worker start --pool my-process-pool
where the pool is a
Process
type. Not sure if that is what you are asking(?)
j

Jake Kaplan

07/25/2023, 12:20 PM
Got it! I have not tried this but you might be able to change the
command
on your work pool? The default command is
python -m prefect.engine
but you may be able to change it to
source /{venv}/ && python -m prefect.engine
This should also be overridable per deployment that uses that work pool. Alternatively: ā€¢ It's not poetry, but one thing you can use is the https://docs.prefect.io/2.11.0/concepts/deployments-ux/#utility-steps
pip_install_requirements
pull step. This will install into your current working environment though while the flow run is running (not a new one). ā€¢ For managing different environments, the "prefect" way to do this is with container based worker/work pools. Aside from the
Process
type pretty much all the other worker/work pool types involve transient containers. This gives you the benefit of having an environment specifically for that 1 run of a flow. You can either use the pull step above for each deployment of a flow that has different dependencies or even bake them into your own images if you prefer.
m

morten

07/25/2023, 12:29 PM
awesome - I'll try that. Speaking of work pool types, do you know where I can find documentation on that? In the Work Pools, Workers & Agents section I can only find "worker" types, not "work pool" types.
Thanks for the answer, its so nice to get some help with this
j

Jake Kaplan

07/25/2023, 12:31 PM
Workers and Work Pools types are 1 and the same (or 1:1 I guess). So for example a
CloudRunWorker
can only read from a
CloudRunWorkPool
. Both have type
"cloud-run"
m

morten

07/25/2023, 12:31 PM
arh got it. Thanks !
j

Jake Kaplan

07/25/2023, 12:31 PM
and np! happy to help
m

morten

07/25/2023, 3:26 PM
@Jake Kaplan Thanks for your help again, I tried the docker solution and got some errors I am debugging now. But do I not need a public repository for the image? or is there a way to keeping it private, assuming I dont want to pay for a private repo that is
j

Jake Kaplan

07/25/2023, 3:45 PM
It doesn't have to be public necessarily? For example if you used AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html you'd just make sure the machine you run your worker on has access to pull images from there
m

morten

07/25/2023, 3:55 PM
nice - I'll try it out. I liked the virtual environment solution, as that would be one less tool I would have to maintain. But maybe I'll create a worker for each project, then I can do a requirement.txt install where it will only be used in the one project.
j

Jake Kaplan

07/25/2023, 5:05 PM
as that would be one less tool I would have to maintain.
I definitely can relate to that feeling šŸ˜… were you able to try changing the command to include activating the environment?
m

morten

07/25/2023, 5:07 PM
no - I will try it tomorrow. I suspect that it will only work if I also install prefect in the virtual environment. And then I might need to login again. But I will write here what happens, if it works I can also update my post about the solution.
Tried to change the command. Hardcoded it for the sake for testing it...
but got this really weird error:
Copy code
06:30:36.449 | INFO    | prefect.flow_runs.worker - Opening process...
06:30:36.454 | ERROR   | prefect.flow_runs.worker - Failed to submit flow run '55bf52ef-755e-4a60-8245-21f34fc16b5b' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 834, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/process.py", line 176, in run
    process = await run_process(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/processutils.py", line 258, in run_process
    async with open_process(
  File "/usr/local/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/processutils.py", line 202, in open_process
    process = await anyio.open_process(command, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_subprocesses.py", line 126, in open_process
    return await get_asynclib().open_process(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1041, in open_process
    process = await asyncio.create_subprocess_exec(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/subprocess.py", line 221, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1694, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 207, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 818, in _start
    self._proc = subprocess.Popen(
                 ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.11/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'source'
06:30:36.481 | INFO    | prefect.flow_runs.worker - Completed submission of flow run '55bf52ef-755e-4a60-8245-21f34fc16b5b'
I have no idea why it tries to think if source as a file or directory
j

Jake Kaplan

07/27/2023, 12:47 AM
took a little digging here, apparently you can't call
source
from inside of a process.popopen
I was able to get this to work by invoking my python executable from the other env directly
so instead of the command being
python -m prefect.engine
you would make it use python executable of the virtualenv you want like
/Users/jakekaplan/opt/anaconda3/envs/py310/bin/python3 -m prefect.engine
but that will run flow run w/ the given virtualenv, I should have realized that sooner šŸ˜…
m

morten

07/28/2023, 10:18 AM
šŸ˜„ np - always good to try things.
I might do a write up of my workflow I am getting to atm. it is rather simple, and somewhat limited by the fact that I use bitbucket, and not github, but limits the things I need to maintain and minimizes the products I have to use. I may write a personal chat to you, and ask if you think it is worth sharing at some point. But still need iron out some things. is that okay?
j

Jake Kaplan

07/28/2023, 11:35 AM
for sure. Would definitely encourage you to share it in #show-us-what-you-got