https://prefect.io logo
Title
j

Jacob Bedard

07/12/2022, 3:33 PM
I've got a virtualenv set up on an EC2 and I'm having trouble getting a flow to run from the UI. I'm seeing this error for a package that I've installed on the env already:
Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'snowflake\'")
I'm running this as a local agent. The flow runs ok when I do a flow.run() on the machine running the agent. What am I missing here?
k

Kevin Kho

07/12/2022, 3:36 PM
Did you run the agent out of the virtual environment?
For Prefect 1, you will have to do that. For Prefect 2, you can choose the environment to run out of
j

Jacob Bedard

07/12/2022, 3:37 PM
Hmmmm, ok
That needs to be specified in the agent start? Because what I did was activate the virtualenv, then I did the agent start command
So I needed to also specify the environment to run in when I initialized the agent?
k

Kevin Kho

07/12/2022, 3:40 PM
Prefect 1 doesn’t have an option to specify environment, only Prefect 2 does. I use conda so have not tried with virtualenv. In conda, you can just run the agent start command inside the conda environment and it will use that environment to run flows. For virtualenv, I am less confident that it would work.
j

Jacob Bedard

07/12/2022, 3:40 PM
Here's what I ran after activating the virtualenv:
prefect agent local start -n "Data Workhorse 1" -l "ec2 worker"
I'm running conda too
I started the terminal from the anaconda navigator environments tab
k

Kevin Kho

07/12/2022, 3:41 PM
Oh it should surely work for conda environments
j

Jacob Bedard

07/12/2022, 3:42 PM
Why might it not be picking up the package then? I've definitely got snowflake installed in that env
k

Kevin Kho

07/12/2022, 3:42 PM
What happens if you run interactive Python and try to install snowflake?
j

Jacob Bedard

07/12/2022, 3:44 PM
Like, execute a bash command from python?
'requirement already satisfied'
k

Kevin Kho

07/12/2022, 3:49 PM
Yeah like:
python
then
import snowflake
j

Jacob Bedard

07/12/2022, 3:50 PM
runs without issue
But it's weird. In my error message on the flow, I see something about python 3.8.10
But neither of my conda environments run that. They're running 3.8.13
k

Kevin Kho

07/12/2022, 3:55 PM
That’s pretty hard to tell. Do you have multiple agents?
j

Jacob Bedard

07/12/2022, 3:58 PM
No, just the one
I'm starting to think it's this environment issue, probably... I can't find an environment on this machine with python version 3.8.10
Have you ever seen this before?
Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'snowflake\'")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n  - python: (flow built with \'3.8.13\', currently running with \'3.8.10\')\nThis also may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
k

Kevin Kho

07/12/2022, 4:26 PM
The error there is a guess that something went wrong with versioning but I have not seen it attached to a standard PyPI module
j

Jacob Bedard

07/12/2022, 4:42 PM
Anything you can point me towards to debug this?
Might it have to do with how snowflake is imported?
import snowflake.connector as snow
But that still doesn't explain how prefect is trying to execute in an environment I can't find anywhere
k

Kevin Kho

07/13/2022, 4:56 AM
I don’t think this is about how it’s imported, though I don’t have any specific material around this. There must be an environment somewhere with that Python version (maybe base)?