https://prefect.io logo
Join Slack
Powered by
I've got a virtualenv set up on an EC2 and I'm hav...
# prefect-community
j
I've got a virtualenv set up on an EC2 and I'm having trouble getting a flow to run from the UI. I'm seeing this error for a package that I've installed on the env already: 
Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'snowflake\'")
I'm running this as a local agent. The flow runs ok when I do a flow.run() on the machine running the agent. What am I missing here?
k
Did you run the agent out of the virtual environment?
For Prefect 1, you will have to do that. For Prefect 2, you can choose the environment to run out of
j
Hmmmm, ok
That needs to be specified in the agent start? Because what I did was activate the virtualenv, then I did the agent start command
So I needed to also specify the environment to run in when I initialized the agent?
k
Prefect 1 doesn’t have an option to specify environment, only Prefect 2 does. I use conda so have not tried with virtualenv. In conda, you can just run the agent start command inside the conda environment and it will use that environment to run flows. For virtualenv, I am less confident that it would work.
j
Here's what I ran after activating the virtualenv:
Copy code
prefect agent local start -n "Data Workhorse 1" -l "ec2 worker"
I'm running conda too
I started the terminal from the anaconda navigator environments tab
k
Oh it should surely work for conda environments
j
Why might it not be picking up the package then? I've definitely got snowflake installed in that env
k
What happens if you run interactive Python and try to install snowflake?
j
Like, execute a bash command from python?
'requirement already satisfied'
k
Yeah like:
Copy code
python
then
Copy code
import snowflake
j
runs without issue
But it's weird. In my error message on the flow, I see something about python 3.8.10
But neither of my conda environments run that. They're running 3.8.13
k
That’s pretty hard to tell. Do you have multiple agents?
j
No, just the one
I'm starting to think it's this environment issue, probably... I can't find an environment on this machine with python version 3.8.10
Have you ever seen this before?
Copy code
Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'snowflake\'")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n  - python: (flow built with \'3.8.13\', currently running with \'3.8.10\')\nThis also may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
k
The error there is a guess that something went wrong with versioning but I have not seen it attached to a standard PyPI module
j
Anything you can point me towards to debug this?
Might it have to do with how snowflake is imported? 
import snowflake.connector as snow
But that still doesn't explain how prefect is trying to execute in an environment I can't find anywhere
k
I don’t think this is about how it’s imported, though I don’t have any specific material around this. There must be an environment somewhere with that Python version (maybe base)?
12 Views
Open in Slack
PreviousNext
Powered by