https://prefect.io logo
#prefect-server
Title
# prefect-server
s

Shuchita Tripathi

04/01/2022, 3:49 PM
I want to add azure storage for my flows. but i got an error - No module named azure. So, I am trying to run - pip install prefect[azure] -> but this is taking too long. it is running from about an hour. Is there anything I can do to get this resolved?
k

Kevin Kho

04/01/2022, 3:52 PM
Oh man. I believe this has to more with the fact that pip 0.24 has a dependency resolver and the Azure packages within themselves have dependency conflicts so you get this loop of not being able to resolve packages. You can try downgrading to pip 0.23
s

Shuchita Tripathi

04/01/2022, 3:58 PM
ok. with poetry, I am getting a different error
k

Kevin Kho

04/01/2022, 4:01 PM
Doesn’t poetry just use pip though? Not sure.
a

Andrew Huang

04/01/2022, 4:05 PM
Try installing just
pip install azure-storage-blob
and try rerunning your flow
s

Shuchita Tripathi

04/01/2022, 4:08 PM
ok
a

Andrew Huang

04/01/2022, 4:17 PM
did that work for you? I created a PR to separate the ML requirements https://github.com/PrefectHQ/prefect/pull/5632
🙏 1
upvote 1
s

Shuchita Tripathi

04/01/2022, 4:34 PM
no, it didn't. i tried adding other libraries as well. including azure-core. but i am still getting No module named azure error
k

Kevin Kho

04/01/2022, 4:48 PM
what is your python version? the azure library might have minimum
s

Shuchita Tripathi

04/01/2022, 4:48 PM
python 3.9.9
k

Kevin Kho

04/01/2022, 5:43 PM
I can’t find it in the
setup.py
of their repo but I suspect your Python might be too high. Could you try 3.8?
a

Andrew Huang

04/01/2022, 5:47 PM
Can you double check whether your python path is correct e.g.
which python
and azure was installed into the proper environment
which pip
pip list
Copy code
(azure-test) [Fri 1 11:46] prefect: pip install azure-core azure-storage-blob
Collecting azure-core
  Using cached azure_core-1.23.1-py3-none-any.whl (178 kB)
Collecting azure-storage-blob
  Using cached azure_storage_blob-12.11.0-py3-none-any.whl (346 kB)
Collecting requests>=2.18.4
  Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting six>=1.11.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting typing-extensions>=4.0.1
  Using cached typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Collecting msrest>=0.6.21
  Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting cryptography>=2.1.4
  Using cached cryptography-36.0.2-cp36-abi3-macosx_10_10_universal2.whl (4.7 MB)
Collecting cffi>=1.12
  Using cached cffi-1.15.0-cp39-cp39-macosx_11_0_arm64.whl (173 kB)
Collecting requests-oauthlib>=0.5.0
  Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting isodate>=0.6.0
  Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Installing collected packages: certifi, urllib3, typing-extensions, six, pycparser, oauthlib, idna, charset-normalizer, requests, isodate, cffi, requests-oauthlib, cryptography, azure-core, msrest, azure-storage-blob
Successfully installed azure-core-1.23.1 azure-storage-blob-12.11.0 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cryptography-36.0.2 idna-3.3 isodate-0.6.1 msrest-0.6.21 oauthlib-3.2.0 pycparser-2.21 requests-2.27.1 requests-oauthlib-1.3.1 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.9
(azure-test) [Fri 1 11:46] prefect: python3
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:06)
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import azure
>>> from azure.storage import blob
>>>
s

Shuchita Tripathi

04/01/2022, 5:56 PM
this is the toml file. we add packages using poetry add <package-name>
so all these packages are installed
a

Andrew Huang

04/01/2022, 6:11 PM
Can you post the traceback of the error you’re receiving?
s

Shuchita Tripathi

04/01/2022, 6:33 PM
This is the actual error.
Can there be some other reason for No module named azure error?
k

Kevin Kho

04/01/2022, 6:38 PM
Not really. For Azure Storage as long as
azure-storage-blob
installed it should work
s

Shuchita Tripathi

04/01/2022, 6:45 PM
ok. what else I can look for to troubleshoot this error
k

Kevin Kho

04/01/2022, 6:47 PM
I am a bit confused. What is your RunConfiguration? Is this LocalRun or something like DockerRun?
s

Shuchita Tripathi

04/01/2022, 6:48 PM
LocalRun. I am running the API from my local machine.
k

Kevin Kho

04/01/2022, 6:49 PM
So you were able to install Azure already?
s

Shuchita Tripathi

04/01/2022, 6:49 PM
no. from the time I have given storage=Azure, things are not working
before this, i tried storage=Local, and it was working
k

Kevin Kho

04/01/2022, 6:50 PM
Ah then you really need to get the Azure module installed locally. You can install
azure-storage-blob
separately by pip installing that
s

Shuchita Tripathi

04/01/2022, 6:51 PM
it is installed.
k

Kevin Kho

04/01/2022, 6:53 PM
And you try starting an interactive terminal and doing
Copy code
>>> import azure
>>> from azure.storage import blob
The second one is more important for <http://>>>%20import%20azure%20>>>%20from%20azure.storage%20import%20blob|Azure> storage
s

Shuchita Tripathi

04/01/2022, 6:55 PM
k

Kevin Kho

04/01/2022, 6:57 PM
maybe you need to restart the local agent to reflect the new packages installed?
s

Shuchita Tripathi

04/01/2022, 6:58 PM
i have restarted everything. restarted my laptop a few minutes back
k

Kevin Kho

04/01/2022, 6:59 PM
are you using virtual environments by chance? is the agent using the same virtual environment as the one with azure installed?
s

Shuchita Tripathi

04/01/2022, 7:04 PM
yes i am using virtual env. i never tried running the agent inside the virtual env. i just did and the agent is not starting because it is trying to go to localhost:4200 instead of IP:4200 my config.toml has that IP. is there some other config.toml required for virtual env?
k

Kevin Kho

04/01/2022, 7:29 PM
what is your virtual env manager?
s

Shuchita Tripathi

04/01/2022, 7:35 PM
poetry
k

Kevin Kho

04/01/2022, 8:20 PM
i think that’s kinda weird. the config.toml should be lived and respected because virtual environments treat it the same
s

Shuchita Tripathi

04/01/2022, 8:46 PM
trying some different ways now. passing container name in code and connection string (PREFECT__CONTEXT__SECRETS__AZURE_STORAGE_CONNECTION_STRING="<value>") as an env variable while agent setup. Now getting this error.
Copy code
raise error
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x000001E292AF2D90>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed
k

Kevin Kho

04/01/2022, 8:58 PM
I think this is specifically an Azure issue. Is this on registration or run time?
That seems like the import is working
s

Shuchita Tripathi

04/01/2022, 8:59 PM
this is run time issue
when I running the API which calls this function, then this issue is coming up
k

Kevin Kho

04/01/2022, 9:00 PM
When you register, it was able to upload the Flow to the blob storage?
s

Shuchita Tripathi

04/01/2022, 9:00 PM
no
i commented stored_as_script because i checked that nothing was uploaded in the blob
k

Kevin Kho

04/01/2022, 9:01 PM
Ah ok and then does that upload?
s

Shuchita Tripathi

04/01/2022, 9:02 PM
no. I am getting the above issue. getaddrinfo failed
k

Kevin Kho

04/01/2022, 9:03 PM
It seems like you are unable to connect to Azure in general. Is your local machine authenticated?
s

Shuchita Tripathi

04/01/2022, 9:03 PM
I am retrying. This is the msg I get when i make the api call.
and then it will error out in 2-3 minutes
i have provided the connection string. will that not work for authentication? other than this i have done az login (azure cli login) but i don't think that is related to this.
it errored out
k

Kevin Kho

04/01/2022, 10:39 PM
Can you try listing the blobs maybe with the Azure client to confirm you can do that? Seems like you can’t connect to Azure
4 Views