hi everyone! I want to install a module callesd"sq...
# ask-community
l
hi everyone! I want to install a module callesd"sqlalchemy" (snowflake connector) into my docker_image on googlecloud, because getting that error:
Copy code
/opt/prefect/healthcheck.py:151: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/storage.html#docker>
  flows = cloudpickle_deserialization_check(flow_file_paths)
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 151, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
ModuleNotFoundError: No module named 'snowflake.sqlalchemy'
Does anybody have experience with that? I tried the following:
Copy code
flow.storage = Docker(registry_url="xxx", image_name="xxx",
                          python_dependencies=["pandas", "oauthlib ", "requests", "requests_oauthlib", "oauth2client",
                                               "snowflake-connector-python", "pyarrow", "fastparquet"],
                          secrets=["GCP_CREDENTIALS"], files={"C:/Users/laura.vaida.000/anaconda3/envs/prefect/Lib/site-packages/sqlalchemy": "/modules/sqlachemy.py"})
1
m
i see in docs here https://docs.snowflake.com/en/user-guide/python-connector-install.html#step-1-install-the-connector you need to install requirements (#2) or
Copy code
# Generated on: Python 3.6.10
# With snowflake-connector-python version: 2.3.4
asn1crypto==1.4.0
azure-common==1.1.25
azure-core==1.8.2
azure-storage-blob==12.5.0
boto3==1.15.18
botocore==1.18.18
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
cryptography==2.9.2
idna==2.10
isodate==0.6.0
jmespath==0.10.0
msrest==0.6.19
oauthlib==3.1.0
oscrypto==1.2.1
pycparser==2.20
pycryptodomex==3.9.8
PyJWT==1.7.1
pyOpenSSL==19.1.0
python-dateutil==2.8.1
pytz==2020.1
requests==2.23.0
requests-oauthlib==1.3.0
s3transfer==0.3.3
six==1.15.0
urllib3==1.25.11
and i believe you need install sqlalchemy too
l
thanks for your reply! well i want to run that on google cloud, on my local machine i already installed that
m
so this dependencies must be install in your docker image. i think you can build new docker image with them and put in image_name="xxx" or add them to python_dependencies
l
thanks! I did it like that now but still giving me the error
Copy code
flow.storage = Docker(registry_url="xxx", image_name="xxx",
                          python_dependencies=["pandas", "oauthlib ", "requests", "requests_oauthlib", "oauth2client",
                                               "snowflake-connector-python", "pyarrow", "fastparquet", "asn1crypto==1.4.0",
                                               "azure-common==1.1.25", "azure-core==1.8.2","azure-storage-blob==12.5.0",
                                               "boto3","botocore","certifi==2020.6.20","cffi==1.14.3",
                                               "chardet==3.0.4","cryptography==2.9.2","idna==2.10","isodate==0.6.0",
                                               "jmespath==0.10.0","msrest==0.6.19","oauthlib==3.1.0","oscrypto==1.2.1",
                                               "pycparser==2.20","pycryptodomex==3.9.8","PyJWT==1.7.1","pyOpenSSL==19.1.0",
                                               "python-dateutil==2.8.1","pytz==2020.1","requests==2.23.0","requests-oauthlib==1.3.0",
                                               "s3transfer==0.3.3","six==1.15.0","urllib3==1.25.11"],secrets=["GCP_CREDENTIALS"],
 files={"C:/Users/laura.vaida.000/anaconda3/envs/prefect/Lib/site-packages/sqlalchemy": "/modules/sqlachemy.py"})
m
can you add sqlalchemy to python_dependencies too
or
Copy code
snowflake-sqlalchemy
l
hii that helped, thanks! 🙂
m
welcome)