Thread
#prefect-community
    Laura Vaida

    Laura Vaida

    1 year ago
    hi everyone! I want to install a module callesd"sqlalchemy" (snowflake connector) into my docker_image on googlecloud, because getting that error:
    /opt/prefect/healthcheck.py:151: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/storage.html#docker>
      flows = cloudpickle_deserialization_check(flow_file_paths)
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 151, in <module>
        flows = cloudpickle_deserialization_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
        flows.append(cloudpickle.loads(flow_bytes))
    ModuleNotFoundError: No module named 'snowflake.sqlalchemy'
    Does anybody have experience with that? I tried the following:
    flow.storage = Docker(registry_url="xxx", image_name="xxx",
                              python_dependencies=["pandas", "oauthlib ", "requests", "requests_oauthlib", "oauth2client",
                                                   "snowflake-connector-python", "pyarrow", "fastparquet"],
                              secrets=["GCP_CREDENTIALS"], files={"C:/Users/laura.vaida.000/anaconda3/envs/prefect/Lib/site-packages/sqlalchemy": "/modules/sqlachemy.py"})
    Marko Herkaliuk

    Marko Herkaliuk

    1 year ago
    i see in docs here https://docs.snowflake.com/en/user-guide/python-connector-install.html#step-1-install-the-connector you need to install requirements (#2) or
    # Generated on: Python 3.6.10
    # With snowflake-connector-python version: 2.3.4
    asn1crypto==1.4.0
    azure-common==1.1.25
    azure-core==1.8.2
    azure-storage-blob==12.5.0
    boto3==1.15.18
    botocore==1.18.18
    certifi==2020.6.20
    cffi==1.14.3
    chardet==3.0.4
    cryptography==2.9.2
    idna==2.10
    isodate==0.6.0
    jmespath==0.10.0
    msrest==0.6.19
    oauthlib==3.1.0
    oscrypto==1.2.1
    pycparser==2.20
    pycryptodomex==3.9.8
    PyJWT==1.7.1
    pyOpenSSL==19.1.0
    python-dateutil==2.8.1
    pytz==2020.1
    requests==2.23.0
    requests-oauthlib==1.3.0
    s3transfer==0.3.3
    six==1.15.0
    urllib3==1.25.11
    and i believe you need install sqlalchemy too
    Laura Vaida

    Laura Vaida

    1 year ago
    thanks for your reply! well i want to run that on google cloud, on my local machine i already installed that
    Marko Herkaliuk

    Marko Herkaliuk

    1 year ago
    so this dependencies must be install in your docker image. i think you can build new docker image with them and put in image_name="xxx" or add them to python_dependencies
    Laura Vaida

    Laura Vaida

    1 year ago
    thanks! I did it like that now but still giving me the error
    flow.storage = Docker(registry_url="xxx", image_name="xxx",
                              python_dependencies=["pandas", "oauthlib ", "requests", "requests_oauthlib", "oauth2client",
                                                   "snowflake-connector-python", "pyarrow", "fastparquet", "asn1crypto==1.4.0",
                                                   "azure-common==1.1.25", "azure-core==1.8.2","azure-storage-blob==12.5.0",
                                                   "boto3","botocore","certifi==2020.6.20","cffi==1.14.3",
                                                   "chardet==3.0.4","cryptography==2.9.2","idna==2.10","isodate==0.6.0",
                                                   "jmespath==0.10.0","msrest==0.6.19","oauthlib==3.1.0","oscrypto==1.2.1",
                                                   "pycparser==2.20","pycryptodomex==3.9.8","PyJWT==1.7.1","pyOpenSSL==19.1.0",
                                                   "python-dateutil==2.8.1","pytz==2020.1","requests==2.23.0","requests-oauthlib==1.3.0",
                                                   "s3transfer==0.3.3","six==1.15.0","urllib3==1.25.11"],secrets=["GCP_CREDENTIALS"],
     files={"C:/Users/laura.vaida.000/anaconda3/envs/prefect/Lib/site-packages/sqlalchemy": "/modules/sqlachemy.py"})
    Marko Herkaliuk

    Marko Herkaliuk

    1 year ago
    can you add sqlalchemy to python_dependencies too
    or
    snowflake-sqlalchemy
    Laura Vaida

    Laura Vaida

    1 year ago
    hii that helped, thanks! 🙂
    Marko Herkaliuk

    Marko Herkaliuk

    1 year ago
    welcome)