Daniel
06/19/2022, 3:44 AMprefecthq/prefect:latest-python3.10
docker image with some additional packages including the snowflake python connector and it's dependencies. Flows without snowflake interactions work perfectly when run from Prefect cloud however my EL flows which insert data into snowflake tables fail to load and execute returning >ModuleNotFoundError("No module named 'snowflake'").
Haven't been able to fine any similar reports among the community so wondering if someone could suggest what I may have done wrong.
With thanks,
Daniel.Kevin Kho
Anna Geller
Daniel
06/22/2022, 8:53 AMasn1crypto==1.5.1
boto3==1.24.7
certifi==2022.5.18.1
cffi==1.15.0
charset-normalizer==2.0.12
cryptography==36.0.2
datetime==4.4
idna==3.3
oscrypto==1.3.0
pycparser==2.21
pycryptodomex==3.14.1
PyJWT==2.4.0
pyOpenSSL==22.0.0
pytz==2022.1
requests==2.28.0
urllib3==1.26.9
snowflake-connector-python==2.7.8Docker file, builds on your image:
# syntax=docker/dockerfile:1
FROM prefecthq/prefect:latest-python3.10
RUN pip install --upgrade pip
COPY requirements.txt .
RUN pip install -r requirements.txt
Storage and run configurations:
FLOW_NAME = "PoC_refresh_medidata_snfk"
STORAGE = S3(
bucket="XXXXXXXXX",
key=f"flows/{FLOW_NAME}.py",
stored_as_script=True,
# this will ensure to upload the Flow script to S3 during registration
local_script_path=f"{FLOW_NAME}.py",
)
RUN_CONFIG = ECSRun(
labels=["prod"],
task_role_arn="arn:aws:iam::XXXXXXXXXXXX:role/prefectTaskRole",
run_task_kwargs=dict(cluster="prefectEcsCluster", launchType="FARGATE",),
)
with Flow(FLOW_NAME, storage=STORAGE, run_config=RUN_CONFIG,) as flow:
ETL_Medidata(select="keyword", keyword="Daniel")
if __name__ == "__main__":
flow.register("tester")
Thanks again,
Daniel.Kevin Kho
Daniel
06/23/2022, 1:13 PM