Michael Bell
01/21/2022, 9:26 PMprefect
and dask-cloudprovider
right now. It seems dask-cloudprovider[aws]
relies on aiobotocore
which pins to a very specific botocore
version and that's causing conflicts when trying to set up my environment. Anyone have any experience with this?Kevin Kho
Anna Geller
FROM prefecthq/prefect:0.15.12-python3.8
RUN pip install "dask-cloudprovider[aws]" --upgrade
Michael Bell
01/21/2022, 9:41 PM[[source]]
url = "<https://pypi.org/simple>"
verify_ssl = true
name = "pypi"
[packages]
prefect = {version = "==0.15.12", extras = ["aws"]}
dask-cloudprovider = {extras = ["aws"], version = "*"}
[dev-packages]
[requires]
python_version = "3.9"
and it's failing with the following message:There are incompatible versions in the resolved dependencies:
botocore<1.23.25,>=1.23.24 (from aiobotocore==2.1.0->dask-cloudprovider[aws]==2022.1.0->-r /tmp/pipenvedm55bbgrequirements/pipenv-q7883dfo-constraints.txt (line 3))
botocore<1.24.0,>=1.23.41 (from boto3==1.20.41->prefect[aws]==0.15.12->-r /tmp/pipenvedm55bbgrequirements/pipenv-q7883dfo-constraints.txt (line 2))
Anna Geller
annageller/prefect-dask-cloudprovider:latest
Michael Bell
01/21/2022, 10:24 PMdask-cloudprovider
just to manage the Dask cluster. Perhaps it's better to isolate that functionality from outside of my flow, maybe to create a Lambda that starts/stops the cluster, then invoke that lambda from the flow as needed? This would remove this dependency from the flow code entirely. Is that a pattern that makes sense?Anna Geller
Zanie