https://prefect.io
Join Slack
Hey folks. I am having a really hard time with dependency resolution between `prefect` and `dask-clo...
# ask-community
m
Hey folks. I am having a really hard time with dependency resolution between 
prefect
and 
dask-cloudprovider
right now. It seems 
dask-cloudprovider[aws]
relies on 
aiobotocore
which pins to a very specific 
botocore
version and that's causing conflicts when trying to set up my environment. Anyone have any experience with this?
k
If you are using pip 0.23 and above, the dependency resolver is painful (slow and doesnt install due to conflicts) so we pin to the one before that. You can also turn off the new dependency resolver. If you are talking about runtime failures, aiobotocore does break dask-cloudprovider on some versions. I think 1.4.0 was breaking but I don’t know if the issues were fixed in 2.0
a
I’ve used it just this week when reproducing this Github issue and it seemed to work using the latest version. Do you run it in a virtual environment or in a Docker container? I was using this Dockerfile and it seemed to work well:
Copy code
FROM prefecthq/prefect:0.15.12-python3.8
RUN  pip install "dask-cloudprovider[aws]" --upgrade
m
Trying to create a virtual environment
👍 1
Let me try that.
I'm trying to install the following pipenv file:
Copy code
[[source]]
url = "<https://pypi.org/simple>"
verify_ssl = true
name = "pypi"

[packages]
prefect = {version = "==0.15.12", extras = ["aws"]}
dask-cloudprovider = {extras = ["aws"], version = "*"}

[dev-packages]

[requires]
python_version = "3.9"
and it's failing with the following message:
Copy code
There are incompatible versions in the resolved dependencies:
  botocore<1.23.25,>=1.23.24 (from aiobotocore==2.1.0->dask-cloudprovider[aws]==2022.1.0->-r /tmp/pipenvedm55bbgrequirements/pipenv-q7883dfo-constraints.txt (line 3))
  botocore<1.24.0,>=1.23.41 (from boto3==1.20.41->prefect[aws]==0.15.12->-r /tmp/pipenvedm55bbgrequirements/pipenv-q7883dfo-constraints.txt (line 2))
a
But don’t you need to package it into a Docker container? 🤔 because to use it with ECS Fargate, Dask must be able to pull your image from some container registry. You can try using my image:
Copy code
annageller/prefect-dask-cloudprovider:latest
m
Yes, sorry I am trying to install requirements in docker. But I can't get the requirements to install in a venv just to work through the issue
👍 1
Let me give that a shot
Ultimately I need 
dask-cloudprovider
just to manage the Dask cluster. Perhaps it's better to isolate that functionality from outside of my flow, maybe to create a Lambda that starts/stops the cluster, then invoke that lambda from the flow as needed? This would remove this dependency from the flow code entirely. Is that a pattern that makes sense?
a
When you use dask cloud provider, Dask creates a new ECS cluster (unless you specify not to) so usually there is no need to do it separately through a lambda function. Feel free to use the code from the Github issue I shared
👍 1
z
I’ve opened https://github.com/PrefectHQ/prefect/pull/5357 to relax our constraints which may help resolve this
🙏 1
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by