Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey folks. I am having a really hard time with dependency resolution between `prefect` and `dask-cloudprovider` right now. It seems `dask-cloudprovider[aws]` relies on `aiobotocore` which pins to a very specific `botocore` version and that's causing conflicts when trying to set up my environment. Anyone have any experience with this?

If you are using pip 0.23 and above, the dependency resolver is painful (slow and doesnt install due to conflicts) so we pin to the one before that. You can also turn off the new dependency resolver. If you are talking about runtime failures, aiobotocore does break dask-cloudprovider on some versions. I think 1.4.0 was breaking but I don’t know if the issues were fixed in 2.0

I’ve used it just this week when reproducing <https://github.com/PrefectHQ/prefect/issues/5330|this Github issue> and it seemed to work using the latest version. Do you run it in a virtual environment or in a Docker container? I was using this Dockerfile and it seemed to work well:
```FROM prefecthq/prefect:0.15.12-python3.8
RUN  pip install "dask-cloudprovider[aws]" --upgrade```


I'm trying to install the following pipenv file:
```[[source]]
url = "<https://pypi.org/simple>"
verify_ssl = true
name = "pypi"

[packages]
prefect = {version = "==0.15.12", extras = ["aws"]}
dask-cloudprovider = {extras = ["aws"], version = "*"}

[dev-packages]

[requires]
python_version = "3.9"```
and it's failing with the following message:

```There are incompatible versions in the resolved dependencies:
  botocore&lt;1.23.25,&gt;=1.23.24 (from aiobotocore==2.1.0-&gt;dask-cloudprovider[aws]==2022.1.0-&gt;-r /tmp/pipenvedm55bbgrequirements/pipenv-q7883dfo-constraints.txt (line 3))
  botocore&lt;1.24.0,&gt;=1.23.41 (from boto3==1.20.41-&gt;prefect[aws]==0.15.12-&gt;-r /tmp/pipenvedm55bbgrequirements/pipenv-q7883dfo-constraints.txt (line 2))```

But don’t you need to package it into a Docker container? :thinking_face: because to use it with ECS Fargate, Dask must be able to pull your image from some container registry. You can try using my image:
```annageller/prefect-dask-cloudprovider:latest```

Yes, sorry I am trying to install requirements in docker. But I can't get the requirements to install in a venv just to work through the issue

Ultimately I need `dask-cloudprovider` just to manage the Dask cluster. Perhaps it's better to isolate that functionality from outside of my flow, maybe to create a Lambda that starts/stops the cluster, then invoke that lambda from the flow as needed? This would remove this dependency from the flow code entirely. Is that a pattern that makes sense?

When you use dask cloud provider, Dask creates a new ECS cluster (unless you specify not to) so usually there is no need to do it separately through a lambda function. Feel free to use the code from the Github issue I shared

I’ve opened <https://github.com/PrefectHQ/prefect/pull/5357> to relax our constraints which may help resolve this