Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, I recently came across Prefect while reading on Dask. Currently I use AWS beanstalk for the deployment of our web application. How can I deploy dask cluster with Prefect in Beanstalk worker environment? Appreciate some help. Thanks

Hi <@UMF9ZCVGR>, unfortunately I’m not familiar with Beanstalk deployments. Perhaps someone else in the community will have some thoughts?

hi <@UMF9ZCVGR>, my sense is that Elastic Beanstalk won't be a good fit for Dask since it's (Beanstalk) focused specifically on deploying web apps. We use AWS EKS (Elastic Kubernetes Service) in conjunction with Dask Kubernetes (<https://kubernetes.dask.org/en/latest/>) That approach has been very successful  for us. Using that, we create a Dask cluster in EKS that we then run Prefect on using their DaskExecutor.

The Dask Kubernetes project (and a related project: <https://zero-to-jupyterhub.readthedocs.io/en/latest/index.html>) are generally well-documented, though I will say if you haven't had experience with Kubernetes there will be some learning curve initially. However, if having a permanent Dask cluster is important to you then I think you'll find learning the basics of Kubernetes to be a worthwhile investment.

<@ULHHD0MPF> Thanks a lot. That is of great help. I am a computer vision person and trying to deploy the analytics pipeline that I developed. I will explore EKS and dask kubernetes.