Hi I recently came across Prefect while reading on Dask Curr

Question

Hi I recently came across Prefect while reading on Dask Currently I use AWS beanstalk for the deployment of our web application How can I deploy dask cluster with Prefect in Beanstalk worker environment Appreciate some help Thanks

Jeremiah · Answer

Hi < Gopal> unfortunately I m not familiar with Beanstalk deployments Perhaps someone else in the community will have some thoughts

Joe Schmid · Answer

hi < Gopal> my sense is that Elastic Beanstalk won t be a good fit for Dask since it s Beanstalk focused specifically on deploying web apps We use AWS EKS Elastic Kubernetes Service in conjunction with Dask Kubernetes That approach has been very successful for us Using that we create a Dask cluster in EKS that we then run Prefect on using their DaskExecutor

Joe Schmid · Answer

The Dask Kubernetes project and a related project <https zero to jupyterhub readthedocs io en latest index html> are generally well documented though I will say if you haven t had experience with Kubernetes there will be some learning curve initially However if having a permanent Dask cluster is important to you then I think you ll find learning the basics of Kubernetes to be a worthwhile investment

Gopal · Answer

< Joe Schmid> Thanks a lot That is of great help I am a computer vision person and trying to deploy the analytics pipeline that I developed I will explore EKS and dask kubernetes