Channels
g
Gopal
08/23/2019, 1:37 AMHi, I recently came across Prefect while reading on Dask. Currently I use AWS beanstalk for the deployment of our web application. How can I deploy dask cluster with Prefect in Beanstalk worker environment? Appreciate some help. Thanks
j
Jeremiah
08/23/2019, 3:20 AM
Hi @Gopal, unfortunately I’m not familiar with Beanstalk deployments. Perhaps someone else in the community will have some thoughts?
j
Joe Schmid
08/23/2019, 2:54 PMhi @Gopal, my sense is that Elastic Beanstalk won't be a good fit for Dask since it's (Beanstalk) focused specifically on deploying web apps. We use AWS EKS (Elastic Kubernetes Service) in conjunction with Dask Kubernetes (https://kubernetes.dask.org/en/latest/) That approach has been very successful for us. Using that, we create a Dask cluster in EKS that we then run Prefect on using their DaskExecutor.
The Dask Kubernetes project (and a related project: https://zero-to-jupyterhub.readthedocs.io/en/latest/index.html) are generally well-documented, though I will say if you haven't had experience with Kubernetes there will be some learning curve initially. However, if having a permanent Dask cluster is important to you then I think you'll find learning the basics of Kubernetes to be a worthwhile investment.
g
Gopal
08/26/2019, 1:27 AM@Joe Schmid Thanks a lot. That is of great help. I am a computer vision person and trying to deploy the analytics pipeline that I developed. I will explore EKS and dask kubernetes.
4 Views