https://prefect.io logo
Title
p

Preston Marshall

02/08/2020, 1:30 AM
Any reason not to spin up a whole dask cluster per flow you're running?
c

Chris White

02/08/2020, 1:32 AM
no reason at all, just depends on what you need! Some people also leave long-running clusters up that they run multiple flows against
j

Jeremiah

02/08/2020, 1:32 AM
Historical note: this was actually one of the ways we recommended running Prefect in the beginning!
p

Preston Marshall

02/08/2020, 1:34 AM
cool, seems like the easiest way to run things, although I could have a cluster Deployment and just expose it as a service via dns
j

Jeff Brainerd

02/08/2020, 1:42 AM
Hi, FWIW we are spinning up a new dask cluster for every flow version that gets deployed. Existing flows can just run to completion using the old version, then we kill off that cluster once all tasks have been drained. The benefit for us is that our frequent deploys don’t interrupt existing flows. We are running on Fargate.
:upvote: 3
It does require us to have a nanny process that monitors and cleans up infra once it quiesces.
p

Preston Marshall

02/08/2020, 1:45 AM
sounds great