Any reason not to spin up a whole dask cluster per flow you're running?
c
Chris White
02/08/2020, 1:32 AM
no reason at all, just depends on what you need! Some people also leave long-running clusters up that they run multiple flows against
j
Jeremiah
02/08/2020, 1:32 AM
Historical note: this was actually one of the ways we recommended running Prefect in the beginning!
p
Preston Marshall
02/08/2020, 1:34 AM
cool, seems like the easiest way to run things, although I could have a cluster Deployment and just expose it as a service via dns
j
Jeff Brainerd
02/08/2020, 1:42 AM
Hi, FWIW we are spinning up a new dask cluster for every flow version that gets deployed. Existing flows can just run to completion using the old version, then we kill off that cluster once all tasks have been drained. The benefit for us is that our frequent deploys don’t interrupt existing flows. We are running on Fargate.
:upvote: 3
It does require us to have a nanny process that monitors and cleans up infra once it quiesces.