https://prefect.io logo
#prefect-contributors
Title
# prefect-contributors
b

Braun Reyes

03/18/2020, 8:33 PM
Howdy...I have potential side project I may start to work on to enable non-kubernetes based ephemeral dask clusters ala dask-kubernetes....would it be better gauge interest here or in main community channel?
a

Alex Cano

03/18/2020, 8:40 PM
Out of curiosity… would it be spinning up a machine (or cluster of machines), register them with dask, do the work, then shut them down?
l

Luke Orland

03/18/2020, 8:40 PM
That seems useful
upvote 3
b

Braun Reyes

03/18/2020, 9:06 PM
so there are 2 ways to do this i think: • via Fargate where your main prefect task acts as the scheduler and it uses setup hook to create the additional worker fargate tasks and wait for them to register with the local scheduler before executing the environment. However you are looking at about 5 minutes from the time that you start the task and it starting because of Fargate start up times for initial task + plus workers. • Could also leverage AWS Batch which has this thing called 'array jobs' which lets you create a group of ECS tasks (AWS Batch is just ECS under the covers)
l

Laura Lorenz (she/her)

03/19/2020, 12:37 AM
Definitely sounds cool and as I understand it there are ton of Prefect users on AWS. (I see you are a main contributor to the fargate agent so you know first hand 🙂 ). I’m a GCP convert now but used to be an AWS gal, hope to be able to help! Did a quick read up on fargate vs batch, I see in the docs for fargate what you mean about the several minutes; for Batch, the way it is described it kind of feels like Batch could be implemented as its own executor (at least, in Prefect-only terms; you may be considering something broader than that) as opposed to a total analogue to dask-kubernetes that is running dask clusters. Sorry if this should be obvious, but what is it that makes a non-kubernetes dask deployment attractive here, instead of running dask-kubernetes on Amazon EKS?
4 Views