Howdy...I have potential side project I may start to work on to enable non-kubernetes based ephemeral dask clusters ala dask-kubernetes....would it be better gauge interest here or in main community channel?

Out of curiosity… would it be spinning up a machine (or cluster of machines), register them with dask, do the work, then shut them down?

That seems useful

so there are 2 ways to do this i think: • via Fargate where your main prefect task acts as the scheduler and it uses setup hook to create the additional worker fargate tasks and wait for them to register with the local scheduler before executing the environment. However you are looking at about 5 minutes from the time that you start the task and it starting because of Fargate start up times for initial task + plus workers. • Could also leverage AWS Batch which has this thing called 'array jobs' which lets you create a group of ECS tasks (AWS Batch is just ECS under the covers)

Definitely sounds cool and as I understand it there are ton of Prefect users on AWS. (I see you are a main contributor to the fargate agent so you know first hand 🙂 ). I’m a GCP convert now but used to be an AWS gal, hope to be able to help! Did a quick read up on fargate vs batch, I see in the docs for fargate what you mean about the several minutes; for Batch, the way it is described it kind of feels like Batch could be implemented as its own executor (at least, in Prefect-only terms; you may be considering something broader than that) as opposed to a total analogue to dask-kubernetes that is running dask clusters. Sorry if this should be obvious, but what is it that makes a non-kubernetes dask deployment attractive here, instead of running dask-kubernetes on Amazon EKS?