Hi does anyone know if the prefect dask lib should be able t

Question

Hi does anyone know if the prefect dask lib should be able to handle the operator KubeCluster from dask kubernetes

Anna Geller · Answer

you could open a feature request on the prefect dask repo but afaik this is not currently supported you could use DaskTaskRunner with a Dask cluster deployed to K8s with KubeCluster though

Sander · Answer

Yep Trying to get that running but feels buggy

Sander · Answer

Might be that setting it up from a kubernetes job works better but need to test it still though

Sean Davis · Answer

< Sander> just curious how you have ended up proceeding I have some embarrassingly parallel tasks in a job that would benefit from running as dask tasks on a dask kubernetes cluster You mentioned that doing so feels buggy I m not looking for deep insight but really just enough to know if we should pursue dask tasks on kubernetes right now

Sander · Answer

< Sean Davis> still work in progress unfortunately No success yet