Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, does anyone know if the prefect-dask lib should be able to handle the operator.KubeCluster from dask_kubernetes?

you could open a feature request on the prefect-dask repo, but afaik this is not currently supported

you could use DaskTaskRunner with a Dask cluster deployed to K8s with KubeCluster though

Yep. Trying to get that running but feels buggy. 

Might be that setting it up from a kubernetes job works better but need to test it still though. 

<@U03CG94GPRC> just curious how you have ended up proceeding. I have some embarrassingly parallel tasks in a job that would benefit from running as dask tasks on a dask kubernetes cluster. You mentioned that doing so feels "buggy". I'm not looking for deep insight, but really just enough to know if we should pursue dask tasks on kubernetes right now.

<@U02KVRDCZAQ> still work in progress unfortunately. No success yet. 