https://prefect.io logo
s

Sander

10/15/2022, 10:16 AM
Hi, does anyone know if the prefect-dask lib should be able to handle the operator.KubeCluster from dask_kubernetes?
1
a

Anna Geller

10/15/2022, 2:19 PM
you could open a feature request on the prefect-dask repo, but afaik this is not currently supported you could use DaskTaskRunner with a Dask cluster deployed to K8s with KubeCluster though
s

Sander

10/15/2022, 2:21 PM
Yep. Trying to get that running but feels buggy.
Might be that setting it up from a kubernetes job works better but need to test it still though.
🙌 1
s

Sean Davis

11/04/2022, 4:15 PM
@Sander just curious how you have ended up proceeding. I have some embarrassingly parallel tasks in a job that would benefit from running as dask tasks on a dask kubernetes cluster. You mentioned that doing so feels "buggy". I'm not looking for deep insight, but really just enough to know if we should pursue dask tasks on kubernetes right now.
s

Sander

11/16/2022, 9:52 PM
@Sean Davis still work in progress unfortunately. No success yet.
3 Views