Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
s
Sander
10/15/2022, 10:16 AMHi, does anyone know if the prefect-dask lib should be able to handle the operator.KubeCluster from dask_kubernetes?
✅ 1
a
Anna Geller
10/15/2022, 2:19 PMyou could open a feature request on the prefect-dask repo, but afaik this is not currently supported
you could use DaskTaskRunner with a Dask cluster deployed to K8s with KubeCluster though
s
Sander
10/15/2022, 2:21 PMYep. Trying to get that running but feels buggy.
Might be that setting it up from a kubernetes job works better but need to test it still though.
🙌 1
s
Sean Davis
11/04/2022, 4:15 PM@Sander just curious how you have ended up proceeding. I have some embarrassingly parallel tasks in a job that would benefit from running as dask tasks on a dask kubernetes cluster. You mentioned that doing so feels "buggy". I'm not looking for deep insight, but really just enough to know if we should pursue dask tasks on kubernetes right now.
s
Sander
11/16/2022, 9:52 PM@Sean Davis still work in progress unfortunately. No success yet.