https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jim Crist-Harif

05/06/2020, 3:45 PM
@Joe Schmid (and anyone else with opinions), I've been thinking about the current state of creating a new
Environment
class per dask-cluster class, and it seems a bit untenable. I've been thinking about making a generic dask environment that takes the cluster-manager class and kwargs and uses that to create a dask cluster. Since dask already has a spec'd interface for this, it seems significantly simpler than having a mirror of each of these in prefect. Something like (class name not decided):
Copy code
environment = DaskClusterEnvironment(
  cls=dask_yarn.YarnCluster,
  kwargs={
    "queue": "engineering",
    "environment": "hdfs://..."
  }
)
👍 1
👀 1
j

Joe Schmid

05/06/2020, 3:50 PM
@Jim Crist-Harif Big +1 from me on this. There would obviously be lots of specifics to work through, but I see a huge opportunity to allow Prefect users to easily get a truly distributed Dask cluster. Today I think the bar is quite high for people to create the Dask cluster for Prefect to run on. It results in many Prefect users who could benefit from Dask's scalability not taking advantage of it because they don't have the experience/time/resources to work through a k8s deployment or other options. In short, I really like your idea!
j

Jim Crist-Harif

05/06/2020, 3:57 PM
Cool, I'll make an issue for this.
d

David Ojeda

05/06/2020, 3:58 PM
An alternative for people not having experience/time/resources for k8s deployments: Creating a helm chart for prefect server with a kubernetes agent
👍 1
j

Jim Crist-Harif

05/06/2020, 4:00 PM
A helm chart as you described would also be useful, but solves a different problem - this is more about creating an
Environment
class that won't require prefect to mirror all of dask's deployment classes.