@Joe Schmid (and anyone else with opinions), I've been thinking about the current state of creating a new

Environment

class per dask-cluster class, and it seems a bit untenable. I've been thinking about making a generic dask environment that takes the cluster-manager class and kwargs and uses that to create a dask cluster. Since dask already has a spec'd interface for this, it seems significantly simpler than having a mirror of each of these in prefect. Something like (class name not decided):

environment = DaskClusterEnvironment(
  cls=dask_yarn.YarnCluster,
  kwargs={
    "queue": "engineering",
    "environment": "hdfs://..."
  }
)

@Jim Crist-Harif Big +1 from me on this. There would obviously be lots of specifics to work through, but I see a huge opportunity to allow Prefect users to easily get a truly distributed Dask cluster. Today I think the bar is quite high for people to create the Dask cluster for Prefect to run on. It results in many Prefect users who could benefit from Dask's scalability not taking advantage of it because they don't have the experience/time/resources to work through a k8s deployment or other options. In short, I really like your idea!

Cool, I'll make an issue for this.

An alternative for people not having experience/time/resources for k8s deployments: Creating a helm chart for prefect server with a kubernetes agent

A helm chart as you described would also be useful, but solves a different problem - this is more about creating an

Environment

class that won't require prefect to mirror all of dask's deployment classes.