Orion feature request/idea: now that the tool is ‘free from the DAG’ I could image that supporting new ‘execution’ layers besides Dask will be much easier. Has the team thought about added native support for Ray.io clusters?
11 months ago
Hey Bob, I've prototyped such an executor and it's definitely very easy to support. Keep an eye out for future announcements in this vein 🙂
Regarding Prefect as an abstraction on top of Dask and Ray. It would be very powerful IMHO if Prefect was able to promote some existing dask layer election parameters as first-class Prefect APIs. I’m thinking of
that let you tune the flow execution. I think this is a killer Dask feature and would love the same high-level API available when running on Ray. No doubt the implementation would be non-trivial.