Thread
#prefect-community
    b

    Bob Colner

    11 months ago
    Orion feature request/idea: now that the tool is ‘free from the DAG’ I could image that supporting new ‘execution’ layers besides Dask will be much easier. Has the team thought about added native support for Ray.io clusters?
    Michael Adkins

    Michael Adkins

    11 months ago
    Hey Bob, I've prototyped such an executor and it's definitely very easy to support. Keep an eye out for future announcements in this vein 🙂
    b

    Bob Colner

    11 months ago
    very cool
    Regarding Prefect as an abstraction on top of Dask and Ray. It would be very powerful IMHO if Prefect was able to promote some existing dask layer election parameters as first-class Prefect APIs. I’m thinking of
    cluster_kwargs
    like
    processes
    ,
    n_workers
    , and
    threads_per_worker
    that let you tune the flow execution. I think this is a killer Dask feature and would love the same high-level API available when running on Ray. No doubt the implementation would be non-trivial.