Are there some public examples where people have tried to pair Prefect with some HPC resources like clusters, schedulers, MPI, etc.. ?
I am aware there is some level of support for Dask, but it is not completely obvious to me how to deploy it on a cluster.
Suppose I can create/setup a Dask cluster, is it enough to provide
to the flow decorators?
How do I control the type of the workers (CPU, GPU, etc..)? Is there MPI support through the Dask-MPI interface?
More generally, what kind of limitations do I have compared to a conventional scheduler interface?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
