One community user tried that and it doesn’t work because kubeflow uses custom Kubernetes job types which won’t work with Prefect Kubernetes agent.
Did you try mapping with Prefect? It allows you to spin up thousands of child tasks that can be parallelized e.g. across a KubeCluster - all with just Prefect and Dask. LMK if you need more info
02/04/2022, 3:26 PM
Are you using MPI because you have an HPC?
02/04/2022, 4:56 PM
The libraries we are working with are built around MPI
I’ve inherited a legacy codebase
I’m quite comfortable wrapping a CRD into a prefect task actually