One community user tried that and it doesn’t work because kubeflow uses custom Kubernetes job types which won’t work with Prefect Kubernetes agent.
Did you try mapping with Prefect? It allows you to spin up thousands of child tasks that can be parallelized e.g. across a KubeCluster - all with just Prefect and Dask. LMK if you need more info
k
Kevin Kho
02/04/2022, 3:26 PM
Are you using MPI because you have an HPC?
e
Evan Curtin
02/04/2022, 4:56 PM
The libraries we are working with are built around MPI
I’ve inherited a legacy codebase
I’m quite comfortable wrapping a CRD into a prefect task actually