https://prefect.io logo
Title
e

Evan Curtin

02/04/2022, 3:13 PM
This is a bit of a stretch I suppose, has anybody run MPI parallel jobs in k8s as a prefect task?
perhaps a light wrapper over https://github.com/kubeflow/mpi-operator ?
a

Anna Geller

02/04/2022, 3:23 PM
One community user tried that and it doesn’t work because kubeflow uses custom Kubernetes job types which won’t work with Prefect Kubernetes agent. Did you try mapping with Prefect? It allows you to spin up thousands of child tasks that can be parallelized e.g. across a KubeCluster - all with just Prefect and Dask. LMK if you need more info
k

Kevin Kho

02/04/2022, 3:26 PM
Are you using MPI because you have an HPC?
e

Evan Curtin

02/04/2022, 4:56 PM
The libraries we are working with are built around MPI
I’ve inherited a legacy codebase
I’m quite comfortable wrapping a CRD into a prefect task actually
k

Kevin Kho

02/04/2022, 4:58 PM
Ah I see
e

Evan Curtin

02/04/2022, 4:59 PM
like for example, if we wanted to do large scale SVD using https://slepc.upv.es/
k

Kevin Kho

02/04/2022, 5:01 PM
Holy cow you must have big data to be needing MPI for an SVD operation 😆. Is this for recommendation?
e

Evan Curtin

02/04/2022, 5:04 PM
we’re very early on investigating options, due to the complexity of that I don’t see us doing it practically but maybe itll be a fun investigation