This is a bit of a stretch I suppose has anybody run MPI par

Question

This is a bit of a stretch I suppose has anybody run MPI parallel jobs in k8s as a prefect task

Evan Curtin · Answer

perhaps a light wrapper over <https github com kubeflow mpi operator>

Anna Geller · Answer

One community user tried that and it doesn t work because kubeflow uses custom Kubernetes job types which won t work with Prefect Kubernetes agent Did you try mapping with Prefect It allows you to spin up thousands of child tasks that can be parallelized e g across a KubeCluster all with just Prefect and Dask LMK if you need more info

Kevin Kho · Answer

Are you using MPI because you have an HPC

Evan Curtin · Answer

The libraries we are working with are built around MPI

Evan Curtin · Answer

I ve inherited a legacy codebase

Evan Curtin · Answer

I m quite comfortable wrapping a CRD into a prefect task actually

Kevin Kho · Answer

Ah I see

Evan Curtin · Answer

like for example if we wanted to do large scale SVD using <https slepc upv es >

Kevin Kho · Answer

Holy cow you must have big data to be needing MPI for an SVD operation laughing Is this for recommendation

Evan Curtin · Answer

we re very early on investigating options due to the complexity of that I don t see us doing it practically but maybe itll be a fun investigation