Hello prefect! I’m an ML engineering working on computer vision for cancer diagnosis.
At our company we have an HPC and use Slurm as our scheduler. We are trying to integrate prefect into our pipeline and are currently working on writing a Slurm task executor, hoping this is all we’ll need to build too use prefect for our usecase.
If anyone has any resources or suggestions for us, they would be much appreciated, thanks!
Rodrigo Ceballos
12/02/2020, 2:51 PM
PS: We cannot use Dask because of incompatibilities with PyTorch
z
Zanie
12/02/2020, 4:07 PM
Welcome @Rodrigo Ceballos! Don’t have any specific suggestions for you, but this is an exciting idea 🙂
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.