07/04/2020, 5:26 AM
Hey guys, I am looking for a solution to abstract computations from supercomputers, mostly for fmri data preprocessing (and, on a lower note, analysis on the underlying output) using already developed binaries within singularity (docker-like) containers on large static dataset. Each fmri session is to be processed parallely. We need to assess the current state of the computing environment (is the inode / lustre file limit close to be exceed, etc) and adapt/restrict the job count to prevent crash + perhaps remotely archive outputs and cleanup space as it is going. This can technically be done beforehand using some heuristic or dynamically. Would you consider prefect (relying on stg like dask_jobqueue.SLURMCluster) to be a good fit to develop such system, or would you have something else to recommend ?