Has anyone used prefect as a wrapper to interact with a HPC cluster? Like wrapping the

bsub

or

lsub

command into a prefect task. Basically how do you use prefect to submit jobs to a compute cluster that has no automatic callback API for job completion and you have to ping it continuously to check the status? How do you check if the HPC job is complete? Do you parse the output of the HPC’s status command, check existence of an output file or something else?

Hello An, At the moment, we don’t have a first-class abstraction for this type of task, however there have been some suggestions posed. Maybe one task submits the job and the next task loops until completion or until it hits a max loop value? Typically HPC systems don’t have a way to wait for a job to complete and there generally isn’t rate limiting put in place, although please forgive me as I may be misunderstanding your inquiry.