Hi everyone. Has anyone here worked with a setup where flow runs would need to be jobs scheduled with SLURM on an HPC cluster or can point me in the direction of related resources? From my current understanding of workers, I assume that it may be a a good idea to define a new type of worker (like the Docker, process, AWS, GCP workers, but specific for SLURM scheduling).

Hey @Wiebke Koepp! You’re absolutely right that creating a new worker type would be the best way to dynamically schedule flow runs via SLURM. I’m not aware of anyone who has created a SLURM worker, but if you’re interested in creating one then I can send you some resources to help you create one.

That would be great @alex, thanks!

Here’s our guide on creating a new worker type and our Docker worker docs/implementation, which should work well as a reference. Feel free to reach out if there’s anything we can help with!

@Wiebke Koepp I’m looking to do the same as you at the moment (schedule flow runs with SLURM on a HPC cluster). I would massively appreciate if you have any advice / experience / code you would consider sharing on how you went about that?