Hi all, is anyone trying to run a prefect workflow using slurm? I am wondering why my DaskTaskRunners are running on the same nodes when I have asynchronous subflows. I would say the flow is just ignoring the task runner directive and just running on the compute node that is launching the orchestration. Very odd
b
Bianca Hoch
12/03/2024, 9:04 PM
Hi Pascal! I've seen a few folks looking to use Prefect with SLURM. In fact, one of our community members developed a Prefect runner for his platform, quacc. It's designed for high-throughput computational materials science. Maybe his guide will help you out
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.