Hi all and @Marvin, does anyone know how to specify a DaskTaskRunner using the SLURMCLuster class that submits jobs to a specific set of nodes?

Well, it seems there's a small hiccup in accessing the information directly from the sources. However, I can still guide you on setting up a DaskTaskRunner with SLURM and offer an idea of where to look next. For configuring

DaskTaskRunner

with SLURM, you'll typically make use of the

dask-jobqueue

package, which includes a

SLURMCluster

class. This class can customize job submissions via SLURM to specific nodes or other criteria. While Prefect handles the high-level orchestration, the specifics of where and how tasks run (like which nodes to use) are handled by configuring

SLURMCluster

in

dask-jobqueue

. You would typically adjust parameters like

job_extra

to control SLURM-specific options, such as node targeting. Here's a brief outline to get you started: 1. Install dask-jobqueue: Ensure this is installed in your environment to use SLURM with Dask. 2. SLURMCluster Configuration:

python
   from dask_jobqueue import SLURMCluster

   cluster = SLURMCluster(
       queue='my_queue',
       project='my_project',
       processes=6,
       cores=12,
       memory='24GB',
       job_extra=['--nodelist=your_specific_nodes']
   )

In the above, replace

'your_specific_nodes'

with your desired node names or patterns. 3. Integrate with Prefect: In your Prefect flow, use it similarly:

python
   from prefect import Flow
   from prefect_dask import DaskTaskRunner

   with Flow("my_flow") as flow:
       # Your tasks here

   flow.run(task_runner=DaskTaskRunner(cluster))

For more detailed and specific commands, please refer to the dask-jobqueue documentation. If I haven't brightened your day completely, try peeking at recent GitHub discussions or issues that might touch on emerging SLURM configurations within the Prefect community. Hopeful, right?