https://prefect.io logo
Title
g

Greg Johnson

02/18/2020, 2:09 AM
Is there tutorial somewhere about how to specify "run one task per gpu" or "one task per worker" at a time? This mostly WRT running GPU jobs on a SLURM cluster
c

Chris White

02/18/2020, 4:08 AM
Hi Greg - thanks for the nice words!! Unfortunately there are no tutorials on this right now. If you are using the
DaskExecutor
with workers on the various machines, you can leverage dask worker resources to achieve this as described in the docstring here: https://docs.prefect.io/api/latest/engine/executors.html#daskexecutor
For more general concurrency limiting in the Prefect ecosystem (including limits across flows /flow runs), you either need to roll your own solution or leverage Prefect Cloud (note that concurrency limiting in Cloud is a licensed feature)