Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi folks, I am starting to consider using Prefect in a project, but one thing remains to be clarified : is that currently possible to specify the worker ressource requirement for a task running on a Dask cluster? E.g.: I have a flow with two tasks A and B. I would like to specify that A should run only on GPU workers while B should run on NO GPU workers. I did not find anything about that in the docs.

if I understand correctly, isn't Dask only used for the cloud offering?

Good news, Dask is available everywhere! For a tutorial, see <https://medium.com/the-prefect-blog/prefect-dask-2da6cdb23ee2>

<@UL3PR144V> it’s an excellent question. We’ve largely focused on affinity at the flow level (by using the `DaskExecutor` to specify a specific execution cluster). We have considered using task tags to allow affinity at the task level as well (basically reusing the task tags as Dask resources). However it’s something we’d like to make sure we support for users whether they use Dask or not, and we’re still iterating on the best way to enable that.

<@UKTUC906M> Thanks for the answer. So if I understand correctly there is no clean way to do it for now.

Is there an issue we can track in Github/etc for notifications on this feature?

I guess, I could also simply build my own flow runner, that would pass as keywords arguments the ressource requirements from the tasks. According to this (<https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/flow_runner.py#L418>) it would be done by passing ```ressources=task.ressources``` to the submit method from the executor
I would simply need to add the ressources at the Task definition level.

Hey everyone in this thread! <@UL3ENSGTZ> <@UL3PR144V> --&gt; this is now possible; see the docs here: <https://docs.prefect.io/api/unreleased/engine/executors.html#daskexecutor>