https://prefect.io logo
r

Romain

07/03/2019, 12:25 PM
Hi folks, I am starting to consider using Prefect in a project, but one thing remains to be clarified : is that currently possible to specify the worker ressource requirement for a task running on a Dask cluster? E.g.: I have a flow with two tasks A and B. I would like to specify that A should run only on GPU workers while B should run on NO GPU workers. I did not find anything about that in the docs.
o

Oli Steadman

07/03/2019, 2:16 PM
Wow I would be keen to know this too!
c

Chris Hart

07/03/2019, 3:15 PM
if I understand correctly, isn't Dask only used for the cloud offering?
j

Jeremiah

07/03/2019, 3:40 PM
Good news, Dask is available everywhere! For a tutorial, see https://medium.com/the-prefect-blog/prefect-dask-2da6cdb23ee2
👀 1
@Romain it’s an excellent question. We’ve largely focused on affinity at the flow level (by using the
DaskExecutor
to specify a specific execution cluster). We have considered using task tags to allow affinity at the task level as well (basically reusing the task tags as Dask resources). However it’s something we’d like to make sure we support for users whether they use Dask or not, and we’re still iterating on the best way to enable that.
r

Romain

07/04/2019, 6:56 AM
@Jeremiah Thanks for the answer. So if I understand correctly there is no clean way to do it for now.
o

Oli Steadman

07/04/2019, 8:59 AM
Is there an issue we can track in Github/etc for notifications on this feature?
r

Romain

07/04/2019, 9:10 AM
I guess, I could also simply build my own flow runner, that would pass as keywords arguments the ressource requirements from the tasks. According to this (https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/flow_runner.py#L418) it would be done by passing
Copy code
ressources=task.ressources
to the submit method from the executor I would simply need to add the ressources at the Task definition level.
c

Chris White

08/05/2019, 9:25 PM
Hey everyone in this thread! @Oli Steadman @Romain --> this is now possible; see the docs here: https://docs.prefect.io/api/unreleased/engine/executors.html#daskexecutor
💯 1
r

Romain

08/06/2019, 4:33 AM
@Chris White Wonderful. Thank you.
5 Views