Hello, I would like to distribute tasks on differe...
# ask-community
t
Hello, I would like to distribute tasks on different servers/workers depending on several criteria (for example: GPU availability, available memory, location, ...) My original plan was to use a Dask cluster and the dask annotation system. However, it seems that it is not currently possible to use Dask annotations with an existing cluster. Would there be an alternative way of achieving the same result? If I understand correctly, I could set up different prefect work pools for different types of servers. But it will assign the whole flow to the same server (not per task) so I would need to replace each task with a subflow. Is that the proper way achieving this? I am thinking to schedule 100s of such tasks at once and I am unsure subflows are suited for this. Thank you!