Jake Schmidt01/20/2020, 1:40 PM
Joe Schmid01/20/2020, 2:43 PM
and then the relevant YAML section for our k8s GPU workers:
@task(tags=["dask-resource:GPU=1"]) def task_that_uses_gpu():
containers: - args: - dask-worker - dask-scheduler:8786 - --resources - "GPU=1"
Jackson Maxfield Brown01/20/2020, 4:33 PM
Joe Schmid01/20/2020, 4:40 PM
and passing that parameter the value
Prefect can then use task tagging (the other snippet I showed) to route tasks only to Dask workers that have appropriate resources. It's really powerful and has been very successful for us.
Jackson Maxfield Brown01/20/2020, 4:43 PM
Joe Schmid01/20/2020, 4:52 PM
Jackson Maxfield Brown01/20/2020, 4:59 PM