Jake Schmidt
01/20/2020, 1:40 PMJoe Schmid
01/20/2020, 2:43 PM@task(tags=["dask-resource:GPU=1"])
def task_that_uses_gpu():
and then the relevant YAML section for our k8s GPU workers:
containers:
- args:
- dask-worker
- dask-scheduler:8786
- --resources
- "GPU=1"
Jackson Maxfield Brown
01/20/2020, 4:33 PMJoe Schmid
01/20/2020, 4:40 PMresources
and passing that parameter the value GPU=1
Prefect can then use task tagging (the other snippet I showed) to route tasks only to Dask workers that have appropriate resources. It's really powerful and has been very successful for us.Jackson Maxfield Brown
01/20/2020, 4:43 PMJoe Schmid
01/20/2020, 4:52 PMJackson Maxfield Brown
01/20/2020, 4:59 PM