In the resource annotation, if my machine has 4 GP...
# best-practicesz
Zheyuan
08/11/2022, 5:33 AMIn the resource annotation, if my machine has 4 GPUs, how do I know which GPU is currently being used by the worker if I use a temporary cluster? Is the temporary worker automatically assigned GPUs?
https://prefecthq.github.io/prefect-dask/
Zheyuan
08/11/2022, 5:40 AMThis is important for running external programs.
We usually need to provide a GPU id to specify the GPU used by the program, and we need to figure out how the temporary cluster assigns the GPU to the task. Do we need to tell the task which GPU id to use?
a
Anna Geller
08/11/2022, 10:14 AM
Interesting. I wonder whether Dask is the right solution to your problem. I wonder whether you might be better off either spinning up separate agents on VMs with GPUs, or even easier using Kubernetes agent with labels indicating GPU to use. Both seems to be less of a blackbox in terms of allocation of your work to the right compute you need
Anna Geller
08/11/2022, 10:14 AM
If you're running on AWS, I can recommend Karpenter which makes it easy to provision the right GPUs for your runs
z
Zheyuan
08/11/2022, 11:37 AM@Anna Geller Thanks Anna. Your suggestion has completely discouraged me from using temporary clusters. Using an existing cluster seems to be the only achievable solution.ðŸ˜
a
Anna Geller
08/11/2022, 11:46 AM
Maybe not only but definitely the easiest. Karpenter can help you spin up GPU compute on Kubernetes for workflows that need it simply by assigning the right labels on your Prefect KubernetesJob block
Anna Geller
08/11/2022, 11:46 AM
And it can spin it down when they are no longer needed too
28 Views