While we are queuing up questions this morning, is there a way to direct which Dask resource a task needs? The obvious example is GPU or CPU. In my case, I have one task that needs a lot of memory and I don't want to set the memory limit on all the workers to that level. I'd just like to have one worker with a high memory limit and send that task to that worker. https://distributed.dask.org/en/latest/resources.html#worker-resources

@Nate Atkins This is absolutely possible with Prefect by tagging tasks with this convention:

"dask-resource:KEY=NUM"

(See: https://docs.prefect.io/api/latest/engine/executors.html#daskexecutor) We use this successfully for both GPUs and for what we call "High Memory" Dask workers, i.e. to execute Prefect tasks that need a lot of RAM. Note that to make this work from the Dask side will depend a lot on how you run a Dask cluster that Prefect executes on. We use a long-running Dask cluster on AWS EKS with different node groups for 3 types of Dask workers: "regular", GPU, and "high memory" and then scale those via Deployment replicas.

Ha, we were discussing the "dask-resource:KEY=NUM" yesterday and I just hadn't made the connection.

👍 Yeah, it's really cool. Here are the docs from the Dask side: https://distributed.dask.org/en/latest/resources.html