I have a small question regarding dedicated execution of prefect tasks using a DaskExecutor. For example let's say I have some special tasks in a flow that need to be executed specifically on a dedicated machine. Is it possible to provide this information in Prefect?
Could you explain the problem you try to solve? Otherwise, I'll give you answers that will only confuse you further.
5 months ago
Yes I present to you Prefect’s least used feature!! Look at the DaskExecutor docs, you can use worker resources annotation to point a task to a specific worker. For example, you have a worker with a GPU
Hahah, Thank you! This solved the problem. I was experimenting with a much more difficult workaround of creating flows with a different executors on the dedicated machines for these tasks and then creating an orchestrating flow, but the Worker Resources are much simpler. Thank you!!🙏