Hello all, I'm trying to evaluate if Perfect fits my company's needs. For our use case, we would have a flow that dispatches CPU and GPU tasks. Is it possible to easily switch between the following execution modes: 1. ALL tasks are scheduled to the same remote cluster 2. CPU and GPU tasks are scheduled to a CPU and GPU remote clusters respectively 3. All tasks are executed locally Documentation that I'm finding suggests that there will be always a single task runner per flow, and that the task runner is specified at flow definition time, ie I cannot change execution mode without modifying code.

Hi Sergiy, What kind of clusters are we discussing here ? You can have an agent that submits flows / tasks to a cpu based cluster in one namespace, and another agent in another namespace that submits to a gpu backed cluster

Hi Christopher, Thanks for your reply. I was under the impression that prefect 1 is legacy and only considered prefect 2. For context, our use case is complex processing of very large images. Each image is processed in small chunks, which would constitute a task. There are different kinds of tasks, like computing optical flow between images, which is GPU bound, or rendering an image with a flow field, which is IO bound. Ideally, the IO bound tasks would not be executed on a GPU cluster. At the same time, our "jobs" have complex dependencies between their subjobs. A simple example would be that optical flow for the whole image needs to be computed before the image is rendered. In my understanding, the way to represent such dependencies in prefect is with a flow, and this is not something that can be handled by the multi-agent approach. Is this correct? If so, are there any other ways to do what we want?

Hi Sergiy, my experience lends itself more to the infra side of the house, and slightly less on the Prefect use cases, so if any of my colleagues have some insight here, they are more than welcome to chime in! What I have seen is a scenario where a customer has an EKS cluster, with two auto scaling groups. one ASG uses GPU provisioned instances one ASG use CPU provisioned instances When the flows / tasks are submitted for execution, they can be tagged so the appropriate agent picks them up for execution. What you are describing does seem like a flow of flows, and I do believe doable, although I’d have to review the specifics of how something like that could be implemented

Thanks Christopher, this is incredibly insightful. If I can get my tasks to be scheduled to the right ASG based on a tag, it would be exactly what I want. Is this something readily available for existing TaskRunners? I see how to tag a task, but I couldn't find any mention of it in the TaskRunner documentation or `prefect-dask`/`prefect-ray` repos

I’d have to review at the task level if that’s possible, I know you can tag your flows / flow-runs to be picked up by a specific agent