Hi Christopher,
Thanks for your reply. I was under the impression that prefect 1 is legacy and only considered prefect 2.
For context, our use case is complex processing of very large images. Each image is processed in small chunks, which would constitute a task. There are different kinds of tasks, like computing optical flow between images, which is GPU bound, or rendering an image with a flow field, which is IO bound. Ideally, the IO bound tasks would not be executed on a GPU cluster.
At the same time, our "jobs" have complex dependencies between their subjobs. A simple example would be that optical flow for the whole image needs to be computed before the image is rendered. In my understanding, the way to represent such dependencies in prefect is with a flow, and this is not something that can be handled by the multi-agent approach. Is this correct? If so, are there any other ways to do what we want?