Hi all, I’m trying to figure out how to create a workflow that uses multiple worker pools. Let’s say I have a batch of 1000 work items. Each item needs to be processed in a high-memory worker, then its result needs to be processed in a GPU worker. What’s the best way to represent this in Prefect? I’ve looked over the docs, Slack history, GitHub issues, and asked ChatGPT, but I’m struggling to figure out out how to combine flows, subflows, tasks, deployments, and worker pools into what I’m looking for. Can you point me in the right direction? Thank you

Hey @Cole Erickson, great question! The best way to use specific infrastructure for different work would be to create deployments for each piece of work requiring a different infrastructure. Those deployments can each be created with different work pool or different job variables to customize what type of infrastructure is used for flow runs created for that deployment. You can trigger them in the order you need by deploying an orchestrator flow that triggers each sub-deployment with the

run_deployment

utility. That was a lot, so I’m happy to clarify further if you have any questions!

Thanks, I'll give this a shot!

Hi @alex, thanks for the tip. This seems the way to go for us as well. Do you have any examples of an orchestrator flow that triggers sub-deployments with

run_deployment

?

Yes! Here’s an example of the orchestrator pattern from our recipes repo: https://github.com/PrefectHQ/prefect-recipes/tree/main/flows-advanced/parent-orchestrator

Thanks!