Hi everyone, my team have been using prefect for a couple years and now we're wanting to migrate to prefect cloud. However in our current setup we're letting one queue handle multiple deployments, so we have a code base with many modules that run at different times, each module having its own deployment. We're unsure of how to do this the right way in prefect cloud. Can someone help guide me the right way, as currently the only way I can get it to work is each deployment having its own queue.

Hi Mads, thanks for considering Prefect Cloud! That setup with a single queue for many deployments is a totally fine way to operate on both Prefect and Prefect Cloud. I'm assuming that all of your deployments run on the same infrastructure, so that should all be fine. If you do find that you're exceeding what your worker can handle, you can either add more workers to your work pool, or you can split up the work pools. Are you hitting errors or is that not otherwise working?

Thanks! We'd somehow missed the upgrade from agents to workers, after reading the documentation it seems like that could be the issue as we were facing, as we were trying to use an agent to interact with the work pool. We'll try this out and I'll report back, thanks again 😄

Just to clarify, you can run many simultaneous flow runs from a single worker as well, so you won't need multiple queues to accomplish that. Here's how I think about it: Work pool: you need one of these for each distinct configuration of infrastructure you want to run on. If you're targeting Kubernetes and all of your flow runs would have the same

Job

configuration, you only need one work pool. Work queue: every work pool comes with a

default

queue, and all work will be placed there by default. You only need multiple queues if you're running into scale limitations of your workers. Worker: you can run as many workers as you like pointing to one work pool and queue. So in your configuration, what if you start with one work pool, one queue, and 10 workers and see how that performs?

Ahh, thanks for the clarification, that's very helpful. We'll continue our migration to cloud and see if we can't get it working using your suggested setup, appreciate it! 🙂

Excellent, let us know how that goes