In prefect 2.0 the current agent solution isn't super price effective for a large number of queues. For instance if i use k8 autopilot in GCP I get charged per hour and since agents are always polling that gets pricey. I can run the agents in VM that is still expensive. Each agent running cost about 100-125 MB in memory. If i had 50 queues then i need 50 agents running which is about (50 * 150MB) we looking at a 8 GB VM to run these agents in). And double that if needed additional 50 queues. I would be looking at solution like cloud run (serverless 60 min max run time) to run the agents on. I was wondering if prefect would allow for queue watcher or alerting/webhook feature down the road?

Why would you need 50 queues? You can have only one queue and one agent. Plus, agent is just a single process. You can in theory run 50 agent processes on the same pod spun up with a supervisor (would be hard to operate but possible)

@Anna Geller So in my company basically we have this setup: A projects can have lots tasks that need to be processed by a python script project ---> tasks ---> scripts We can have a lot of projects going at once. I would want each dev that is working on a project to have his own work-queue that way if something goes wrong or they need to modify something they can easily go into the UI and pause there queue and make the changes without affecting anybody else. If we had single queue i would have to build some logic to filter the queue in case we want to pause all flow runs from a particular project.

I understand your problem but I don't think that a work queue is a right way to build staging environments for individual contributors in the team. An easier solution would be to allow each of those engineers to have either an individual workspace or an individual Cloud 2.0 account. This way, they can manage their sandbox environment themselves as needed without polluting the shared production/staging workspace with development flows definitely something you need to decide on your own based on your preference but I believe a workspace or even a local Orion instance as a sandbox environment seems like a more manageable setup than one work queue and agent per developer - perhaps you can build a PoC with that and decide what works best for your team?

Just curious is there a limitation to the number of queues i can create? I would assume sqlite has upper limit. I be more curious about the limit in prefect cloud

No limits