https://prefect.io logo
#prefect-community
Title
# prefect-community
r

Rajvir Jhawar

07/04/2022, 1:29 PM
In prefect 2.0 the current agent solution isn't super price effective for a large number of queues. For instance if i use k8 autopilot in GCP I get charged per hour and since agents are always polling that gets pricey. I can run the agents in VM that is still expensive. Each agent running cost about 100-125 MB in memory. If i had 50 queues then i need 50 agents running which is about (50 * 150MB) we looking at a 8 GB VM to run these agents in). And double that if needed additional 50 queues. I would be looking at solution like cloud run (serverless 60 min max run time) to run the agents on. I was wondering if prefect would allow for queue watcher or alerting/webhook feature down the road?
1
a

Anna Geller

07/04/2022, 2:41 PM
Why would you need 50 queues? You can have only one queue and one agent. Plus, agent is just a single process. You can in theory run 50 agent processes on the same pod spun up with a supervisor (would be hard to operate but possible)
Serverless is definitely something we are exploring for the future and are open to hear more about your serverless use cases
r

Rajvir Jhawar

07/04/2022, 3:51 PM
@Anna Geller So in my company basically we have this setup: A projects can have lots tasks that need to be processed by a python script project ---> tasks ---> scripts We can have a lot of projects going at once. I would want each dev that is working on a project to have his own work-queue that way if something goes wrong or they need to modify something they can easily go into the UI and pause there queue and make the changes without affecting anybody else. If we had single queue i would have to build some logic to filter the queue in case we want to pause all flow runs from a particular project.
Having multiple work-queues all pointing to scalable single cluster is a lot easier to handle with multiple people: VM (agent 1 ...N) ---> prefect (queue 1..N) ---> single k8 cluster
a

Anna Geller

07/04/2022, 5:30 PM
I understand your problem but I don't think that a work queue is a right way to build staging environments for individual contributors in the team. An easier solution would be to allow each of those engineers to have either an individual workspace or an individual Cloud 2.0 account. This way, they can manage their sandbox environment themselves as needed without polluting the shared production/staging workspace with development flows definitely something you need to decide on your own based on your preference but I believe a workspace or even a local Orion instance as a sandbox environment seems like a more manageable setup than one work queue and agent per developer - perhaps you can build a PoC with that and decide what works best for your team?
also, keep in mind that your development flows can be triggered from a local machine without work queues or agents and still be tracked in Cloud 2.0 - work queues and agents are only needed for scheduled flows
r

Rajvir Jhawar

07/04/2022, 5:40 PM
Just curious is there a limitation to the number of queues i can create? I would assume sqlite has upper limit. I be more curious about the limit in prefect cloud
a

Anna Geller

07/04/2022, 6:18 PM
No limits
1
5 Views