https://prefect.io logo
m

Max Eggers

08/25/2023, 4:05 PM
I have a few questions about k8s workers: • How many flow runs can a given worker handle at a time? • Does that amount scale with the resources allocated to the worker? • What are pros/cons of increasing replica count of a given worker deployment against a work pool vs multiple worker deployments against a work pool? (assuming that multiple workers against a pool is supported and that concurrency limits are still respected) • If I have multiple k8s clusters, can I have workers in each cluster subscribe to a single work pool, pulling only work designated for the target cluster, while still respecting work pool and work queue concurrency limits?
Please let me know if there is a better forum to ask these
a

Alex Wilcoxson

08/25/2023, 4:10 PM
i

Islam Otmani

09/14/2023, 9:09 PM
Hi Max and Alex, sorry I just saw your questions. For #1, #2, and #3: Absolutely, if your worker has more resources, it will be able to run more flows at the same time. Adding concurrency limits here is important so not all your flows are taken by just one worker. Overall, it's worth trying to optimize your resources based on the nature of the flows you're running. e.g. If they're CPU bound, that's the bottle neck, and adding more memory won't help much. For #3 in particular, I'd opt for increasing replica count. Although they can both arguably be done to achieve the higher concurrency, I like to think of it as one for functional need, vs underlying infra. As for #4, I think you'll need separate work pools to achieve that if I understand it correctly. Same goes for @Alex Wilcoxson's question. The closest thing we support from what I can see is Work Queues, but that's more for prioritization rather than complete flow selection similar to Taints and Tolerations in Kubernetes.
I hope this helps
1
thank you 1
Having said that, I do think you both have a point in wanting to cherry pick worker vs flow at a finer granularity than just "topics", and tags can come in handy for this to let people do whatever they want kind of thing. So I'll keep this in mind and I can bring it up internally.
👍 1