Hello everyone, I am using Prefect Cloud, I created one deployment for one work-pool which has one work queue.
Now in kubernetes, I created a Process Worker pod and I scaled it to 10 pods horizontally.
I put 10 as limit of flows should run on each worker.
Now
if I create 100 flows, so technically, all 10 prefect worker (process
type) pods should pick 10 flows and execute in paralelly.
But that is not happening.
There is any guide how to scale workers on single work-pool to execute flows parallelly?
k
Kevin Grismore
12/12/2023, 12:21 PM
If you set up a kubernetes worker rather than a process worker, each of your flow runs will be executed inside their own pod. Then the limits for scaling are determined by your cluster's configuration, like node and node pool sizes. https://docs.prefect.io/latest/guides/deployment/kubernetes/
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.