Hello everyone, I am using Prefect Cloud, I create...
# prefect-cloud
m
Hello everyone, I am using Prefect Cloud, I created one deployment for one work-pool which has one work queue. Now in kubernetes, I created a Process Worker pod and I scaled it to 10 pods horizontally. I put 10 as limit of flows should run on each worker. Now if I create 100 flows, so technically, all 10 prefect worker (process type) pods should pick 10 flows and execute in paralelly. But that is not happening. There is any guide how to scale workers on single work-pool to execute flows parallelly?
k
If you set up a kubernetes worker rather than a process worker, each of your flow runs will be executed inside their own pod. Then the limits for scaling are determined by your cluster's configuration, like node and node pool sizes. https://docs.prefect.io/latest/guides/deployment/kubernetes/