Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all,
I'm still on prefect 2.16, self-hosted and running kubernetes agent. Looking forward to migration to prefect 3, but as first step consider migrating to worker pool (agent is deprecated in prefect 3, right?). The question is - is it a normal practice to have worker pool per deployment? I have around 1000 deployments, each running different image + tons of different k8s configurations (config maps, secrets, resource limits etc). All served by single k8s agent, each deployment declares own job infrastructure with custom k8s settings. I would end up with 1000 work pools, is that ok?

Hi VK! Work pools and deployments don't have to be 1:1. Typically we recommend having one work pool per environment. In your case, I believe that it will be one work pool for your k8s cluster.

The work pool can have all of the default configurations you want for your jobs, but you if need to override those defaults, you can on a per-deployment basis (with job_variables)!

• *<https://docs-3.prefect.io/3.0/resources/upgrade-agents-to-workers|Upgrade from agents to workers>*
• *<https://docs.prefect.io/3.0/deploy/infrastructure-examples/kubernetes#create-a-cluster|Run flows on Kubernetes>*

^ If you haven't seen these already, these guides should help