https://prefect.io logo
g

Geese Howard

09/14/2023, 8:28 AM
Once I deploy a prefect-worker via helm chart, using value.yaml file, I have only one work-pool mapped to the newly generated worker. Is there any way to assign multiple workpools to a single worker? Here is the values.yaml file I have been using, but this one only activates gke-cpu-5000m-4Gi workpool.
Copy code
namespaceOverride: prefect
worker:
  cloudApiConfig:
    accountId: HIDE
    workspaceId: HIDE
  config:
    workPool: gke
    workPool: gke-cpu-5000m-4Gi
serviceAccount:
  create: false
  name: "HIDE"
1
c

Christopher Boyd

09/14/2023, 1:53 PM
no - one work pool per worker
if you want more work pools, you’d need to deploy separate workers
one per - I have dynamic / templated values.yaml in helm and CI/CD so they get interpreted based on some team values and/or branch (dev/stage/prod)
💯 1
g

Geese Howard

09/14/2023, 3:35 PM
Sounds reasonable. I was just wondering if that would make more resource wastage compared to agent-runner combinations, where only one single agent was managing all tasks. It seems like in workers case, there are needs for developing multiple workers for multiple work pools which leads to creation of multiple kubernetes pods. In case you are knowledgeable about this, can you let me know what's the philosophy behind this? I meant, why the brand new generation of agent (known as worker) looks more inefficient here?
c

Christopher Boyd

09/14/2023, 3:46 PM
Sure, workers are intended to be much more tightly coupled to the infrastructure. Some of these issues we saw with agents, was there was too much customization, and some users got themselves stuck trying to do something in one way that was un-intuitive. Workers couple infrastructure to the work-pool - that is, a Docker work pool handles, docker based flow runs, and all related configuration Kubernetes workers manage k8s work-pools, and you customize that base job template right on the worker. As a result, all flows / deployments that use that work-pool, will receive that configuration as a default, which you can overide / customize as needed. The advantage is it can separate configuration / infrastructure for someone who might just be a code author and know nothing about hte underlying infrastructure
The workers themselves aren’t very resource heavy, they are just polling for and submitting jobs for execution to the infrastructure
g

Geese Howard

09/14/2023, 3:49 PM
Ahh. I was thinking worker = agent + runner, but while following tutorials, I noticed workers are still able to create additional pods for job execution. So considering workers just poll and submit jobs with minimal resources, your answers make sense for me. Thanks!
🙌 1