<@ULVA73B9P> I am currently using a single work po...
# ask-marvin
j
@Marvin I am currently using a single work pool and worker in a Kubernetes cluster to run workflows. It has a default memory and CPU request and limit for each pod. I want to create another option for a deployment with a larger memory and CPU request and limit, for flows that are more memory or CPU-intensive. What's the best way to do so?
m
thought for 218.5 seconds
Great question — you’ve got two good patterns in Prefect 3.x with the Kubernetes worker to offer “standard” vs “high-resource” runs: 1) One pool, per-deployment overrides (simplest operationally) - Keep your existing Kubernetes work pool and worker. - Expose CPU/memory as variables in the pool’s base job template. - Set different
job_variables
per deployment. Steps: - Export and edit the base job template to add resource variables and wire them into the pod spec:
Copy code
prefect work-pool get-default-base-job-template --type kubernetes --file base-template.json
Add variables and inject them into the container resources (example snippet):
Copy code
{
  "variables": {
    "cpu_request": {
      "type": "string",
      "default": "100m"
    },
    "cpu_limit": {
      "type": "string",
      "default": "1000m"
    },
    "memory_request": {
      "type": "string",
      "default": "256Mi"
    },
    "memory_limit": {
      "type": "string",
      "default": "1Gi"
    }
  },
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "resources": {
                  "requests": {
                    "cpu": "{{ cpu_request }}",
                    "memory": "{{ memory_request }}"
                  },
                  "limits": {
                    "cpu": "{{ cpu_limit }}",
                    "memory": "{{ memory_limit }}"
                  }
                }
              }
            ]
          }
        }
      }
    }
  }
}
Update your pool:
Copy code
prefect work-pool update my-k8s-pool --base-job-template base-template.json
Then set per-deployment
job_variables
in your prefect.yaml:
Copy code
deployments:
  - name: standard-run
    entrypoint: flows/my_flow.py:main
    work_pool:
      name: my-k8s-pool
      job_variables:
        cpu_request: "200m"
        cpu_limit: "1000m"
        memory_request: "512Mi"
        memory_limit: "1Gi"

  - name: high-resource-run
    entrypoint: flows/my_flow.py:main
    work_pool:
      name: my-k8s-pool
      job_variables:
        cpu_request: "2000m"
        cpu_limit: "4000m"
        memory_request: "4Gi"
        memory_limit: "8Gi"
Pros: single worker, minimal infra changes. Cons: you must ensure those variables exist in the base template. 2) Two pools (resource tiers), two workers (clean isolation) - Create a second Kubernetes work pool with a base job template that defaults to higher resources. - Run a second worker pointing at that pool. - Point heavy deployments to the high-resource pool. Example: - Create a high-resource template file (as above) but with higher defaults, then:
Copy code
prefect work-pool create k8s-high --type kubernetes --base-job-template high-template.json
prefect worker start --pool k8s-high --type kubernetes
Point deployments to the right pool:
Copy code
deployments:
  - name: standard-run
    entrypoint: flows/my_flow.py:main
    work_pool:
      name: k8s-standard

  - name: high-resource-run
    entrypoint: flows/my_flow.py:main
    work_pool:
      name: k8s-high
Pros: very clear separation, easy defaults per tier, can target different nodes via nodeSelector/tolerations. Cons: you’ll run a second worker. Optional: on-demand overrides at run time - You can override resources for a single run (if the variables exist in the template):
Copy code
prefect deployment run my-flow/high-resource-run \
  --job-variable cpu_request=3000m \
  --job-variable memory_limit=12Gi
Recommendation - If you’re fine running a second worker, separate pools (“standard” and “high”) is the cleanest and gives you isolation and clear defaults. - If you prefer to keep one worker, add resource variables to the existing pool’s base template and set
job_variables
per deployment. Quick questions to tailor the guidance: - Do you prefer to keep a single worker, or are you okay running a second worker for a high-resource pool? - Do you already have a custom base job template for your current pool, or are you using the default?