https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> how can i limit and equest the ressou...
# ask-marvin
o
@Marvin how can i limit and equest the ressources of spawened batch jobs of a deployment?
m
✅ thought for 173.1 seconds
If by “spawned batch jobs” you mean the worker-created infrastructure for a deployment, the answer is: you set resource requests/limits with 
job_variables
on the deployment (or defaults on the work pool). The exact fields depend on the work pool type. Most common case: Kubernetes work pool Use these deployment `job_variables`: - 
cpu_request
- 
cpu_limit
- 
memory_request
- 
memory_limit
Example in `prefect.yaml`:
Copy code
yaml
deployments:
  - name: my-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-kubernetes-pool
      job_variables:
        cpu_request: "500m"
        cpu_limit: "1"
        memory_request: "512Mi"
        memory_limit: "1Gi"
Or in Python:
Copy code
python
from prefect import flow

@flow
def my_flow():
    print("hello")

if __name__ == "__main__":
    my_flow.deploy(
        name="my-deployment",
        work_pool_name="my-kubernetes-pool",
        job_variables={
            "cpu_request": "500m",
            "cpu_limit": "1",
            "memory_request": "512Mi",
            "memory_limit": "1Gi",
        },
    )
A few important notes: - 
request
= what Kubernetes reserves / schedules against - 
limit
= hard max the container can use - values must use Kubernetes-style units like 
"500m"
, 
"2Gi"
, 
"256Mi"
Where to put this You have 2 main options: 1. Work pool defaults Set default values in the work pool’s base job template / variables. 2. Deployment override Set 
work_pool.job_variables
in 
prefect.yaml
or 
job_variables=
in 
.deploy()
. Deployment-level values override the pool defaults. Other work pool types If you are not using Kubernetes, the resource fields differ: - ECS work pool: typically 
cpu
and 
memory
- Azure Container Instance: typically 
cpu
and 
memory
- Vertex AI: usually resource sizing is via 
machine_type
rather than k8s-style requests/limits So the right answer depends on your infrastructure. Useful references - Prefect deployments: Deployments docs - Work pools: Work pools docs - Kubernetes integration: prefect-kubernetes docs If you want, tell me which work pool type you use (
kubernetes
, 
ecs
, 
aci
, etc.) and I can give you the exact config snippet for your deployment.
o
@Marvin yeah i did this but the batch job is consuming 25 GB { "env": { "PREFECT_API_URL": "http://prefect/api", }, "cpu_limit": "512m", "namespace": "test-v7", "cpu_request": "128m", "memory_limit": "1Gi", "memory_request": "128Mi", "finished_job_ttl": 3600, "image_pull_policy": "Always" }
m
✅ thought for 134.5 seconds
That strongly suggests one of these is happening: 1. the Kubernetes pod never actually got those limits applied, or 2. you’re looking at node/pod memory usage instead of the main Prefect container’s limit/usage, or 3. there’s another container in the pod/job consuming memory. With your config, a container with 
memory_limit: "1Gi"
should typically be OOM-killed before it can truly use 25 GB. Most likely cause: the work pool’s base job template is not referencing those variables, so your 
job_variables
are being accepted by Prefect but not injected into the actual Job manifest. Your variables look correct:
Copy code
json
{
  "cpu_limit": "512m",
  "cpu_request": "128m",
  "memory_limit": "1Gi",
  "memory_request": "128Mi"
}
But they only work if the work pool template contains something like:
Copy code
json
"resources": {
  "limits": {
    "cpu": "{{ cpu_limit }}",
    "memory": "{{ memory_limit }}"
  },
  "requests": {
    "cpu": "{{ cpu_request }}",
    "memory": "{{ memory_request }}"
  }
}
First thing to check Inspect the work pool:
Copy code
bash
prefect work-pool inspect YOUR_POOL_NAME --output json
Look at 
base_job_template.job_configuration.job_manifest
and confirm the 
resources
block is present for the 
prefect-job
container. You can also inspect the actual Kubernetes Job/Pod that Prefect created:
Copy code
bash
kubectl get jobs -n test-v7
kubectl get job <job-name> -n test-v7 -o yaml
kubectl get pod <pod-name> -n test-v7 -o yaml
Then check this exact path in the pod spec:
Copy code
bash
kubectl get pod <pod-name> -n test-v7 -o jsonpath='{.spec.containers[0].resources}'
You should see something like:
Copy code
json
{
  "limits": {
    "cpu": "512m",
    "memory": "1Gi"
  },
  "requests": {
    "cpu": "128m",
    "memory": "128Mi"
  }
}
If you do not see that, Prefect is not rendering those values into the manifest. Also check actual container usage, not just pod/node totals
Copy code
bash
kubectl top pod <pod-name> -n test-v7 --containers
That helps distinguish: - the main Prefect container - any sidecar/service-mesh container - total pod usage If you have something like Istio/Linkerd or another injected sidecar, that can explain surprising memory numbers. Important detail In Prefect’s Kubernetes worker, those resource fields apply to the main 
prefect-job
container. If your cluster injects sidecars, those sidecars need their own limits too. What I’d verify in order 1. Work pool template contains 
{{ memory_limit }}
and friends 2. Created pod spec actually shows those limits 3. You’re measuring container usage, not node usage 4. No sidecar/init container is unbounded Useful commands
Copy code
bash
prefect work-pool inspect YOUR_POOL_NAME --output json
Copy code
bash
kubectl get pod <pod-name> -n test-v7 -o yaml
Copy code
bash
kubectl top pod <pod-name> -n test-v7 --containers
If you want, paste the output of:
Copy code
bash
prefect work-pool inspect YOUR_POOL_NAME --output json
or just the 
base_job_template
section, and I can tell you immediately whether the resource settings are actually wired up.
o
@Marvin sudo kubectl get pod aloof-marmoset-n9w7t-xnh8k -n um-demo-v6 -o jsonpath='{.spec.containers[0].resources}' {} sudo kubectl get pod fascinating-serval-s9pt4-fx8fw -n um-demo-v6 -o jsonpath='{.spec.containers[0].resources}' {} sudo kubectl get pod prefect-worker-68fb485895-dpdwh -n um-demo-v6 -o jsonpath='{.spec.containers[0].resources}' {"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"100m","memory":"256Mi"}}
m
✅ thought for 12.3 seconds
Yep — that confirms it. Your worker pod has resources set, but the flow-run pods / batch job pods have:
Copy code
json
{}
So the spawned jobs are being created without any container resource requests/limits. That means your deployment 
job_variables
are not being applied to the Kubernetes Job manifest. Why this happens For Kubernetes work pools, 
job_variables
only matter if the work pool’s 
base_job_template
includes those variables in the job manifest, e.g.:
Copy code
json
"resources": {
  "limits": {
    "cpu": "{{ cpu_limit }}",
    "memory": "{{ memory_limit }}"
  },
  "requests": {
    "cpu": "{{ cpu_request }}",
    "memory": "{{ memory_request }}"
  }
}
If that block is missing from the pool template, Prefect will accept:
Copy code
json
{
  "cpu_limit": "512m",
  "cpu_request": "128m",
  "memory_limit": "1Gi",
  "memory_request": "128Mi"
}
but the created pod will still have:
Copy code
json
{}
because nothing in the template references those values. What to do next Run:
Copy code
bash
prefect work-pool inspect YOUR_POOL_NAME --output json
and inspect:
Copy code
text
base_job_template.job_configuration.job_manifest.spec.template.spec.containers
You will likely find that the container spec has no 
resources
section. Fix Update the Kubernetes work pool’s base job template so the main container includes the resource placeholders. It should look roughly like this:
Copy code
json
{
  "job_configuration": {
    "job_manifest": {
      "apiVersion": "batch/v1",
      "kind": "Job",
      "metadata": {
        "namespace": "{{ namespace }}",
        "generateName": "{{ name }}-"
      },
      "spec": {
        "ttlSecondsAfterFinished": "{{ finished_job_ttl }}",
        "template": {
          "spec": {
            "restartPolicy": "Never",
            "serviceAccountName": "{{ service_account_name }}",
            "containers": [
              {
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}",
                "args": "{{ command }}",
                "env": "{{ env }}",
                "resources": {
                  "requests": {
                    "cpu": "{{ cpu_request }}",
                    "memory": "{{ memory_request }}"
                  },
                  "limits": {
                    "cpu": "{{ cpu_limit }}",
                    "memory": "{{ memory_limit }}"
                  }
                }
              }
            ]
          }
        }
      }
    }
  }
}
Also make sure those variables exist in the template’s 
variables
schema: - 
cpu_request
- 
cpu_limit
- 
memory_request
- 
memory_limit
Two important notes 1. This is a work pool configuration issue, not a deployment issue Your deployment can send the variables, but the pool template decides whether they are used. 2. The worker pod resources are unrelated to flow-run pod resources The 
prefect-worker-...
pod having limits does not constrain the jobs it launches. Docs / references - Work pools: Prefect work pools - Kubernetes integration: prefect-kubernetes If you want, paste the output of:
Copy code
bash
prefect work-pool inspect YOUR_POOL_NAME --output json
and I can help you build the exact corrected 
base_job_template
for your pool.
o
@Marvin lso make sure those variables exist in the template’s 
variables
schema: - 
cpu_request
- 
cpu_limit
- 
memory_request
- 
memory_limit
m
✅ thought for 9.6 seconds
Yes — exactly. You need both of these: 1. the variables defined in 
base_job_template.variables
2. those variables referenced in 
base_job_template.job_configuration.job_manifest
If either part is missing, the limits won’t show up in the spawned pod. Example shape:
Copy code
json
{
  "variables": {
    "properties": {
      "cpu_request": {
        "title": "CPU Request",
        "type": "string",
        "default": "100m"
      },
      "cpu_limit": {
        "title": "CPU Limit",
        "type": "string",
        "default": "1"
      },
      "memory_request": {
        "title": "Memory Request",
        "type": "string",
        "default": "128Mi"
      },
      "memory_limit": {
        "title": "Memory Limit",
        "type": "string",
        "default": "1Gi"
      }
    },
    "type": "object"
  },
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "resources": {
                  "requests": {
                    "cpu": "{{ cpu_request }}",
                    "memory": "{{ memory_request }}"
                  },
                  "limits": {
                    "cpu": "{{ cpu_limit }}",
                    "memory": "{{ memory_limit }}"
                  }
                }
              }
            ]
          }
        }
      }
    }
  }
}
Rule of thumb: - Defined in 
variables
only → not enough - Referenced in manifest only but not declared properly → may fail validation / not be configurable - Need both Since your pod shows:
Copy code
json
{}
the missing piece is almost certainly the manifest reference, and possibly the variable definitions too. If you paste your work pool’s 
base_job_template
, I can point out the exact missing lines.
o
@Marvin how to change the base job template with helm deploy
m
✅ thought for 18.2 seconds
If you created the work pool via Helm / worker chart, the key point is: Helm deploys the worker, but the work pool’s 
base_job_template
lives in Prefect, not in Kubernetes. So changing the Helm values for the worker deployment usually does not retroactively update the Prefect work pool template unless your chart/run setup is explicitly creating/updating the pool with those values. In practice, you usually have 2 options: ### Option 1: Update the work pool in Prefect directly This is the most reliable route. Inspect current pool:
Copy code
bash
prefect work-pool inspect YOUR_POOL_NAME --output json
Then update the pool’s 
base_job_template
in Prefect Cloud / Server. If you use the UI: - Go to Work Pools - Open your pool - Edit the Advanced / Base job template section - Add the 
resources
block and variable schema entries ### Option 2: Manage the pool definition through Helm values If your Helm chart is creating the work pool, you need to check whether the chart supports passing a custom 
baseJobTemplate
or similar values into the worker startup configuration. This depends on which Prefect Helm chart/version you’re using. In general, the worker chart can: - start a worker deployment in Kubernetes - optionally create a work pool - sometimes accept work pool metadata/settings But the actual 
base_job_template
support is chart-version-specific. So I’d want to know: - which Helm chart you’re using - your 
values.yaml
- whether the work pool is being auto-created by the chart or created separately Important distinction These are separate things: 1. Worker Kubernetes Deployment Controlled by Helm Affects 
prefect-worker-...
pod resources 2. Work pool base job template Stored in Prefect API Affects spawned flow-run job pods You already proved #1 is configured and #2 is not. Common pattern if using Helm People set worker pod resources in Helm like this:
Copy code
yaml
worker:
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 1
      memory: 1Gi
But that only affects the worker pod itself, not the jobs it launches. To affect launched jobs, you need the work pool 
base_job_template
to include:
Copy code
json
"resources": {
  "requests": {
    "cpu": "{{ cpu_request }}",
    "memory": "{{ memory_request }}"
  },
  "limits": {
    "cpu": "{{ cpu_limit }}",
    "memory": "{{ memory_limit }}"
  }
}
Best next step Please paste: - your Helm chart name/version - the relevant part of your 
values.yaml
Then I can tell you whether: - Helm can manage this directly, or - you should patch the work pool after deploy If you want a quick path right now, the safest answer is: edit the work pool’s base job template in Prefect, not Helm. Useful docs: - Helm chart repo: prefect-helm - Work pools: Work pools docs
o
@Marvin observerClusterWide: false ## If unspecified, Prefect will use the default base job template for the given worker type. If the work pool already exists, this will be ignored. ## e.g.: ## baseJobTemplate: ## configuration: | ## { ## "variables": { ## ... ## }, ## "job_configuration": { ## ... ## } ## } ## OR ## baseJobTemplate: ## existingConfigMapName: "my-existing-config-map" baseJobTemplate: # -- the name of an existing ConfigMap containing a base job template. NOTE - the key must be 'baseJobTemplate.json' existingConfigMapName: "" # -- JSON formatted base job template. If data is provided here, the chart will generate a configmap and mount it to the worker pod configuration: null # -- optionally override the default name of the generated configmap # name: ""
m
✅ thought for 18.3 seconds
Yes — that Helm block is the relevant one. The important line is this comment: “If the work pool already exists, this will be ignored.” So with Helm: - 
worker.resources
configures the worker pod - 
worker.baseJobTemplate.configuration
configures the work pool base job template - but only when the pool is created by the chart - if the pool already exists in Prefect, changing Helm values alone won’t update it That matches exactly what you’re seeing. ### What to do #### If the work pool already exists You need to either: 1. delete and recreate the work pool with the Helm-managed template, or 2. edit the work pool directly in Prefect If you keep the same existing pool, Helm will not overwrite its 
base_job_template
. --- ### How to set it in Helm You can set:
Copy code
yaml
baseJobTemplate:
  configuration: |
    {
      "variables": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string"
          },
          "namespace": {
            "type": "string",
            "default": "um-demo-v6"
          },
          "env": {
            "type": "object",
            "additionalProperties": {
              "type": "string"
            }
          },
          "image": {
            "type": "string"
          },
          "image_pull_policy": {
            "type": "string",
            "default": "Always"
          },
          "finished_job_ttl": {
            "type": "integer",
            "default": 3600
          },
          "cpu_request": {
            "type": "string",
            "default": "128m"
          },
          "cpu_limit": {
            "type": "string",
            "default": "512m"
          },
          "memory_request": {
            "type": "string",
            "default": "128Mi"
          },
          "memory_limit": {
            "type": "string",
            "default": "1Gi"
          },
          "command": {
            "type": "string"
          }
        }
      },
      "job_configuration": {
        "job_manifest": {
          "apiVersion": "batch/v1",
          "kind": "Job",
          "metadata": {
            "namespace": "{{ namespace }}",
            "generateName": "{{ name }}-"
          },
          "spec": {
            "ttlSecondsAfterFinished": "{{ finished_job_ttl }}",
            "template": {
              "spec": {
                "restartPolicy": "Never",
                "containers": [
                  {
                    "name": "prefect-job",
                    "image": "{{ image }}",
                    "imagePullPolicy": "{{ image_pull_policy }}",
                    "env": "{{ env }}",
                    "args": "{{ command }}",
                    "resources": {
                      "requests": {
                        "cpu": "{{ cpu_request }}",
                        "memory": "{{ memory_request }}"
                      },
                      "limits": {
                        "cpu": "{{ cpu_limit }}",
                        "memory": "{{ memory_limit }}"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      }
    }
That’s the core idea: define the variables and wire them into the manifest. --- ### But be careful A full base job template often includes more fields than this minimal example, such as: - labels / annotations - service account name - image pull secrets - finished job ttl - namespace - command formatting - cluster config / in-cluster settings So the safest approach is: 1. get the current default Kubernetes base job template 2. add the 
resources
block and resource variables 3. use that complete JSON in 
baseJobTemplate.configuration
--- ### Why your Helm change may not work immediately Because the chart explicitly says: If the work pool already exists, this will be ignored. So even if you add the right JSON to 
values.yaml
, you may still see:
Copy code
json
{}
until you either:
- recreate the work pool, or - manually update the existing one in Prefect --- ### Recommended rollout 1. Add the correct 
baseJobTemplate.configuration
to Helm 2. Create a new work pool name so Helm creates a fresh pool 3. Point your deployment at the new pool 4. Run a test 5. Verify:
Copy code
bash
kubectl get pod <pod-name> -n um-demo-v6 -o jsonpath='{.spec.containers[0].resources}'
You should then see the requests/limits instead of 
{}
. --- ### Best practical fix If you want the least risky approach: - create a new Kubernetes work pool with a new name via Helm - include the fixed 
baseJobTemplate
- update your deployments to use that new pool That avoids fighting the “existing pool will be ignored” behavior. If you want, I can help you build a complete Helm 
baseJobTemplate.configuration
 from the Prefect Kubernetes default so you don’t accidentally omit required fields.
Open in Slack
PreviousNext
Powered by