KK7NZY
11/13/2025, 8:54 PMMarvin
11/13/2025, 8:55 PMMarvin
11/13/2025, 9:05 PMjob_variables.
- Those variables must be defined in your work pool’s base job template. If the template doesn’t expose resource fields, you’ll need to add them once to the pool; then you can set them per deployment.
Helpful docs
- Customize job variables: Customize job variables
- Manage work pools and base job templates: Manage work pools
- Kubernetes worker guide: Kubernetes worker
- Docker worker guide: Docker worker
- Get default base job template (so you can customize it):
- prefect work-pool get-default-base-job-template --type kubernetes
- prefect work-pool get-default-base-job-template --type docker
Kubernetes worker: set container.resources.requests/limits
1) Update your pool’s base job template to expose resources.
- Start from the default template:
prefect work-pool get-default-base-job-template --type kubernetes > k8s-template.json
- Edit the first container to include a resources placeholder and add a matching variable in the template’s variables schema. Minimal example change (only showing the key parts):
k8s-template.json (snippet)
{
"job_configuration": {
"job_manifest": {
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "prefect-job",
"image": "{{ image }}",
"imagePullPolicy": "{{ image_pull_policy }}",
"args": "{{ command }}",
"env": "{{ env }}",
"resources": "{{ resources }}"
}
],
"serviceAccountName": "{{ service_account_name }}"
}
}
}
}
},
"variables": {
"properties": {
"resources": {
"type": [
"object",
"null"
],
"default": null,
"description": "Kubernetes container resources (requests/limits).",
"examples": [
{
"requests": {
"cpu": "500m",
"memory": "512Mi"
},
"limits": {
"cpu": "1",
"memory": "1Gi"
}
}
]
}
}
}
}
- Apply the updated template to your pool:
prefect work-pool create "my-k8s-pool" --type kubernetes --base-job-template ./k8s-template.json --overwrite
2) Set resources per deployment via Python or prefect.yaml
- Python deploy:
from prefect import flow
@flow
def my_flow():
...
my_flow.from_source(
source="<https://github.com/org/repo>",
entrypoint="flows/my_flow.py:my_flow",
).deploy(
name="k8s-with-resources",
work_pool_name="my-k8s-pool",
job_variables={
"resources": {
"requests": {"cpu": "500m", "memory": "512Mi"},
"limits": {"cpu": "1", "memory": "1Gi"}
}
},
)
- prefect.yaml deployment block:
deployments:
- name: k8s-with-resources
work_pool:
name: my-k8s-pool
entrypoint: flows/my_flow.py:my_flow
job_variables:
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"
Notes for Kubernetes
- The effective key path in the pod spec is: spec.template.spec.containers[0].resources.requests.cpu (and .memory) and resources.limits.cpu (.memory).Marvin
11/13/2025, 9:05 PMresources placeholder, per-deployment overrides will not take effect — add it once as shown above.
Docker worker: set memory and CPU per deployment
- The Docker worker exposes memory directly and CPU via container create kwargs. From the Docker worker configuration, these keys are available:
- mem_limit (e.g., "2g" or "2048m")
- memswap_limit (optional)
- container_create_kwargs (pass through to docker-py, e.g., nano_cpus, cpu_shares, etc.)
- Python deploy:
from prefect import flow
@flow
def my_flow():
...
my_flow.from_source(
source="<https://github.com/org/repo>",
entrypoint="flows/my_flow.py:my_flow",
).deploy(
name="docker-with-limits",
work_pool_name="my-docker-pool",
job_variables={
"mem_limit": "2g",
"container_create_kwargs": {
"nano_cpus": 2000000000 # ~2 CPUs
}
},
)
- prefect.yaml deployment block:
deployments:
- name: docker-with-limits
work_pool:
name: my-docker-pool
entrypoint: flows/my_flow.py:my_flow
job_variables:
mem_limit: "2g"
container_create_kwargs:
nano_cpus: 2000000000
CLI usage tip
- The most reliable path is to put job_variables in prefect.yaml and run:
prefect deploy
This avoids guessing CLI flags and ensures the values are captured in your deployment definition.
Would you confirm which worker you’re using (Kubernetes or Docker)? If it’s Kubernetes, I can share a complete, ready-to-paste base job template including the variables schema so you can drop it into your pool and start overriding per deployment.