rania3
09/02/2025, 7:40 AMMarvin
09/02/2025, 7:40 AMMarvin
09/02/2025, 7:44 AMjob_variables.annotations
instead of `labels`—annotation values can be arbitrary strings.
- It’s best to supply lowercase keys/values in job_variables.labels
so what you see is what you get.
If this doesn’t match what you’re seeing (e.g., only values are lowercased, or specific keys), can you share:
- An example job_variables
snippet you set on the worker/pool
- The resulting labels on the Pod/Job
- Your Prefect and prefect-kubernetes versions
References:
- Kubernetes label syntax and character set: K8s docs
- Prefect Kubernetes integration overview: Prefect Kubernetes integrationrania3
09/02/2025, 8:11 AMjob_variables={ "labels": {
"jumbo_env": "test",
"jumbo_component": "LAE"
}}
and the pod labels
Pod Template:
Labels:
jumbo_component=lae
jumbo_env=test
Marvin
09/02/2025, 8:12 AMMarvin
09/02/2025, 8:13 AMjob_variables
. As a result, jumbo_component: "LAE"
becomes jumbo_component=lae
on the Pod. This is done to keep labels “safe” and consistent (DNS-1123–style), and the easiest way to guarantee that is to lowercase. While Kubernetes does allow uppercase characters in label values, many tooling paths and some prefixes (DNS subdomains) do not, so Prefect errs on the side of safety.
If you need to preserve case:
- Prefer annotations for case-sensitive data:
job_variables = {
"labels": {
"jumbo_env": "test",
"jumbo_component": "LAE"
},
"annotations": {
"jumbo_component_original": "LAE"
}
}
- Or encode the information in another place (e.g., environment variable) that doesn’t get normalized.
A couple of quick checks:
- Are you using the default Kubernetes worker job template, or a custom job template in your work pool? If it’s custom, look for any | lower
or slugify-like filters in the Jinja template that might be enforcing lowercase.
- Could you share your Prefect and prefect-kubernetes versions? If you’d like, I can open an issue/feature request to make this behavior configurable since Kubernetes label values can legally keep uppercase.
References:
- Kubernetes label syntax and character set: K8s docs
- Prefect Kubernetes integration overview: Prefect Kubernetes integrationrania3
09/02/2025, 8:15 AMMarvin
09/02/2025, 8:15 AMMarvin
09/02/2025, 8:20 AMprefect work-pool get-default-base-job-template -t kubernetes
In the template, {{ labels }}
is inserted directly (no | lower
).
Given that, the lowercasing you’re seeing likely comes from one of:
1) A custom work pool/job template that applies a Jinja filter (e.g., | lower
) to labels
2) A validator/normalizer in the kubernetes worker’s variables/config model in your installed version
Next steps to pinpoint it:
- Inspect your actual pool’s base job template to confirm whether it’s coming from Jinja:
prefect work-pool inspect "<your-pool-name>"
Look for any | lower
or “slugify”-style filters applied anywhere labels are rendered in the job manifest (metadata.labels and spec.template.metadata.labels).
- Share the versions you’re on:
prefect version
pip show prefect-kubernetes
With the version, I can map to the exact source and link you to the file/function if the normalization is happening in Prefect’s kubernetes worker code.
References:
- Default Kubernetes template: run the command above
- Kubernetes labels syntax: K8s docs
- Prefect Kubernetes integration: Prefect Kubernetes integration
If you paste the pool inspection output or your versions here, I’ll locate the exact spot in the code (file/function) and link it.rania3
09/02/2025, 12:40 PMprefect version
Version: 3.4.4
API version: 0.8.4
Python version: 3.12.10
Git commit: 0367d7aa
Built: Thu, May 29, 2025 09:40 PM
OS/Arch: linux/x86_64
Profile: ephemeral
Server type: ephemeral
Pydantic version: 2.11.5
Server:
Database: postgresql
Integrations:
prefect-kubernetes: 0.6.1
prefect-redis: 0.2.2
Marvin
09/02/2025, 12:40 PMMarvin
09/02/2025, 12:44 PMlabels
(you can verify with prefect work-pool get-default-base-job-template -t kubernetes
).
Most likely sources:
- A custom base job template on your pool that applies a Jinja filter (e.g., | lower
) on labels
- A template or transform in your environment that sanitizes metadata before submission
Can you paste the actual base job template your pool is using?
- Run:
prefect work-pool inspect "<your-pool-name>"
Then look for the job_configuration.job_manifest.metadata.labels
and ...spec.template.metadata.labels
sections to see if there’s any | lower
or other filters around {{ labels }}
.
If there’s no filter in the template, the next best step is to capture the rendered manifest the worker submits:
- Start your Kubernetes worker with debug logging and share the “Submitting job manifest”/“Rendered job manifest” log snippet. That will show whether the dict Prefect hands to Kubernetes already has lae
, or if something downstream is changing it.
If you can share:
- The work-pool inspect
output
- The debug log snippet with the rendered manifest
…I’ll trace it to the exact function/file and link you to the source. If it turns out Prefect is lowercasing values internally, we can file an issue/feature request to make this configurable since Kubernetes label values can legally contain uppercase.
References:
- Default Kubernetes template: run the command above
- Prefect Kubernetes integration: Prefect Kubernetes integration
- Kubernetes label syntax: K8s label rules