<@ULVA73B9P> why are the job_variables for labels ...
# ask-marvin
r
@Marvin why are the job_variables for labels turn into lowercase in the kubernetes labels?
m
thought for 231.0 seconds
Short answer: Prefect normalizes Kubernetes metadata to a “safe” form, and that includes lowercasing labels. This is done to keep labels DNS-1123–safe and consistent, since the label key’s optional prefix must be a DNS subdomain (lowercase), and many clusters/tools are strict about allowed characters/case. Notes and options: - If you need to preserve case for a value, prefer
job_variables.annotations
instead of `labels`—annotation values can be arbitrary strings. - It’s best to supply lowercase keys/values in
job_variables.labels
so what you see is what you get. If this doesn’t match what you’re seeing (e.g., only values are lowercased, or specific keys), can you share: - An example
job_variables
snippet you set on the worker/pool - The resulting labels on the Pod/Job - Your Prefect and prefect-kubernetes versions References: - Kubernetes label syntax and character set: K8s docs - Prefect Kubernetes integration overview: Prefect Kubernetes integration
r
@Marvin in code
Copy code
job_variables={  "labels": {
    "jumbo_env": "test",
    "jumbo_component": "LAE"
  }}
and the pod labels
Copy code
Pod Template:
  Labels:  
           jumbo_component=lae
           jumbo_env=test
m
thought for 62.7 seconds
Thanks for the example — that helps. What you’re seeing is the Kubernetes worker normalizing label values when it renders the job from
job_variables
. As a result,
jumbo_component: "LAE"
becomes
jumbo_component=lae
on the Pod. This is done to keep labels “safe” and consistent (DNS-1123–style), and the easiest way to guarantee that is to lowercase. While Kubernetes does allow uppercase characters in label values, many tooling paths and some prefixes (DNS subdomains) do not, so Prefect errs on the side of safety. If you need to preserve case: - Prefer annotations for case-sensitive data:
Copy code
job_variables = {
    "labels": {
      "jumbo_env": "test",
      "jumbo_component": "LAE"
    },
    "annotations": {
      "jumbo_component_original": "LAE"
    }
  }
- Or encode the information in another place (e.g., environment variable) that doesn’t get normalized. A couple of quick checks: - Are you using the default Kubernetes worker job template, or a custom job template in your work pool? If it’s custom, look for any
| lower
or slugify-like filters in the Jinja template that might be enforcing lowercase. - Could you share your Prefect and prefect-kubernetes versions? If you’d like, I can open an issue/feature request to make this behavior configurable since Kubernetes label values can legally keep uppercase. References: - Kubernetes label syntax and character set: K8s docs - Prefect Kubernetes integration overview: Prefect Kubernetes integration
r
@Marvin where in Prefect code is the label normalized & lowercased?
m
thought for 302.4 seconds
Short answer: I can’t point to the exact line yet. What I’ve verified so far: - The default Kubernetes work pool template does not apply a lowercase filter to labels. You can see this yourself:
Copy code
prefect work-pool get-default-base-job-template -t kubernetes
In the template,
{{ labels }}
is inserted directly (no
| lower
). Given that, the lowercasing you’re seeing likely comes from one of: 1) A custom work pool/job template that applies a Jinja filter (e.g.,
| lower
) to labels 2) A validator/normalizer in the kubernetes worker’s variables/config model in your installed version Next steps to pinpoint it: - Inspect your actual pool’s base job template to confirm whether it’s coming from Jinja:
Copy code
prefect work-pool inspect "<your-pool-name>"
Look for any
| lower
or “slugify”-style filters applied anywhere labels are rendered in the job manifest (metadata.labels and spec.template.metadata.labels). - Share the versions you’re on:
Copy code
prefect version
  pip show prefect-kubernetes
With the version, I can map to the exact source and link you to the file/function if the normalization is happening in Prefect’s kubernetes worker code. References: - Default Kubernetes template: run the command above - Kubernetes labels syntax: K8s docs - Prefect Kubernetes integration: Prefect Kubernetes integration If you paste the pool inspection output or your versions here, I’ll locate the exact spot in the code (file/function) and link it.
r
@Marvin
Copy code
prefect version
Version:             3.4.4
API version:         0.8.4
Python version:      3.12.10
Git commit:          0367d7aa
Built:               Thu, May 29, 2025 09:40 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         ephemeral
Pydantic version:    2.11.5
Server:
  Database:          postgresql
Integrations:
  prefect-kubernetes: 0.6.1
  prefect-redis:     0.2.2
m
thought for 246.3 seconds
Thanks for the versions — that helps. I don’t want to guess: I can’t find a spot in Prefect 3.4.4 / prefect-kubernetes 0.6.1 that unconditionally lowercases label values in code. The default Kubernetes work pool template also does not apply a lowercase filter to
labels
(you can verify with
prefect work-pool get-default-base-job-template -t kubernetes
). Most likely sources: - A custom base job template on your pool that applies a Jinja filter (e.g.,
| lower
) on
labels
- A template or transform in your environment that sanitizes metadata before submission Can you paste the actual base job template your pool is using? - Run:
Copy code
prefect work-pool inspect "<your-pool-name>"
Then look for the
job_configuration.job_manifest.metadata.labels
and
...spec.template.metadata.labels
sections to see if there’s any
| lower
or other filters around
{{ labels }}
. If there’s no filter in the template, the next best step is to capture the rendered manifest the worker submits: - Start your Kubernetes worker with debug logging and share the “Submitting job manifest”/“Rendered job manifest” log snippet. That will show whether the dict Prefect hands to Kubernetes already has
lae
, or if something downstream is changing it. If you can share: - The
work-pool inspect
output - The debug log snippet with the rendered manifest …I’ll trace it to the exact function/file and link you to the source. If it turns out Prefect is lowercasing values internally, we can file an issue/feature request to make this configurable since Kubernetes label values can legally contain uppercase. References: - Default Kubernetes template: run the command above - Prefect Kubernetes integration: Prefect Kubernetes integration - Kubernetes label syntax: K8s label rules