<@ULVA73B9P> In Prefect 3.6.12 (self hosted server...
# ask-marvin
n
@Marvin In Prefect 3.6.12 (self hosted server), the Kubernetes jobs created by the workpool do not have the
<http://batch.kubernetes.io/job-name|batch.kubernetes.io/job-name>
and
job-name
labels. These are present on the Pod template, but not on the job itself. Why? How do I add them?
m
thought for 159.5 seconds
This is expected Kubernetes behavior, and it comes from how Prefect creates the Job. Why this happens In Prefect 3.6.12, the Kubernetes work pool's default Job manifest uses
metadata.generateName
, not
metadata.name
. That matters because Kubernetes only auto-populates
<http://batch.kubernetes.io/job-name|batch.kubernetes.io/job-name>
and
job-name
on the Job object when the Job has an explicit
name
. When a Job is created with
generateName
, Kubernetes generates the final name server-side, and those
job-name
labels are not added to the Job itself. So in short: - Prefect uses
generateName
by default
for K8s work pool jobs - Kubernetes does not add
job-name
labels to Jobs created that way
- The Pod template may still show those labels because labels are propagated there separately This is not specific to self-hosted Prefect Server; it's from the Kubernetes Job API behavior plus Prefect's default worker template design. Why Prefect uses
generateName
Prefect does this so each run gets a unique Job name automatically, avoiding collisions across runs/retries. How to add them You have two options: Option 1: Manually add labels to the Job metadata in the work pool job template Edit your work pool's base job template and add the labels under
job_configuration.job_manifest.metadata.labels
/ the Job
metadata.labels
section. Something like:
Copy code
yaml
metadata:
  annotations: "{{ annotations }}"
  labels:
    "{{ labels }}"
    <http://batch.kubernetes.io/job-name|batch.kubernetes.io/job-name>: "{{ name }}"
    job-name: "{{ name }}"
  namespace: "{{ namespace }}"
  generateName: "{{ name }}-"
This will put labels on the Job object, but note: - if you're still using
generateName
, the actual final Job name will likely be something like
my-flow-run-abc123
- your manual label value may not exactly match the final generated Kubernetes Job name unless you control naming more explicitly So this is fine if you just want consistent labels for selection/filtering, but not if you need them to exactly equal the final Job name. Option 2: Replace
generateName
with
name
If you want Kubernetes itself to manage those labels in the standard way, change the manifest to use
name
instead:
Copy code
yaml
metadata:
  annotations: "{{ annotations }}"
  labels: "{{ labels }}"
  namespace: "{{ namespace }}"
  name: "{{ name }}"
Then Kubernetes should add the
<http://batch.kubernetes.io/job-name|batch.kubernetes.io/job-name>
and
job-name
labels automatically. But there are tradeoffs: - you must ensure Job names are unique - retries/resubmissions can hit name conflicts - this is generally less flexible than Prefect's default
generateName
setup So for most Prefect users, Option 1 is safer unless you specifically need exact native Kubernetes
job-name
semantics. Recommendation If your goal is just to be able to filter/query Jobs by a stable label, add your own custom label instead, for example:
Copy code
yaml
metadata:
  labels:
    "{{ labels }}"
    <http://prefect.io/job-base-name|prefect.io/job-base-name>: "{{ name }}"
That avoids relying on Kubernetes' special
job-name
behavior while still giving you something predictable to query. If you want, I can help you draft the exact work pool job template update for your current Kubernetes worker config.