Aric Huang
07/08/2025, 11:52 PM"job_manifest": {
"kind": "Job",
"spec": {
"template": {
"spec": {
"containers": [
{
"env": "{{ env }}",
I am able to pass environment variables to the job parameters e.g. {"foo": "bar"}
. These seem to get placed into the kubernetes job as:
env:
- name: foo
value: bar
Now, I want to add some fixed kubernetes env vars using the valueFrom
field. For example:
env:
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
How do I support these in the work pool job template, without removing the ability to pass other environment variables?Marvin
07/08/2025, 11:52 PMMarvin
07/08/2025, 11:53 PMjson
"job_manifest": {
"kind": "Job",
"spec": {
"template": {
"spec": {
"containers": [
{
"env": [
{
"name": "K8S_NODE_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "spec.nodeName"
}
}
},
"{{ env }}"
]
}
]
}
}
}
}
The key points about this solution:
1. The env
field is now an array that contains both static and dynamic entries
2. The fixed valueFrom
environment variable is defined explicitly first
3. The "{{ env }}"
template variable is included as an array item, which Prefect will expand with any environment variables passed as job parameters
When you pass environment variables through job parameters (e.g., {"foo": "bar"}
), Prefect will properly merge them with your fixed environment variables. The resulting Kubernetes job will contain both types:
yaml
env:
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: foo
value: bar
You can add as many fixed valueFrom
environment variables as needed following this pattern. The dynamic variables passed through job parameters will always be added alongside your fixed configurations.
For more details on managing work pools and their configurations, you can refer to the work pools management guide and Kubernetes deployment guide.Aric Huang
07/09/2025, 12:10 AM"message":"Job.batch \"sigma22-idran-field-29hqb\" is invalid: spec.template.spec.containers[0].env[13].name: Required value","reason":"Invalid","details":{"name":"sigma22-idran-field-29hqb","group":"batch","kind":"Job","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.template.spec.containers[0].env[13].name"}]
additionally, there is a warning:
Warning': '299 - "unknown field \\"spec.template.spec.containers[0].env[13].MY_ENV\\""
which seems to indicate the additional env
fields are getting put into the template like this:
yaml
env:
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- MY_ENV: my_value
How to fix this?Marvin
07/09/2025, 12:10 AMMarvin
07/09/2025, 12:10 AMjson
"job_manifest": {
"kind": "Job",
"spec": {
"template": {
"spec": {
"containers": [
{
"env": [
{
"name": "K8S_NODE_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "spec.nodeName"
}
}
}
],
"envFrom": [
{
"configMapRef": {
"name": "{{ env }}"
}
}
]
}
]
}
}
}
}
However, this approach requires an additional step. You'll need to create a ConfigMap in your Kubernetes cluster that contains your dynamic environment variables before running the job.
Alternatively, you can try this more direct approach that should work with the current Prefect implementation:
json
"job_manifest": {
"kind": "Job",
"spec": {
"template": {
"spec": {
"containers": [
{
"env": [
{
"name": "K8S_NODE_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "spec.nodeName"
}
}
}
]
}
]
}
}
}
}
Then in your deployment configuration, you would specify your environment variables separately. These will be automatically merged with the template's environment variables by Prefect.
This is admittedly not an ideal solution, and there are open issues tracking improvements to this functionality:
- Issue #17041 - Extending Pod information to containers through environment variables
- Issue #15450 - Handling custom environment variable types in work pool templates
If you need more complex environment variable configurations, you might want to consider using Kubernetes secrets or ConfigMaps and referencing them in your job template, rather than trying to mix dynamic and static environment variables directly in the template.