Chris L.
09/11/2022, 6:51 PM[ "python", "-m", "prefect.engine" ]
for KubernetesJob
18:49:52.809 | INFO | prefect.agent - Submitting flow run 'faa2ebac-ee94-4384-9a4f-16ad28d6fa7e'
....
File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 171, in __init__
super().__init__(*args, **kwargs)
File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for KubernetesJob
command
none is not an allowed value (type=type_error.none.not_allowed)
k8s_container = {
"name": "prefect-job",
"env": [
{
"name": "SECRET_API_KEY",
"valueFrom": {
"secretKeyRef": {
"name": "secret-auth",
"key": "password",
}
},
}
]
}
KubernetesJob(
namespace="default",
image=f"{os.environ['AWS_ECR_URL']}/prefect",
image_pull_policy=KubernetesImagePullPolicy.ALWAYS,
# Base job template:
# <https://github.com/PrefectHQ/prefect/blob/main/src/prefect/infrastructure/kubernetes.py#L142>
job={
"apiVersion": "batch/v1",
"kind": "Job",
"metadata": {"labels": {}},
"spec": {
"ttlSecondsAfterFinished": 1800,
"template": {
"spec": {
"restartPolicy": "Never",
"parallelism": 1,
"completions": 1,
"containers": [k8s_container],
"affinity": {
"nodeAffinity": {
"preferredDuringSchedulingIgnoredDuringExecution": [
{
"weight": 1,
"preference": {
"matchExpressions": [
{
"key": "nodegroup-name",
"operator": "In",
"values": ["compute"],
}
]
},
}
]
}
},
}
},
},
},
stream_output=True,
pod_watch_timeout_seconds=60,
job_watch_timeout_seconds=60,
)
KubernetesJob
block?command
if missing, but v2.2.0 does. Fixed after specifying the command explicitly!Anthony Head
09/11/2022, 11:07 PMNick Coy
09/14/2022, 8:46 PM