Robert Denham
03/23/2023, 6:28 AMttlSecondsAfterFinished: 100
template:
spec:
dnsPolicy: Default
completions: 1
containers: # the first container is required
- env: [] # env is required, even if empty
name: prefect-job
parallelism: 1
restartPolicy: Never
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: project
operator: In
values:
- fcv3
I was just wondering how I go about modifying that section, to, for example, use a different key , different values. I want to use this in the deployment, so using infra_overrides
The example override looks like:
infra_overrides={
"customizations": [
{
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {
"requests": {
"cpu": "2000m",
"memory": "4gi"
},
"limits": {
"cpu": "4000m",
"memory": "8Gi",
"<http://nvidia.com/gpu|nvidia.com/gpu>": "1"
}
},
}
]
}
what would the equivalent be in my case?
I had a stab at it, like (actually trying to add two over-rides, an add and a replace)
aws_infra_overrides = {
"customizations": [
{
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
},
{
"op": "replace",
"path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0/",
"value'":{"key":"location", 'operator':'In', 'values':"aws"},
}
]
}
redsquare
03/23/2023, 7:37 AMRobert Denham
03/23/2023, 7:38 AMdeployment = Deployment.build_from_flow(
flow=fc_flow,
name="fcv3 thurs",
version=2,
description="""Create the input images""",
tags=["test run"],
work_pool_name="rss-pool",
infrastructure=k8block,
storage=s3_block,
parameters={"satid": "fill_me_in"},
infra_overrides=aws_infra_overrides,
)
deployment.apply()
redsquare
03/23/2023, 7:39 AMRobert Denham
03/23/2023, 7:39 AM16:21:15.031 | ERROR | prefect.agent - Failed to submit flow run '722902ed-b25d-4b50-bb94-cb6c96d199c8' to infrastructure.
Traceback (most recent call last):
File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 319, in apply
value = self.operation["value"]
KeyError: 'value'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/agent.py", line 484, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 299, in run
manifest = self.build_job()
File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 359, in build_job
job_manifest = self.customizations.apply(job_manifest)
File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 669, in apply
obj = operation.apply(obj)
File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 321, in apply
raise InvalidJsonPatch(
jsonpatch.InvalidJsonPatch: The operation does not contain a 'value' member
redsquare
03/23/2023, 7:40 AMRobert Denham
03/23/2023, 7:42 AMjsonpatch.InvalidJsonPatch: The operation does not contain a 'value' member
redsquare
03/23/2023, 7:44 AMRobert Denham
03/23/2023, 7:45 AMaws_infra_overrides = {
"customizations": [
{
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
},
{
"op": "replace",
"path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0/",
"value":{"key":"location", 'operator':'In', 'values':"aws"},
}
]
}
now produces
rastructure.
Traceback (most recent call last):
File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/agent.py", line 484, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 299, in run
manifest = self.build_job()
File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 359, in build_job
job_manifest = self.customizations.apply(job_manifest)
File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 669, in apply
obj = operation.apply(obj)
File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 339, in apply
raise JsonPatchConflict(msg)
jsonpatch.JsonPatchConflict: can't replace a non-existent object ''
redsquare
03/23/2023, 7:45 AMRobert Denham
03/23/2023, 7:50 AMkubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a95db761-dd2a-4b7f-8d2f-1f2db71120b1', 'Cache-Control': 'no-cache, no-store, must-revalidate, no-cache, private', 'Content-Length': '386', 'Content-Type': 'application/json', 'Date': 'Thu, 23 Mar 2023 07:49:06 GMT', 'X-Api-Cattle-Auth': 'true', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'a7753913-783b-4421-9436-798ebbed5820', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fa89f132-55a3-4cf2-abdd-8b7e29df7c57'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Job in version \"v1\" cannot be handled as a Job: json: cannot unmarshal string into Go struct field NodeSelectorRequirement.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions.values of type []string","reason":"BadRequest","code":400}
Thanks @redsquare but I think I might need to do a bit of trial and error + googling to get this right.redsquare
03/23/2023, 7:51 AMRobert Denham
03/23/2023, 8:21 AMaws_infra_overrides = {
"customizations": [
{
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
},
{
"op": "replace",
"path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0",
"value":{"key":"location", 'operator':'In', 'values':["aws"]},
}
]
}
redsquare
03/23/2023, 8:21 AMRobert Denham
03/23/2023, 8:22 AMredsquare
03/23/2023, 8:22 AMRobert Denham
03/23/2023, 8:23 AMredsquare
03/23/2023, 8:24 AMRobert Denham
03/23/2023, 8:24 AM