https://prefect.io logo
Title
r

Robert Denham

03/23/2023, 6:28 AM
hey, I was wondering if someone could help me with the overrides thing. I've currently got a node-selector section in my kubernetes job which looks like:
ttlSecondsAfterFinished: 100
  template:
    spec:
      dnsPolicy: Default
      completions: 1
      containers:  # the first container is required
      - env: []  # env is required, even if empty
        name: prefect-job
      parallelism: 1
      restartPolicy: Never
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: project
                operator: In
                values:
                - fcv3
I was just wondering how I go about modifying that section, to, for example, use a different key , different values. I want to use this in the deployment, so using infra_overrides The example override looks like:
infra_overrides={ 
    "customizations": [
            {
                "op": "add",
                "path": "/spec/template/spec/containers/0/resources",
                "value": {
                    "requests": {
                        "cpu": "2000m",
                        "memory": "4gi"
                    },
                    "limits": {
                        "cpu": "4000m",
                        "memory": "8Gi",
                        "<http://nvidia.com/gpu|nvidia.com/gpu>": "1"
                }
            },
        }
    ]
}
what would the equivalent be in my case? I had a stab at it, like (actually trying to add two over-rides, an add and a replace)
aws_infra_overrides = {
    "customizations": [
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/resources",
            "value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
        },
        {
            "op": "replace",
            "path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0/",
            "value'":{"key":"location", 'operator':'In', 'values':"aws"},
        }
    ]
}
r

redsquare

03/23/2023, 7:37 AM
It looks ok to me at a glance, how are you adding it to the deployment?
r

Robert Denham

03/23/2023, 7:38 AM
like this:
deployment = Deployment.build_from_flow(
    flow=fc_flow,
    name="fcv3 thurs",
    version=2,
    description="""Create the input images""",
    tags=["test run"],
    work_pool_name="rss-pool",
    infrastructure=k8block,
    storage=s3_block,
    parameters={"satid": "fill_me_in"},
    infra_overrides=aws_infra_overrides,
)

deployment.apply()
r

redsquare

03/23/2023, 7:39 AM
on the bottom one try /- at the end of the path
r

Robert Denham

03/23/2023, 7:39 AM
and the output is like:
16:21:15.031 | ERROR   | prefect.agent - Failed to submit flow run '722902ed-b25d-4b50-bb94-cb6c96d199c8' to infrastructure.
Traceback (most recent call last):
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 319, in apply
    value = self.operation["value"]
KeyError: 'value'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/agent.py", line 484, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 299, in run
    manifest = self.build_job()
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 359, in build_job
    job_manifest = self.customizations.apply(job_manifest)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 669, in apply
    obj = operation.apply(obj)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 321, in apply
    raise InvalidJsonPatch(
jsonpatch.InvalidJsonPatch: The operation does not contain a 'value' member
r

redsquare

03/23/2023, 7:40 AM
secretPatch = { "op": "add", "path": "/spec/template/spec/containers/0/envFrom/-", "value": { "secretRef": { "name": secret.lower() } }}
thats one from mine
r

Robert Denham

03/23/2023, 7:42 AM
I get the same thing:
jsonpatch.InvalidJsonPatch: The operation does not contain a 'value' member
wait, I think there is a stray quote in there...
r

redsquare

03/23/2023, 7:44 AM
ah yeah
single quote
r

Robert Denham

03/23/2023, 7:45 AM
I'm getting closer;
aws_infra_overrides = {
    "customizations": [
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/resources",
            "value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
        },
        {
            "op": "replace",
            "path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0/",
            "value":{"key":"location", 'operator':'In', 'values':"aws"},
        }
    ]
}
now produces
rastructure.
Traceback (most recent call last):
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/agent.py", line 484, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 299, in run
    manifest = self.build_job()
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 359, in build_job
    job_manifest = self.customizations.apply(job_manifest)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 669, in apply
    obj = operation.apply(obj)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 339, in apply
    raise JsonPatchConflict(msg)
jsonpatch.JsonPatchConflict: can't replace a non-existent object ''
r

redsquare

03/23/2023, 7:45 AM
remove the end slash
r

Robert Denham

03/23/2023, 7:50 AM
kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a95db761-dd2a-4b7f-8d2f-1f2db71120b1', 'Cache-Control': 'no-cache, no-store, must-revalidate, no-cache, private', 'Content-Length': '386', 'Content-Type': 'application/json', 'Date': 'Thu, 23 Mar 2023 07:49:06 GMT', 'X-Api-Cattle-Auth': 'true', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'a7753913-783b-4421-9436-798ebbed5820', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fa89f132-55a3-4cf2-abdd-8b7e29df7c57'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Job in version \"v1\" cannot be handled as a Job: json: cannot unmarshal string into Go struct field NodeSelectorRequirement.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions.values of type []string","reason":"BadRequest","code":400}
Thanks @redsquare but I think I might need to do a bit of trial and error + googling to get this right.
I think better would be to have no node affinity stuff in my kubernetes job, and add a complete override, rather than trying to replace one.
r

redsquare

03/23/2023, 7:51 AM
maybe your values needs to be an array
the fcv3
i think thats it given the new error
patchstr='[{"op":"add","path":"/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0","value":{"key":"icp4data","operator":"In","values":["database-bigsql"]}},
example ^^
r

Robert Denham

03/23/2023, 8:21 AM
Brilliant! Thanks @redsquare that was the answer:
aws_infra_overrides = {
    "customizations": [
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/resources",
            "value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
        },
        {
            "op": "replace",
            "path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0",
            "value":{"key":"location", 'operator':'In', 'values':["aws"]},
        }
    ]
}
r

redsquare

03/23/2023, 8:21 AM
Yay 👍
r

Robert Denham

03/23/2023, 8:22 AM
I find that overrides thing tricky to interpret, so hopefully examples like these, which are useful to me, might be useful to others.
r

redsquare

03/23/2023, 8:22 AM
yeah a few struggle in here especially with the correct constructs
r

Robert Denham

03/23/2023, 8:23 AM
some of us are learning as we go, I'm more at the scientist end of the workflow, rather than the dev ops end, so kubernetes is still a struggle for me.
r

redsquare

03/23/2023, 8:24 AM
I find it helps to add a k8s block first and load it then you can see the shape
r

Robert Denham

03/23/2023, 8:24 AM
yep, agreed. I got pretty close, just needed a push 🙂
🙌 1