hey, I was wondering if someone could help me with the overrides thing. I've currently got a node-selector section in my kubernetes job which looks like:

ttlSecondsAfterFinished: 100
  template:
    spec:
      dnsPolicy: Default
      completions: 1
      containers:  # the first container is required
      - env: []  # env is required, even if empty
        name: prefect-job
      parallelism: 1
      restartPolicy: Never
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: project
                operator: In
                values:
                - fcv3

I was just wondering how I go about modifying that section, to, for example, use a different key , different values. I want to use this in the deployment, so using infra_overrides The example override looks like:

infra_overrides={ 
    "customizations": [
            {
                "op": "add",
                "path": "/spec/template/spec/containers/0/resources",
                "value": {
                    "requests": {
                        "cpu": "2000m",
                        "memory": "4gi"
                    },
                    "limits": {
                        "cpu": "4000m",
                        "memory": "8Gi",
                        "<http://nvidia.com/gpu|nvidia.com/gpu>": "1"
                }
            },
        }
    ]
}

what would the equivalent be in my case? I had a stab at it, like (actually trying to add two over-rides, an add and a replace)

aws_infra_overrides = {
    "customizations": [
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/resources",
            "value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
        },
        {
            "op": "replace",
            "path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0/",
            "value'":{"key":"location", 'operator':'In', 'values':"aws"},
        }
    ]
}

It looks ok to me at a glance, how are you adding it to the deployment?

like this:

deployment = Deployment.build_from_flow(
    flow=fc_flow,
    name="fcv3 thurs",
    version=2,
    description="""Create the input images""",
    tags=["test run"],
    work_pool_name="rss-pool",
    infrastructure=k8block,
    storage=s3_block,
    parameters={"satid": "fill_me_in"},
    infra_overrides=aws_infra_overrides,
)

deployment.apply()

on the bottom one try /- at the end of the path

and the output is like:

16:21:15.031 | ERROR   | prefect.agent - Failed to submit flow run '722902ed-b25d-4b50-bb94-cb6c96d199c8' to infrastructure.
Traceback (most recent call last):
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 319, in apply
    value = self.operation["value"]
KeyError: 'value'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/agent.py", line 484, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 299, in run
    manifest = self.build_job()
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 359, in build_job
    job_manifest = self.customizations.apply(job_manifest)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 669, in apply
    obj = operation.apply(obj)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 321, in apply
    raise InvalidJsonPatch(
jsonpatch.InvalidJsonPatch: The operation does not contain a 'value' member

secretPatch = { "op": "add", "path": "/spec/template/spec/containers/0/envFrom/-", "value": { "secretRef": { "name": secret.lower() } }}

I get the same thing:

jsonpatch.InvalidJsonPatch: The operation does not contain a 'value' member

ah yeah

I'm getting closer;

aws_infra_overrides = {
    "customizations": [
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/resources",
            "value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
        },
        {
            "op": "replace",
            "path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0/",
            "value":{"key":"location", 'operator':'In', 'values':"aws"},
        }
    ]
}

now produces

rastructure.
Traceback (most recent call last):
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/agent.py", line 484, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 299, in run
    manifest = self.build_job()
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 359, in build_job
    job_manifest = self.customizations.apply(job_manifest)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 669, in apply
    obj = operation.apply(obj)
  File "/home/denhamr/.venv310/lib/python3.10/site-packages/jsonpatch.py", line 339, in apply
    raise JsonPatchConflict(msg)
jsonpatch.JsonPatchConflict: can't replace a non-existent object ''

remove the end slash

kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a95db761-dd2a-4b7f-8d2f-1f2db71120b1', 'Cache-Control': 'no-cache, no-store, must-revalidate, no-cache, private', 'Content-Length': '386', 'Content-Type': 'application/json', 'Date': 'Thu, 23 Mar 2023 07:49:06 GMT', 'X-Api-Cattle-Auth': 'true', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'a7753913-783b-4421-9436-798ebbed5820', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fa89f132-55a3-4cf2-abdd-8b7e29df7c57'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Job in version \"v1\" cannot be handled as a Job: json: cannot unmarshal string into Go struct field NodeSelectorRequirement.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions.values of type []string","reason":"BadRequest","code":400}

Thanks @redsquare but I think I might need to do a bit of trial and error + googling to get this right.

maybe your values needs to be an array

Brilliant! Thanks @redsquare that was the answer:

aws_infra_overrides = {
    "customizations": [
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/resources",
            "value": {"requests": {"memory": "2Gi", "cpu": "1000m"}, "limits": {"memory": "4Gi", "cpu": "1500m"}},
        },
        {
            "op": "replace",
            "path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/0",
            "value":{"key":"location", 'operator':'In', 'values':["aws"]},
        }
    ]
}

Yay 👍

I find that overrides thing tricky to interpret, so hopefully examples like these, which are useful to me, might be useful to others.

yeah a few struggle in here especially with the correct constructs

some of us are learning as we go, I'm more at the scientist end of the workflow, rather than the dev ops end, so kubernetes is still a struggle for me.

I find it helps to add a k8s block first and load it then you can see the shape

yep, agreed. I got pretty close, just needed a push 🙂