https://prefect.io logo
Powered by
Title
a

Abhinav Chordia

04/17/2023, 6:01 PM
Hello, I’m having issue submitting a flow onto our k8s cluster. I’m getting a 403 on the submission. More details in 🧵
I’m using the service account created via the helm chart. 
Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd8c61061-378f-4686-8163-07b21c220c17', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '903a45a7-c309-4b50-9720-817e715a41a4', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a496d18f-252e-4832-8af8-99bab3405eea', 'Date': 'Mon, 17 Apr 2023 18:00:51 GMT', 'Content-Length': '346'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
And deploying via the python sdk: 
deployment = Deployment.build_from_flow(
    flow=call_api,
    name="example",
    version=1,
    infrastructure=kubernetes_job_block,
    work_queue_name="default",
    path="/user/abhinav",
    storage=az_block
)

deployment.apply()
I created the k8s block via this: 
k8s_job = KubernetesJob(
        namespace="prefect",
        image="<http://docker.gh.st/hypo:latest|docker.gh.st/hypo:latest>",
        image_pull_policy=KubernetesImagePullPolicy.ALWAYS,
        finished_job_ttl=300,
        job_watch_timeout_seconds=600,
        pod_watch_timeout_seconds=600,
        service_account_name="prefect-server",
        customizations=customizations,
    )
k8s_job.save("devk8s", overwrite=True)
I’ve deployed prefect worker, agent, and server in the prefect namespace. The service accounts are also in the prefect namespace.
It’s possible the helm charts are missing the ClusterRole definition
r

redsquare

04/17/2023, 6:34 PM
yup, missing clusterrole sounds on the money
j

jawnsy

04/17/2023, 6:43 PM
We create a Role/RoleBinding because we deploy into the same namespace by default, but if you want your agent/worker to be in a different namespace from where your jobs run, then you will need a ClusterRole/ClusterRoleBinding, yes
a

Abhinav Chordia

04/17/2023, 6:48 PM
I deployed it in the 
prefect
namespace and wanted to run it there as well but it’s still requiring being able to list the kube-system namespace
The output of 
prefect kubernetes manifest agent
also has: 
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: prefect-agent
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
  name: prefect-agent-cluster-role-binding
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default
roleRef:
  kind: ClusterRole
  name: prefect-agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Which is not part of the helm charts
Yup adding the following to the prefect-agent charts fixed it: 
apiVersion: {{ include "common.capabilities.rbac.apiVersion" . }}
kind: ClusterRole
metadata:
  name: {{ include "common.names.fullname" . }}
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
apiVersion: {{ include "common.capabilities.rbac.apiVersion" . }}
kind: ClusterRoleBinding
metadata:
  name: {{ include "common.names.fullname" . }}
subjects:
  - kind: ServiceAccount
    name: {{ template "agent.serviceAccountName" . }}
    namespace: {{ include "common.names.namespace" . | quote }}
roleRef:
  kind: ClusterRole
  name: {{ template "common.names.fullname" . }}
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Created issue: https://github.com/PrefectHQ/prefect-helm/issues/153
j

jawnsy

04/17/2023, 7:30 PM
It looks in the kube-system namespace just to get the Cluster UID, you can override that with a value as well https://github.com/PrefectHQ/prefect-helm/blob/a5c7eab12e5d4b2b42fe5c5a997ddb8e268b962c/charts/prefect-agent/values.yaml#L15-L16
The lookup should be happening at install time in Helm, so the service account should not require a ClusterRoleBinding at runtime https://github.com/PrefectHQ/prefect-helm/blob/e615bddd706000964fc2c85d5d16b991c7aca6df/charts/prefect-worker/templates/_helpers.tpl#L30-L33
a

Abhinav Chordia

04/17/2023, 7:31 PM
That’s not what i’m observing.
j

Jamie Zieziula

04/18/2023, 2:39 AM
I’m not sure how you’re installing the chart, but as @jawnsy mentioned, at install time Helm should be doing a lookup to get the 
kube-system
namespace ID. If for some reason that is not working with your install path, you can set a UID here. This will eliminate the need for a cluster role & binding
I’m going to close the issue you opened, but please reopen or respond here if setting the UID does not work.
a

Abhinav Chordia

04/18/2023, 4:11 AM
We are deploying it via argo.
j

jawnsy

04/18/2023, 2:16 PM
You should be able to specify a value (really any value unique to your cluster is OK) as a Helm value along with your other settings
a

Abhinav Chordia

04/18/2023, 2:19 PM
Will try this. Though it would be useful if this was clear somewhere in the documentation.
j

jawnsy

04/18/2023, 3:44 PM
Sure, I’ll add a note to the README
:thank_you: 1
#prefect-getting-startedJoin Slack
Powered by