https://prefect.io logo
Powered by
Title
d

Deceivious

04/20/2023, 1:42 PM
Hi folks, I have a kubernetes infrastucture setup with namespace set to 
pefect2-worker-dev
[image below]. When I try to execute a flow run in that infrastructure, it gives out the following error. 
Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e6<<SOME STUFF HERE>>37:14 GMT', 'Content-Length': '330'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:prefect2-worker-dev:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
I am unsure as to what actually determines which namespace the job is created in. [prefect v2.10.4 on python 3.10]
👀 1
also in #prefect-kubernetes https://prefect-community.slack.com/archives/C048SVCEFF0/p1681980426131549
image.png
r

redsquare

04/20/2023, 1:50 PM
me again 🙂 have you seen https://prefect-community.slack.com/archives/C048ZHT5U3U/p1681785564962719?thread_ts=1681754473.501149&amp;cid=C048ZHT5U3U
🙌 1
d

Deceivious

04/20/2023, 2:12 PM
How do i get the cluster UID? 😅
r

redsquare

04/20/2023, 2:16 PM
think you have to set it in helm
d

Deceivious

04/20/2023, 2:44 PM
Unsure but the link you have posted seems like different issues. In my usecase, the worker is trying to create a job in the default NS and not the NS I have explicitly provided, which results in an exception because the SA has no access to default NS. My question is why is the SA trying to run the job in 
default
NS and not the one I explicitly provided and how do I change it?
r

redsquare

04/20/2023, 2:47 PM
Yeah I am looking back when I had similar error messages (4/5 months) - at the time my fix was to create a ClusterRoleBinding for the service account
before they introduced what the link said
d

Deceivious

04/20/2023, 2:49 PM
M checking prefect-helm for worker and it does have cluster role binding set by default . https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-worker/templates/rolebinding.yaml
r

redsquare

04/20/2023, 2:51 PM
I am sure @jawnsy will assist when he is around
👀 1
d

Deceivious

04/20/2023, 3:46 PM
OK Ive run some tests. Deployed 2 set of workers in 2 NS. NS#1: helm_workers [Uses helm deployed workers] NS#2: test_workers [Uses prefect kubernetes manifest command worker] When I only have helm workers it always tries to run jobs in 
default
NS no matter what I put in the KubernetesJob during deployment. The deployment being same, when I turn down the helm workers and switch to manifest worker. The SA automatically picks the NS specified in KubernetesJob during deployment. So either theres something wrong on the helm chart OR I am not deploying helm chart correctly. I use 
helm install --namespace=prefect2-worker-dev  --values=./worker.dev.yaml prefect-worker prefect/prefect-worker
to deploy.
j

jawnsy

04/20/2023, 5:05 PM
That’s very strange, the jobs that get created are controlled by the Kubernetes Worker (via KubernetesJob as you say), so as long as they’re set in the worker pool when creating it (and you’re not overriding it in the job you create), see screenshot, they should go to the correct namespace. What version of prefect & prefect-kubernetes do you have?
d

Deceivious

04/20/2023, 5:34 PM
Prefect is 2.10.4
I don't think i use prefct Kubernetes
Yes i just checked my pyproject config doesn't have prefect Kubernetes specified
from prefect.infrastructure.kubernetes import KubernetesJob
I am using this as infrastructure.
a

alex

04/21/2023, 3:07 PM
Hey @Deceivious, if you’re using the 
KubernetesJob
infrastructure block, then you’ll need to use an agent instead of the Kubernetes worker. You can use the agent prefect helm chart to deploy an agent instead of the worker helm chart.
d

Deceivious

04/21/2023, 3:21 PM
This is a bit confusing to me. New to kuber and helm. But isnt Prefect worker an native concept of Prefect and has nothing to do with Kubernetes ? If I can deploy prefect agent with helm, why is it that deploying worker with helm causes issue?
@alex
a

alex

04/21/2023, 3:35 PM
Workers and agents are Prefect specific concepts. Agents work with infrastructure blocks like the 
KubernetesJob
block and poll for flow runs from 
prefect-agent
typed work pools. Workers are a newer concept (they’re still in beta), but they are like an agent and an infrastructure block combined and poll for flow runs from typed work pools. All this means that you either need to use a agent + infra block or a worker.
d

Deceivious

04/21/2023, 3:46 PM
Thanks @alex 😒aluting_face:
s

sjammula

04/21/2023, 7:24 PM
@alex I'm facing similar issue in my machine where I'm not using any helm chart.can you suggest me a way forward to overcome this error. Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '85efeb11-33b6-4a58-91ad-d0a60bffce1a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '513745c7-aa7c-4245-8349-ec7b488ba2ba', 'X-Kubernetes-Pf-Prioritylevel-Uid': '17e7873d-25c0-45c1-955e-4c4692a6bb21', 'Date': 'Fri, 21 Apr 2023 18:25:18 GMT', 'Content-Length': '311'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system😒erviceaccount:prefect:prefect\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403
d

Deceivious

04/21/2023, 8:41 PM
@sjammula If you arent using the helm file but are using the output of 
prefect kubernetes manifest agent
, you need to specify which name space to run it in [theres an cli parameter for it]. And when you deploy the flow, you need to ensure that the KubernetesJob has the same namespace.
Hi @alex, sorry to ping again. I am still a bit confused. I kinda get the difference between agent and worker. BUT, why is it that deploying the worker with 
prefect kubernetes manifest
command works with the correct queue and the 
helm
deployment fails. If both the workers are equivalent despite the deployment method, either both should work or both should fail. Unsure about the change in behavior based on deployment method.
a

alex

04/21/2023, 8:59 PM
The manifest that get’s generated from 
prefect kubernetes manifest
is a manifest for agent deployment. We haven’t added a worker manifest to that CLI command yet. You might be deploying an agent with one method and a worker with the other method, depending on which helm chart you’re using.
🙌 1
d

Deceivious

04/24/2023, 7:37 AM
THanks
r

redsquare

04/24/2023, 7:44 AM
Just so that I am clear what are the advantages of a worker in a k8s env?
👀 1
j

Joshua Greenhalgh

06/06/2023, 11:29 AM
@alex would really appreciate some more detail around the differences between an agent and a worker?
Are there some docs?
j

jawnsy

06/06/2023, 1:51 PM
@Joshua Greenhalgh Agents were our first generation system for collecting/running work. Workers are an updated version. If you’re just starting out today, use a worker. There are docs here: https://docs.prefect.io/2.10.12/concepts/work-pools/
:upvote: 1
a

alex

06/06/2023, 1:54 PM
To add on to what @jawnsy said, workers are scoped to a specific type of infrastructure and offer more customization compared to agents. Workers also have more observability features compared to agents when used with Prefect Cloud.
👍 1
j

Joshua Greenhalgh

06/06/2023, 2:08 PM
Ok so I tried a worker and I got exactly the same issue as the OP - tried to create the Job in default ns but service account only has permissions for "prefect:?
j

jawnsy

06/06/2023, 2:10 PM
When you create the worker, it will default to the ‘default’ namespace, you need to change that to ‘prefect’ or the namespace you deployed the worker into
Apologies that I haven’t been following this whole thread but I’ve created an issue for the default namespace problem here: https://github.com/PrefectHQ/prefect/issues/9845 This is something we can improve :)
r

redsquare

06/06/2023, 2:27 PM
@alex will agents be depreciated/removed any time soon or are we safe for a while
a

alex

06/06/2023, 2:32 PM
We have a 6 month deprecation period before removing functionality, so there will be plenty of notice before agents are removed.
👍 1
j

Joshua Greenhalgh

06/06/2023, 2:40 PM
its not that the agent is in the wrong namespace its that the job thats created is in the wrong namespace?
I used raw manifests the following; 
---
# Source: prefect-worker/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prefect-worker
  namespace: "prefect"
---
# Source: prefect-worker/templates/role.yaml
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-worker
  namespace: "prefect"
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
# Source: prefect-worker/templates/rolebinding.yaml
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-worker
  namespace: "prefect"
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-worker
subjects:
  - kind: ServiceAccount
    name: prefect-worker
    namespace: "prefect"
---
# Source: prefect-worker/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-worker
  namespace: "prefect"
  labels:
    app: prefect-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-worker
  template:
    metadata:
      labels:
        app: prefect-worker
    spec:
      serviceAccountName: prefect-worker
      securityContext:
        fsGroup: 1001
        runAsNonRoot: true
        runAsUser: 1001
      containers:
        - name: prefect-worker
          image: "prefecthq/prefect:2.10.12-python3.9-kubernetes"
          imagePullPolicy: IfNotPresent
          command:
            - prefect
            - worker
            - start
            - --type
            - kubernetes
            - --pool
            - default-agent-pool
            - --work-queue
            - default
          workingDir: /home/prefect
          env:
            - name: HOME
              value: /home/prefect
            - name: PREFECT_AGENT_PREFETCH_SECONDS
              value: "10"
            - name: PREFECT_AGENT_QUERY_INTERVAL
              value: "5"
            - name: PREFECT_API_ENABLE_HTTP2
              value: "true"
            - name: PREFECT_API_URL
              value: "<http://host.docker.internal:4200/api>"
            - name: PREFECT_KUBERNETES_CLUSTER_UID
              value: ""
            - name: PREFECT_DEBUG_MODE
              value: "false"
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi
            requests:
              cpu: 100m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1001
          volumeMounts:
            - mountPath: /home/prefect
              name: scratch
              subPathExpr: home
            - mountPath: /tmp
              name: scratch
              subPathExpr: tmp
      volumes:
        - name: scratch
          emptyDir: {}
working on a local minikube at the momentr
2 Views
#prefect-communityJoin Slack
Powered by