Hi I'm getting submission errors in Kube when trying to run my flows with a custom serviceAccount:

Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd3e45036-d667-49d4-8e3b-3a73bec788e1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2681d750-c112-4385-86a3-ed13b9c2ebf2', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f3844bc0-2265-4354-9d81-5b753f0d513e', 'Date': 'Mon, 30 Jan 2023 21:34:52 GMT', 'Content-Length': '313'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:data-eng:default\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"data-eng\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

the deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-agent
  namespace: data-eng
  labels:
    app: prefect-agent
spec:
  selector:
    matchLabels:
      app: prefect-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - name: agent
        image: prefecthq/prefect:2.7.8-python3.9
        command: ["prefect", "agent", "start", "-q", "k8s-us-west-2-prod"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: <https://api.prefect.cloud/api/accounts/XX/workspaces/YY>
          - name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: prefect-cloud-api-token
                key: prefect-cloud-api-token-value
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  namespace: data-eng
  name: prefect-agent
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-agent-role-binding
  namespace: data-eng
subjects:
- kind: ServiceAccount
  name: prefect-agent 
  namespace: data-eng
roleRef:
  kind: Role
  name: prefect-agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>

---
# The default manifest generated by `prefect kubernetes manifest agent` uses the default sa in the namespace.
# We will explicitly create a sa and attach the annotation required.

kind: ServiceAccount
metadata:
  name: prefect-agent 
  namespace: data-eng
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::XX:role/an-irsa-role
---

Needles to say that I have specified the same namespace and serviceAccount name in the Blocks but clearly the agent still tries to use the default serviceAccount to create jobs. any idea?

Not affiliated with prefect, and only a k8s novice. But just in case it helps: Perhaps are you missing the

spec.template.spec.serviceAccountName

property on the

Deployment

for the agent? In my setup I have 2 service accounts. One for the agent (with RBAC permissions for

jobs.batch

etc.), and one for the jobs (in my case, to grant it access to an AWS IAM role). The agent is the one that needs permission for

jobs.batch

, but I believe the Service Account Name property on the Block in your screenshot is what would be used by the Jobs.

it looks like the job is trying to be submitted with the

default

service account:

\"system:serviceaccount:data-eng:default\"

What @Timo Vink suggested did the trick, flows run flawlessly now, thank you both 🙂

My apologies, re-reading my own response and your question, I see what you meant, and @Timo Vink as well. Took me looking at our helm chart to understand what you meant , sorry! https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-agent/templates/deployment.yaml#L38