Farid
01/30/2023, 10:01 PMSubmission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd3e45036-d667-49d4-8e3b-3a73bec788e1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2681d750-c112-4385-86a3-ed13b9c2ebf2', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f3844bc0-2265-4354-9d81-5b753f0d513e', 'Date': 'Mon, 30 Jan 2023 21:34:52 GMT', 'Content-Length': '313'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:data-eng:default\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"data-eng\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
the deployments:
apiVersion: apps/v1
kind: Deployment
metadata:
name: prefect-agent
namespace: data-eng
labels:
app: prefect-agent
spec:
selector:
matchLabels:
app: prefect-agent
replicas: 1
template:
metadata:
labels:
app: prefect-agent
spec:
containers:
- name: agent
image: prefecthq/prefect:2.7.8-python3.9
command: ["prefect", "agent", "start", "-q", "k8s-us-west-2-prod"]
imagePullPolicy: "IfNotPresent"
env:
- name: PREFECT_API_URL
value: <https://api.prefect.cloud/api/accounts/XX/workspaces/YY>
- name: PREFECT_API_KEY
valueFrom:
secretKeyRef:
name: prefect-cloud-api-token
key: prefect-cloud-api-token-value
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
namespace: data-eng
name: prefect-agent
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/status"]
verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
name: prefect-agent-role-binding
namespace: data-eng
subjects:
- kind: ServiceAccount
name: prefect-agent
namespace: data-eng
roleRef:
kind: Role
name: prefect-agent
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
---
# The default manifest generated by `prefect kubernetes manifest agent` uses the default sa in the namespace.
# We will explicitly create a sa and attach the annotation required.
kind: ServiceAccount
metadata:
name: prefect-agent
namespace: data-eng
annotations:
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::XX:role/an-irsa-role
---
Needles to say that I have specified the same namespace and serviceAccount name in the Blocks but clearly the agent still tries to use the default serviceAccount to create jobs. any idea?Timo Vink
01/31/2023, 3:52 AMspec.template.spec.serviceAccountName
property on the Deployment
for the agent?
In my setup I have 2 service accounts. One for the agent (with RBAC permissions for jobs.batch
etc.), and one for the jobs (in my case, to grant it access to an AWS IAM role).
The agent is the one that needs permission for jobs.batch
, but I believe the Service Account Name property on the Block in your screenshot is what would be used by the Jobs.Christopher Boyd
01/31/2023, 3:41 PMdefault
service account:
\"system:serviceaccount:data-eng:default\"
data-eng
is your namespace, prefect-agent
is your service accountdefault
rightfully doesn’t have permissionskubectl get sa -n data-eng
show, as well as kubectl get sa -A
Farid
01/31/2023, 9:16 PMChristopher Boyd
01/31/2023, 9:44 PM