Sean Turner
10/04/2022, 3:30 PMkubernetesJob
on EKS
orion, agent, and prefect cli are all 2.4.5
When I submit a simple task I get the following error in the agent logs:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
Seems that for some reason the service account called agent
isn't being observed? system:anonymous
is being used instead. The role
and roleBinding
both match what is in the helm chart so permissions should be fine.
$ kgp agent-5fbdcf4bbb-zrrg7 -o yaml | grep serviceAccount
serviceAccount: agent
serviceAccountName: agent
I was not having these problems before upgrading from 2.4.0 -> 2.4.5 😞. I created a new kubernetesjob block for 2.4.5 to silence an errorJamie Zieziula
10/04/2022, 5:14 PMSean Turner
10/04/2022, 5:44 PMservice_account_name
wasn't appearing in deployment.yaml
infrastructure.service_account_name
when running 2.4.5. No idea why 🤷Jamie Zieziula
10/04/2022, 5:57 PMdeployment.yaml
you mean the prefect deployment (not a k8s deplyoment) ?service account name
in your infra block will need to specify a valid k8s service account if its doing anything special within the cluster (i believe the default behavior if no SA is provided is to use the default
SA which doesn’t have any real permissions)k8s infra block service account
Sean Turner
10/04/2022, 6:13 PMdeployment
. I wasn't seeing the service_account_name
anywhere in there when I was trying 2.4.5 which was causing the issue. I think something got seriously messed up with my k8s
infra blocks when I upgraded which is where the value comes from. I did re-create those so I don't really know.