Sean Turner
10/04/2022, 3:30 PMkubernetesJob on EKS
orion, agent, and prefect cli are all 2.4.5
When I submit a simple task I get the following error in the agent logs:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
Seems that for some reason the service account called agent isn't being observed? system:anonymous is being used instead. The role and roleBinding both match what is in the helm chart so permissions should be fine.
$ kgp agent-5fbdcf4bbb-zrrg7 -o yaml | grep serviceAccount
serviceAccount: agent
serviceAccountName: agent
I was not having these problems before upgrading from 2.4.0 -> 2.4.5 😞. I created a new kubernetesjob block for 2.4.5 to silence an errorJamie Zieziula
10/04/2022, 5:14 PMSean Turner
10/04/2022, 5:44 PMservice_account_name wasn't appearing in deployment.yaml infrastructure.service_account_name when running 2.4.5. No idea why 🤷Jamie Zieziula
10/04/2022, 5:57 PMdeployment.yaml you mean the prefect deployment (not a k8s deplyoment) ?Jamie Zieziula
10/04/2022, 6:00 PMservice account name in your infra block will need to specify a valid k8s service account if its doing anything special within the cluster (i believe the default behavior if no SA is provided is to use the default SA which doesn’t have any real permissions)Jamie Zieziula
10/04/2022, 6:01 PMk8s infra block service accountSean Turner
10/04/2022, 6:13 PMdeployment. I wasn't seeing the service_account_name anywhere in there when I was trying 2.4.5 which was causing the issue. I think something got seriously messed up with my k8s infra blocks when I upgraded which is where the value comes from. I did re-create those so I don't really know.