https://prefect.io logo
#prefect-community
Title
# prefect-community
s

Sean Turner

10/04/2022, 3:30 PM
Hi, I'm running prefect (orion and agent) on EKS with
kubernetesJob
on EKS orion, agent, and prefect cli are all
2.4.5
When I submit a simple task I get the following error in the agent logs:
Copy code
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
Seems that for some reason the service account called
agent
isn't being observed?
system:anonymous
is being used instead. The
role
and
roleBinding
both match what is in the helm chart so permissions should be fine.
Copy code
$ kgp agent-5fbdcf4bbb-zrrg7 -o yaml | grep serviceAccount
  serviceAccount: agent
  serviceAccountName: agent
I was not having these problems before upgrading from 2.4.0 -> 2.4.5 😞. I created a new kubernetesjob block for 2.4.5 to silence an error
1
j

Jamie Zieziula

10/04/2022, 5:14 PM
can you share what your role & rolebindings look like?
s

Sean Turner

10/04/2022, 5:44 PM
Thanks for the response. I nuked everything (including the database) and got it working by rolling back to 2.4.0. I think the main issue is that my
service_account_name
wasn't appearing in
deployment.yaml
infrastructure.service_account_name
when running 2.4.5. No idea why 🤷
j

Jamie Zieziula

10/04/2022, 5:57 PM
hm interesting. by
deployment.yaml
you mean the prefect deployment (not a k8s deplyoment) ?
👍 1
The
service account name
in your infra block will need to specify a valid k8s service account if its doing anything special within the cluster (i believe the default behavior if no SA is provided is to use the
default
SA which doesn’t have any real permissions)
the error you provided indicates that the agent wasnt able to schedule jobs within the cluster which is odd, and shouldnt be related to the
k8s infra block service account
s

Sean Turner

10/04/2022, 6:13 PM
Yeah, in the prefect
deployment
. I wasn't seeing the
service_account_name
anywhere in there when I was trying 2.4.5 which was causing the issue. I think something got seriously messed up with my
k8s
infra blocks when I upgraded which is where the value comes from. I did re-create those so I don't really know.
👍 1
3 Views