Kyle Hoffman
07/12/2023, 1:08 PMkubernetes-job
block is not adhering to the namespace
or service account name
input fields. Also when deploying both the agent and server into the "prefect" namespace, I get an error when running a flow:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:prefect:prefect-agent\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
This is even after setting the service account name to some dummy value as well as setting the ns to be "prefect"
The next step is "well if it needs access to the default name space, let's just deploy it into the default namespace", after doing that, I get the exact same issue, except now the error points to the job wanting to be created within the "kube-system" namespace?
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:prefect-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
So yeah, i'm pretty lost for what to do, I've even double checked the prefect-agent service account, and it 100% has access to the resources it claims it doesn't.
Versions running:
- name: prefect-agent
version: 2023.07.07
repository: <https://prefecthq.github.io/prefect-helm>
- name: prefect-server
version: 2023.07.07
repository: <https://prefecthq.github.io/prefect-helm>
Christopher Boyd
07/12/2023, 1:23 PMdefault
and a prefect-agent
kubectl describe role prefect-worker
or prefect-agent
in your case to see what permissions the attached role has on the service account
$ k describe role prefect-worker
Name: prefect-worker
Labels: <http://app.kubernetes.io/component=worker|app.kubernetes.io/component=worker>
<http://app.kubernetes.io/instance=aks-worker|app.kubernetes.io/instance=aks-worker>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=prefect-worker|app.kubernetes.io/name=prefect-worker>
<http://helm.sh/chart=prefect-worker-2023.04.13|helm.sh/chart=prefect-worker-2023.04.13>
prefect-version=2.10.4-python3.11-kubernetes
Annotations: <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: aks-worker
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: default
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
jobs.batch [] [] [get list watch create update patch delete]
pods/log [] [] [get watch list]
pods/status [] [] [get watch list]
pods [] [] [get watch list]
Kyle Hoffman
07/12/2023, 1:27 PMName: prefect-agent
Labels: <http://app.kubernetes.io/component=agent|app.kubernetes.io/component=agent>
<http://app.kubernetes.io/instance=prefect-agent-primary|app.kubernetes.io/instance=prefect-agent-primary>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=prefect-agent|app.kubernetes.io/name=prefect-agent>
<http://argocd.argoproj.io/instance=prefect-agent-primary|argocd.argoproj.io/instance=prefect-agent-primary>
<http://helm.sh/chart=prefect-agent-2023.07.07|helm.sh/chart=prefect-agent-2023.07.07>
prefect-version=2.10.20-python3.10
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
jobs.batch [] [] [get list watch create update patch delete]
pods/log [] [] [get watch list]
pods/status [] [] [get watch list]
pods [] [] [get watch list]
cannot create resource \"jobs\"
But when deploying into the "default" ns, it fails on cannot get resource \"namespaces\"
Christopher Boyd
07/12/2023, 2:31 PMKyle Hoffman
07/12/2023, 2:50 PMBernard Greyling
07/13/2023, 7:51 AMenvFrom
reference.
@Kyle Hoffman Could it be related to that?Christopher Boyd
07/13/2023, 12:15 PMnamespace="prefect",
service_account_name="prefect-agent",
if you aren’t specifying those in your job, it’s using defaults for both