Hello, I am trying to calculate the costs of different project running jobs in my prefect-on-kubernetes deployment and I am thinking on running each project's jobs in a different namespace and then calculate costs using kubecost. I have not found much though on how easily can prefect manage jobs and tasks that run in a different namespace that itself. Has anybody tried something similar? Do you think it is feasible? Thank you!

The helm chart for the kubernetes worker creates a Role, RoleBinding, and ServiceAccount in the namespace you deploy your worker in which enable the worker to start, watch, and generally manage jobs in that namespace. Under this default setup, if you tried to start a job in a different namespace (either by setting the namespace on your work pool or with a

job_variable

override in your deployment), the worker will get a 403 and fail to start the job. My worker is in the

prefect

namespace so I made a namespace called

prefect2

and edited my work pool to start jobs there to demonstrate:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "jobs.batch is forbidden: User \"system:serviceaccount:prefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect2\"",
  "reason": "Forbidden",
  "details": {
    "group": "batch",
    "kind": "jobs"
  },
  "code": 403
}

The easiest way around this would probably be to create a ClusterRole with the same rules as the existing Role, with the addition of the

events

resource in the first rule:

rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status", "events"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]

and then a ClusterRoleBinding that references this ClusterRole and targets the

prefect-worker

ServiceAccount that already exists. Then your worker can create and monitor jobs in any namespace in the cluster.

Hmm very useful information. It will definitely help me take some decisions. Thank you very much!