I am getting this error

prefect.exceptions.FailedRun: Submission failed. kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'af4884b6-f13d-4813-9b63-79f859e962d0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '07cd00e8-f26e-4ed1-b441-b0eb2115b3b7', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f9d0e8bc-cf15-4ed5-8169-c4cddb53fff8', 'Date': 'Mon, 29 May 2023 15:05:49 GMT', 'Content-Length': '323'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:sandbox-bidw:prefect-agent\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"dev-bidw\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

Anybody knows how to fix this? I do not see batch options in roles/permissions section.

your agent is in a different namespace than where you are trying to run your flows. Put them in the same namespace, or add additional permissions to your agents service account

I verified both are in same namespaces. So, that’s not an issue looks like. Which permission I need to add?

your error message tells me they are not in the same namespace. your agent is in the sandbox-bidw while your flow is in dev-bidw

Hmm you are right do I need to set namespace somewhere on prefect side too? I meant in UI

can you verify if you have a second agent running in the sandbox namespace?

strangely enough when I do

kubectl get namespace

I do not even see that namespace.

do you have multiple agents polling the work pool that your flows are being submitted to?

yeah two of them

are you still receiving the original 403 error that specifies the sandbox-bidw namespace?

Its failing right away not sure what went wrong this time. I am getting no logs either.

i’m wondering if you have a second cluster somewhere that could be running the agent in the namespace

sandbox-bidw

?

yeah I am trying to find that out. I will let you know. Thank you very much for your help.

did you get that when retrying the flow or on a fresh run?

these are all simple dummy task I am not sure why retry would be needed but let me check. I am doing this

deployment = run_deployment("transform-data/poc-prefect-multiple-job-test-main-transform")

logger().info("Deployment status: transform-data/poc-
prefect-multiple-job-test-main-transform")

logger().info(deployment.state.result())

return {"result": deployment.state.result()}

np!