Hello Community. We have a strange issue - the kub...
# prefect-community
Hello Community. We have a strange issue - the kubernetes agent (2.8.0, image_: prefecthq/prefect:2.8-python3.8_) is connected with Cloud API (work queue is “Healthy”), it even successfully gets the pending flow run but during the execution it can’t update the state of the flow run with “403 Forbidden” error. Attaching the log from the agent. Any ideas, hints how debug/resolve this?
Thanks for the report! Let me raise this with someone who can look into it further.
Can you share the command you used to start your agent?
Could you also turn on DEBUG level logs?
@Zanie Thank you a lot! I'll doublecheck the command with DevOps guy. @Christopher Boyd - we'll check that.
@Christopher Boyd the key is fine:
@Zanie It seems the command is this:
prefect agent start -q default -q no-concurrency
HI Viktor - the response looks good. Is the API key a user api key, or a service account key? Regardless, what role is assigned to that user/sa ?
@Christopher Boyd - it is service account key with the runner role assigned.
I believe you need the “Create, update, and run flows” in RBAC on that role. When the flow changes state, it’s an update on the API to change the task/ flow run states
We are looking into some better terminology on this, the RUNNER role is intended to be a human using the UI to run flows, versus a service account doing so via API
Thank you Christopher, we will check that. Very similar to root cause indeed.
@Christopher Boyd, @Zanie - thanks again, you suggestions helped to resolve the original issue and the agent is shown as Healthy and not dies.