Hello Community. We have a strange issue - the kub...
# ask-communityv
Viktor Moyseyenko
02/14/2023, 10:07 AMHello Community. We have a strange issue - the kubernetes agent (2.8.0, image_: prefecthq/prefect:2.8-python3.8_) is connected with Cloud API (work queue is “Healthy”), it even successfully gets the pending flow run but during the execution it can’t update the state of the flow run with “403 Forbidden” error. Attaching the log from the agent.
Any ideas, hints how debug/resolve this?
z
Zanie
02/14/2023, 3:58 PM
Thanks for the report! Let me raise this with someone who can look into it further.
Zanie
02/14/2023, 4:03 PM
Can you share the command you used to start your agent?
Zanie
02/14/2023, 4:04 PM
Could you also turn on DEBUG level logs?
c
Christopher Boyd
02/14/2023, 4:05 PMAlso, if you can verify your api key?
https://discourse.prefect.io/t/how-can-i-tell-if-there-is-an-issue-with-my-api-key/2185
v
Viktor Moyseyenko
02/14/2023, 6:22 PM@Zanie Thank you a lot! I'll doublecheck the command with DevOps guy.
@Christopher Boyd - we'll check that.
Viktor Moyseyenko
02/14/2023, 6:33 PM@Christopher Boyd the key is fine:
✅ 1
Viktor Moyseyenko
02/14/2023, 6:40 PM@Zanie
It seems the command is this:
prefect agent start -q default -q no-concurrencyc
Christopher Boyd
02/14/2023, 7:49 PMHI Viktor - the response looks good. Is the API key a user api key, or a service account key? Regardless, what role is assigned to that user/sa ?
v
Viktor Moyseyenko
02/15/2023, 10:33 AM@Christopher Boyd - it is service account key with the runner role assigned.
c
Christopher Boyd
02/15/2023, 1:18 PMI believe you need the “Create, update, and run flows” in RBAC on that role. When the flow changes state, it’s an update on the API to change the task/ flow run states
Christopher Boyd
02/15/2023, 1:19 PMWe are looking into some better terminology on this, the RUNNER role is intended to be a human using the UI to run flows, versus a service account doing so via API
v
Viktor Moyseyenko
02/15/2023, 1:20 PMThank you Christopher, we will check that. Very similar to root cause indeed.
Viktor Moyseyenko
02/16/2023, 8:56 AM@Christopher Boyd, @Zanie - thanks again, you suggestions helped to resolve the original issue and the agent is shown as Healthy and not dies.