Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
v
Viktor Moyseyenko
02/14/2023, 10:07 AMHello Community. We have a strange issue - the kubernetes agent (2.8.0, image_: prefecthq/prefect:2.8-python3.8_) is connected with Cloud API (work queue is “Healthy”), it even successfully gets the pending flow run but during the execution it can’t update the state of the flow run with “403 Forbidden” error. Attaching the log from the agent.
Any ideas, hints how debug/resolve this?
z
Zanie
02/14/2023, 3:58 PMThanks for the report! Let me raise this with someone who can look into it further.
Can you share the command you used to start your agent?
Could you also turn on DEBUG level logs?
c
Christopher Boyd
02/14/2023, 4:05 PMAlso, if you can verify your api key?
https://discourse.prefect.io/t/how-can-i-tell-if-there-is-an-issue-with-my-api-key/2185
v
Viktor Moyseyenko
02/14/2023, 6:22 PM@Zanie Thank you a lot! I'll doublecheck the command with DevOps guy.
@Christopher Boyd - we'll check that.
@Christopher Boyd the key is fine:
✅ 1
@Zanie
It seems the command is this:
prefect agent start -q default -q no-concurrencyc
Christopher Boyd
02/14/2023, 7:49 PMHI Viktor - the response looks good. Is the API key a user api key, or a service account key? Regardless, what role is assigned to that user/sa ?
v
Viktor Moyseyenko
02/15/2023, 10:33 AM@Christopher Boyd - it is service account key with the runner role assigned.
c
Christopher Boyd
02/15/2023, 1:18 PMI believe you need the “Create, update, and run flows” in RBAC on that role. When the flow changes state, it’s an update on the API to change the task/ flow run states
We are looking into some better terminology on this, the RUNNER role is intended to be a human using the UI to run flows, versus a service account doing so via API
v
Viktor Moyseyenko
02/15/2023, 1:20 PMThank you Christopher, we will check that. Very similar to root cause indeed.
@Christopher Boyd, @Zanie - thanks again, you suggestions helped to resolve the original issue and the agent is shown as Healthy and not dies.