Hello Community. We have a strange issue - the kubernetes agent (2.8.0, image_: prefecthq/prefect:2....

Viktor Moyseyenko

02/14/2023, 10:07 AM

Hello Community. We have a strange issue - the kubernetes agent (2.8.0, image_: prefecthq/prefect:2.8-python3.8_) is connected with Cloud API (work queue is “Healthy”), it even successfully gets the pending flow run but during the execution it can’t update the state of the flow run with “403 Forbidden” error. Attaching the log from the agent. Any ideas, hints how debug/resolve this?

Zanie

02/14/2023, 3:58 PM

Thanks for the report! Let me raise this with someone who can look into it further.

Zanie

02/14/2023, 4:03 PM

Can you share the command you used to start your agent?

Zanie

02/14/2023, 4:04 PM

Could you also turn on DEBUG level logs?

Christopher Boyd

02/14/2023, 4:05 PM

Also, if you can verify your api key? https://discourse.prefect.io/t/how-can-i-tell-if-there-is-an-issue-with-my-api-key/2185

Viktor Moyseyenko

02/14/2023, 6:22 PM

@Zanie Thank you a lot! I'll doublecheck the command with DevOps guy. @Christopher Boyd - we'll check that.

Viktor Moyseyenko

02/14/2023, 6:33 PM

@Christopher Boyd the key is fine:

✅ 1

Viktor Moyseyenko

02/14/2023, 6:40 PM

@Zanie It seems the command is this:

prefect agent start -q default -q no-concurrency

Christopher Boyd

02/14/2023, 7:49 PM

HI Viktor - the response looks good. Is the API key a user api key, or a service account key? Regardless, what role is assigned to that user/sa ?

Viktor Moyseyenko

02/15/2023, 10:33 AM

@Christopher Boyd - it is service account key with the runner role assigned.

Christopher Boyd

02/15/2023, 1:18 PM

I believe you need the “Create, update, and run flows” in RBAC on that role. When the flow changes state, it’s an update on the API to change the task/ flow run states

Christopher Boyd

02/15/2023, 1:19 PM

We are looking into some better terminology on this, the RUNNER role is intended to be a human using the UI to run flows, versus a service account doing so via API

Viktor Moyseyenko

02/15/2023, 1:20 PM

Thank you Christopher, we will check that. Very similar to root cause indeed.

Viktor Moyseyenko

02/16/2023, 8:56 AM

@Christopher Boyd, @Zanie - thanks again, you suggestions helped to resolve the original issue and the agent is shown as Healthy and not dies.

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.