Hey there, Is there a general way of how to troubleshoot kubernetes agents? I spun up an

EKS cluster

using

pulumi

and created an agent locally using

prefect agent start kubernetes --token <token_id> --label k8s

. It seems like the kubernetes agent is running correctly (see attached images). However, when I submit a flow that has the same label

k8s

, it does not execute (see image)… 1. How do I make sure the kubernetes agent is set up properly? 2. Which tests does prefect cloud already perform by itself, to make sure that an agent is set up properly?

Hey @Robin first things first you probably should revoke that token you posted 😅 1. The agent appears to be set up properly because those labels look to match. Can you verify the labels on the flow page? 2. Cloud doesn’t perform any tests but the agent does “register” itself with Cloud by verifying that it is authorized to pull

Oh yeah, revoked! 😄 1. See below the label of the flow. 2. So from prefect there is no way to see whether the kubernetes cluster is set up in a way that flows can be run? It would be cool to have the option of submitting a dummy flow that checks this whenever a kubernetes agent is created…

Yeah there could have been! That error looks to be an issue with a kwarg that was introduced in the last release so either your local version of core is ahead of the version in the image or vice versa

I changed

flow.storage = Docker(registry_url="ragnarob", prefect_version="master")

to

flow.storage = Docker(registry_url="ragnarob")

Now, I get the attached error 😞 I have to mention, that I have a very unstable internet connection at the moment. Could that be a reason?

@Robin hmm generally API exceptions I have seen are due to the RBAC on the cluster. Are you able to see the logs for the prefect job?

Yes, how can I check whether rbac is set up falsely?

Try applying these permissions outlined here https://docs.prefect.io/orchestration/agents/kubernetes.html#requirements 🙂