I also had an issue with the k8s agent and the new API key. From the new docs it seemed like you should just remove or blank out the old AUTH_TOKEN env var, but when I created a new service account key and deployed it, I get a message like

failed to authenticate, missing token

. It works if I set the key in both variables, is this expected?

env:
- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
  value: XXXXXXXX
- name: PREFECT__CLOUD__API
  value: <https://api.prefect.io>
- name: PREFECT__BACKEND
  value: cloud
- name: PREFECT__CLOUD__API_KEY
  value: XXXXXXXX
- name: PREFECT__CLOUD__TENANT_ID
  value: TTTTTTTT
image: prefecthq/prefect:0.15.0-python3.7

Hey @Brad I, I’ll ask the team about this

Hey @Brad I -- could you give me the exact error message you saw? This should work but perhaps there's something weird about the K8s agent specifically.

Hi @Zanie, I was also getting our infra setup using Pulumi, so I’ve lost the full log but the error message was:

Failed to verify authentication

Hi Brad! Can you verify for me that if the auth token var is unset entirely, it works as expected? And that the error you’re seeing is the agent talking to Cloud directly

The code that would produce that is...

# Verify API tokens -- API keys do not need a type-check
        if config.backend == "cloud" and not self.client.api_key:
            self.logger.debug("Verifying authentication with Prefect Cloud...")
            try:
                self._verify_token(self.client.get_auth_token())
                self.logger.debug("Authentication successful!")
            except Exception as exc:
                self.logger.error("Failed to verify authentication.")
                raise RuntimeError(
                    f"Error while contacting API at {config.cloud.api}",
                ) from exc

I tried it with the

PREFECT__CLOUD__AGENT__AUTH_TOKEN

env var completely removed and with it set to an empty string, received the same error message for both variations. That looks like the same code I found when googling the error message, I didn’t have to change my API_KEY var to make it work so I’m not sure why that block executed. I can try it again at some point today when I spin up our prod agent.

Very peculiar. We'll also try to reproduce over here.

Thanks @Zanie!