https://prefect.io logo
Join Slack
Powered by
Hi there, I have generated a new API key and am tr...
# ask-community
b
Hi there, I have generated a new API key and am trying to use it in a Kubernetes Agent, but I'm getting an unexpected error.
Copy code
│ [2021-08-11 15:18:35,139] ERROR - agent | Failed to verify authentication.                                                                                                        │
│ /usr/local/lib/python3.7/site-packages/prefect/client/client.py:171: UserWarning: Client was created with an API token configured for authentication. API tokens are deprecated,  │
│   "Client was created with an API token configured for authentication. "                                                                                                          │
│ Traceback (most recent call last):                                                                                                                                                │
│   File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 901, in _setup_api_connection                                                                        │
│     self._verify_token(self.client.get_auth_token())                                                                                                                              │
│   File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 839, in _verify_token                                                                                │
│     raise AuthorizationError("Provided token does not have a RUNNER scope.")                                                                                                      │
│ prefect.exceptions.AuthorizationError: Provided token does not have a RUNNER scope.
As I understand it, it is not possible to give RUNNER scope to new style API keys, so I'm not clear on why I'm getting this error. Prefect version 0.15.3
k
Hey @Billy McMonagle, how did you authenticate?
👋 1
b
I have the agent set up as a deployment, with the api key as an ExternalSecret:
Copy code
spec:
      containers:
      - name: agent
        image: {{ .Values.agent.image }}
        args:
        - prefect agent kubernetes start
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          valueFrom:
            secretKeyRef:
              name: {{ .Release.Name }}
              key: prefect_api_key
m
Hello Billy! Can you exec into pod (
kubectl exec -it <pod name> -- env
) and verify that 
PREFECT__CLOUD__AGENT__AUTH_TOKEN
is set with a new service account key?
b
Hi @Mariia Kerimova, I'd love to do that but it is not possible to shell into the pod because it is in CrashLoopBackOff
I'm fairly sure it is working because this is how I mounted the old style token. I've changed the name of the key from 
prefect_runner_token
to 
prefect_api_key
so I know it is using the latest configuration. additionally, i have inspected the 
externalsecret
to verify that it's pointing at the correct key in our secret store (AWS SSM).
I've attached a different role to the service account... maybe I should regenerate the api key for that service account and see what happens.
I can replicate locally by authing with my old style token vs my new style key: token
Copy code
❯ prefect auth login -t <TOKEN>
WARNING: You logged in with an API key using the `--token` flag which is deprecated. Please use `--key` instead.
❯ python
>>> from prefect.client import Client
>>> client = Client()
>>> client.graphql(query="query { auth_info { api_token_scope } }")
{
    "data": {
        "auth_info": {
            "api_token_scope": "RUNNER"
        }
    }
}
key
Copy code
❯ prefect auth login --key <KEY>

❯ python
>>> from prefect.client import Client
>>> client = Client()
>>> client.graphql(query="query { auth_info { api_token_scope } }")
{
    "data": {
        "auth_info": {
            "api_token_scope": null
        }
    }
}
🤔 1
Looks like I should be using 
PREFECT__CLOUD__AGENT__API_KEY
instead of 
PREFECT__CLOUD__AGENT__AUTH_TOKEN
... let's see if that makes a difference.
OK that brought the agents back up!
🎉 1
m
I'll continue to investigate this issue, the 
PREFECT__CLOUD__AGENT__AUTH_TOKEN
variable should have worked, but looks like it wasn't working for you. I'll open a ticket if needed
b
Thanks @Mariia Kerimova!
I have a new error, which I don't see in the prefect console, but I am seeing in k8s directly. Going to try adding permissions to the service account and see what happens. One thing I'd love to see eventually is documentation on minimum permissions necessary for an agent to run successfully :)
Copy code
│ ERROR:agent:Error while managing existing k8s jobs                                                                                                                                
│ Traceback (most recent call last):                                                                                                                                                
│   File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 400, in heartbeat                                                                         
│     self.manage_jobs()                                                                                                                                                            
│   File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 275, in manage_jobs                                                                       
│     timestamp=event.last_timestamp.isoformat(),                                                                                                                                   
│   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1985, in write_run_logs                                                                            
│     mutation, variables=dict(input=dict(logs=logs))                                                                                                                               
│   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 564, in graphql                                                                                    
│     raise ClientError(result["errors"])                                                                                                                                           
│ prefect.exceptions.ClientError: [{'path': ['write_run_logs'], 'message': 'Unauthorized', 'extensions': {'code': 'FORBIDDEN'}}]
m
Yes, there is definitely a need to improve docs. The minimal example for agent is here https://docs.prefect.io/orchestration/agents/kubernetes.html#rbac. Also, here you can find a more extensive manifest https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/rbac.yaml
b
Ah, I should clarify - I mean prefect cloud permissions, not k8s permissions. I think I'm hitting a permissions issue on the cloud api. 
write_run_logs
seems to be the issue?
m
Oh, sorry, misunderstood 🙂 Are you on Enterprise tier? Can you check custom permissions for this role? https://docs.prefect.io/orchestration/rbac/overview.html#custom-roles-enterprise-only
b
Yes, enterprise tier. OK I think I'm good now but let me describe what I did... (1) Added permissions to my custom role attached to service account + restarted agent deployment - this did not work (2) Attached "Administrator" role to service account + restarted agent deployment - this did not work (3) Revoked + regenerated API key + restarted agent deployment - THIS WORKED.
Based on this, it seems like the permissions are scoped to whatever the service account permissions were at time of API key generation - would love to confirm this.
2 Views
Open in Slack
PreviousNext
Powered by