https://prefect.io logo
Powered by
Title
a

Aaron Goebel

05/26/2022, 11:11 PM
I'm triggering a flow from the cloud UI to run on kubernetes. Kubernetes agent is authenticated. The kubernetes agent is spawning jobs, but they fail with this error:
Traceback (most recent call last):                                         │
│   File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 9 │
│     return complexjson.loads(self.text, **kwargs)                          │
│   File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads     │
│     return _default_decoder.decode(s)                                      │
│   File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())                      │
│   File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode │
│     raise JSONDecodeError("Expecting value", s, err.value) from None       │
│ json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)    │
│                                                                            │
│ During handling of the above exception, another exception occurred:        │
│                                                                            │
│ Traceback (most recent call last):                                         │
│   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py",  │
│     json_resp = response.json()                                            │
│   File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 9 │
│     raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)                     │
│ requests.exceptions.JSONDecodeError: [Errno Expecting value] <html><head>< │
│ ;;window.rbzns={bereshit: "1", seed: "ExxyQQQD1rSDX5MexzAkXeQUPJG05r9sjzWM │
│ : 0                                                                        │
│                                                                            │
│ The above exception was the direct cause of the following exception:       │
│                                                                            │
│ Traceback (most recent call last):                                         │
│   File "/usr/local/bin/prefect", line 8, in <module>                       │
│     sys.exit(cli())                                                        │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130,  │
│     return self.main(*args, **kwargs)                                      │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055,  │
│     rv = self.invoke(ctx)                                                  │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657,  │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657,  │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404,  │
│     return ctx.invoke(self.callback, **ctx.params)                         │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, i │
│     return __callback(*args, **kwargs)                                     │
│   File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", li │
│     result = client.graphql(query)                                         │
│   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py",  │
│     result = <http://self.post|self.post>(                                                    │
│   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py",  │
│     response = self._request(                                              │
│   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py",  │
│     raise AuthorizationError(                                              │
│ prefect.exceptions.AuthorizationError: Malformed response received from Cl │
│ Stream closed EOF for default/prefect-job-dc313633-l9ttb (flow)
k

Kevin Kho

05/26/2022, 11:11 PM
Oh you have just edited the previous message but no worries
Thanks! How did you authenticate the agent?
a

Aaron Goebel

05/26/2022, 11:12 PM
I deployed a k8s resource like such for the agent 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start 
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: KEY
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: default
        - name: IMAGE_PULL_SECRETS
          value: ''
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[]'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: ''
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: cloud
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        image: prefecthq/prefect:latest-python3.8
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent
---
kind: Role
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: prefect-agent-rbac
  namespace: default
rules:
- apiGroups:
  - batch
  - extensions
  resources:
  - jobs
  verbs:
  - '*'
- apiGroups:
  - ''
  resources:
  - events
  - pods
  verbs:
  - '*'
- apiGroups:
  - ""  # indicates the core API group
  resources:
  - "pods/log"
  verbs:
  - "get"
  - "list"
- apiGroups:
  - "" # indicates the core API group
  resources:
  - "services"
  verbs:
  - "get"
  - "list"
  - "watch"
  - "create"
  - "delete"
- apiGroups:
  - "policy"  # indicates the policy API group
  resources:
  - "poddisruptionbudgets"
  verbs:
  - "get"
  - "list"
  - "watch"
  - "create"
  - "delete"
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-agent-rbac
  namespace: default
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-agent-rbac
subjects:
- kind: ServiceAccount
  name: default
it initially wasn't logged in but I ran 
prefect auth login -k {API_KEY}
and successfully authenticated
Triggering the flow from the UI is causing the agent to spin up job pods
k

Kevin Kho

05/26/2022, 11:15 PM
But I think the pods don’t end up authenticated right? Instead of auth_token, can you use: 
- name: PREFECT__CLOUD__API_KEY
          value: ''
instead and see if that helps?
What Prefect version are you on and where did you make your token/key?
a

Aaron Goebel

05/26/2022, 11:16 PM
version 1.2.2 and the token was made in the cloud UI under service accounts
I'm just getting started w/ prefect and know I'll want to use dask so I've been using some starter material referencing prefect images invoking Dask on k8s. But it looks like Orion is in a separate repository and that I may want to start with that?
k

Kevin Kho

05/26/2022, 11:18 PM
Ok that’s right. I would try using Key. 0.15.0 and up you should use Key instead of Token
💯 1
a

Aaron Goebel

05/26/2022, 11:18 PM
Ok your env variable change worked
k

Kevin Kho

05/26/2022, 11:21 PM
For Dask, you can use it in both 1.0 and 2.0. You can try 2.0 already here. There is an article here you can read that may help you deciding. If you need something in production and battle tested today, 1.0 would be the choice. If you have simple use cases or use-cases that can’t be done by 1.0, then you can try 2.0
5 Views
#best-practices-coordination-planeJoin Slack
Powered by