https://prefect.io logo
Title
m

M Taufik

11/09/2020, 5:49 PM
Hi I'm trying install prefect server in k8s cluster using helm, and the server is up. but when kubernetes agent installation i'm facing an issue, error log attached. thank you for checking, appreciate it
kubectl logs prefect-agent-dbf98c798-9mskh
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 418, in _request
    json_resp = response.json()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:

   retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 426, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.utilities.exceptions.ClientError: Malformed response received from API.
pods
kubectl describe pods prefect-agent-dbf98c798-9mskh
Name:         prefect-agent-dbf98c798-9mskh
Namespace:    default
Priority:     0
Node:         gke-prefect-cluster-default-pool-31acb6fa-s7m2/10.128.0.7
Start Time:   Mon, 09 Nov 2020 17:44:00 +0000
Labels:       app=prefect-agent
              pod-template-hash=dbf98c798
Annotations:  <none>
Status:       Running
IP:           xx
IPs:
  IP:           xx
Controlled By:  ReplicaSet/prefect-agent-dbf98c798
Containers:
  agent:
    Container ID:  <docker://d799c81d5eecef9a7f10d938d0ceeba3b7adfc57e499b8314a1f7378c5187616>
    Image:         prefecthq/prefect:latest
    Image ID:      <docker-pullable://prefecthq/prefect@sha256:7a3afaef2864945113cb82d970e8d6127412ece7df184e4b9201338b6e690250>
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
    Args:
      prefect agent kubernetes start
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 09 Nov 2020 17:50:37 +0000
      Finished:     Mon, 09 Nov 2020 17:50:45 +0000
    Ready:          False
    Restart Count:  6
Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Liveness:  http-get http://:8080/api/health delay=40s timeout=1s period=40s #success=1 #failure=2
    Environment:
      PREFECT__CLOUD__AGENT__AUTH_TOKEN:     
      PREFECT__CLOUD__API:                   <https://api.prefect.io>
      NAMESPACE:                             default
      IMAGE_PULL_SECRETS:                    
      PREFECT__CLOUD__AGENT__LABELS:         []
      JOB_MEM_REQUEST:                       
      JOB_MEM_LIMIT:                         
      JOB_CPU_REQUEST:                       
      JOB_CPU_LIMIT:                         
      IMAGE_PULL_POLICY:                     
      SERVICE_ACCOUNT_NAME:                  
      PREFECT__BACKEND:                      server
      PREFECT__CLOUD__AGENT__AGENT_ADDRESS:  http://:8080
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p6x2x (ro)
Conditions:
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-p6x2x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p6x2x
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  7m32s                   default-scheduler  Successfully assigned default/prefect-agent-dbf98c798-9mskh to gke-prefect-cluster-default-pool-31acb6fa-s7m2
  Normal   Pulled     6m21s (x4 over 7m31s)   kubelet            Successfully pulled image "prefecthq/prefect:latest"
  Normal   Created    6m21s (x4 over 7m31s)   kubelet            Created container agent
  Normal   Started    6m21s (x4 over 7m31s)   kubelet            Started container agent
  Normal   Pulling    5m33s (x5 over 7m31s)   kubelet            Pulling image "prefecthq/prefect:latest"
  Warning  BackOff    2m26s (x23 over 7m13s)  kubelet            Back-off restarting failed container
c

Chris White

11/09/2020, 5:53 PM
Hi — this suggests that you are actually hitting the Prefect Cloud API and not your locally hosted API. You’ll need to ensure you run
prefect backend server
or set the appropriate config setting on your agent job
Ah I see you have that set, maybe try updating the API value?
m

M Taufik

11/09/2020, 5:58 PM
yes already changed to prefect backend server
okay let me try, thanks
Hi @Chris White i'm trying to deploy k8s agent, but the agent can't connect to apollo/graphql, what should i check?
[2020-11-11 18:03:21,265] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-11 18:03:21,266] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-11 18:03:21,266] INFO - agent | Agent connecting to the Prefect API at <http://10.3.252.38:4200>
[2020-11-11 18:03:21,281] INFO - agent | Waiting for flow runs...
[2020-11-11 18:03:21,443] ERROR - agent | 400 Client Error: Bad Request for url: <http://10.3.252.38:4200>

This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: get_runs_in_queue_input!) {
            get_runs_in_queue(input: $input) {
                flow_run_ids
        }
    }
}
variables {
    {"input": {"before": "2020-11-11T18:25:17.297587+00:00", "labels": [], "tenant_id": null}}
from the UI able to connect to graphql
i'm using GKE and deploy prefect server using helm chart https://github.com/PrefectHQ/server/tree/helm-chart/helm/prefect-server