kubectl logs prefect-agent-dbf98c798-9mskh
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 418, in _request
    json_resp = response.json()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:

   retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 426, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.utilities.exceptions.ClientError: Malformed response received from API.

kubectl describe pods prefect-agent-dbf98c798-9mskh
Name:         prefect-agent-dbf98c798-9mskh
Namespace:    default
Priority:     0
Node:         gke-prefect-cluster-default-pool-31acb6fa-s7m2/10.128.0.7
Start Time:   Mon, 09 Nov 2020 17:44:00 +0000
Labels:       app=prefect-agent
              pod-template-hash=dbf98c798
Annotations:  <none>
Status:       Running
IP:           xx
IPs:
  IP:           xx
Controlled By:  ReplicaSet/prefect-agent-dbf98c798
Containers:
  agent:
    Container ID:  <docker://d799c81d5eecef9a7f10d938d0ceeba3b7adfc57e499b8314a1f7378c5187616>
    Image:         prefecthq/prefect:latest
    Image ID:      <docker-pullable://prefecthq/prefect@sha256:7a3afaef2864945113cb82d970e8d6127412ece7df184e4b9201338b6e690250>
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
    Args:
      prefect agent kubernetes start
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 09 Nov 2020 17:50:37 +0000
      Finished:     Mon, 09 Nov 2020 17:50:45 +0000
    Ready:          False
    Restart Count:  6
Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Liveness:  http-get http://:8080/api/health delay=40s timeout=1s period=40s #success=1 #failure=2
    Environment:
      PREFECT__CLOUD__AGENT__AUTH_TOKEN:     
      PREFECT__CLOUD__API:                   <https://api.prefect.io>
      NAMESPACE:                             default
      IMAGE_PULL_SECRETS:                    
      PREFECT__CLOUD__AGENT__LABELS:         []
      JOB_MEM_REQUEST:                       
      JOB_MEM_LIMIT:                         
      JOB_CPU_REQUEST:                       
      JOB_CPU_LIMIT:                         
      IMAGE_PULL_POLICY:                     
      SERVICE_ACCOUNT_NAME:                  
      PREFECT__BACKEND:                      server
      PREFECT__CLOUD__AGENT__AGENT_ADDRESS:  http://:8080
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p6x2x (ro)
Conditions:
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-p6x2x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p6x2x
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  7m32s                   default-scheduler  Successfully assigned default/prefect-agent-dbf98c798-9mskh to gke-prefect-cluster-default-pool-31acb6fa-s7m2
  Normal   Pulled     6m21s (x4 over 7m31s)   kubelet            Successfully pulled image "prefecthq/prefect:latest"
  Normal   Created    6m21s (x4 over 7m31s)   kubelet            Created container agent
  Normal   Started    6m21s (x4 over 7m31s)   kubelet            Started container agent
  Normal   Pulling    5m33s (x5 over 7m31s)   kubelet            Pulling image "prefecthq/prefect:latest"
  Warning  BackOff    2m26s (x23 over 7m13s)  kubelet            Back-off restarting failed container

[2020-11-11 18:03:21,265] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-11 18:03:21,266] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-11 18:03:21,266] INFO - agent | Agent connecting to the Prefect API at <http://10.3.252.38:4200>
[2020-11-11 18:03:21,281] INFO - agent | Waiting for flow runs...
[2020-11-11 18:03:21,443] ERROR - agent | 400 Client Error: Bad Request for url: <http://10.3.252.38:4200>

This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: get_runs_in_queue_input!) {
            get_runs_in_queue(input: $input) {
                flow_run_ids
        }
    }
}
variables {
    {"input": {"before": "2020-11-11T18:25:17.297587+00:00", "labels": [], "tenant_id": null}}