M Taufik
11/09/2020, 5:49 PMkubectl logs prefect-agent-dbf98c798-9mskh
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 418, in _request
json_resp = response.json()
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 426, in _request
raise ClientError("Malformed response received from API.") from exc
prefect.utilities.exceptions.ClientError: Malformed response received from API.
kubectl describe pods prefect-agent-dbf98c798-9mskh
Name: prefect-agent-dbf98c798-9mskh
Namespace: default
Priority: 0
Node: gke-prefect-cluster-default-pool-31acb6fa-s7m2/10.128.0.7
Start Time: Mon, 09 Nov 2020 17:44:00 +0000
Labels: app=prefect-agent
pod-template-hash=dbf98c798
Annotations: <none>
Status: Running
IP: xx
IPs:
IP: xx
Controlled By: ReplicaSet/prefect-agent-dbf98c798
Containers:
agent:
Container ID: <docker://d799c81d5eecef9a7f10d938d0ceeba3b7adfc57e499b8314a1f7378c5187616>
Image: prefecthq/prefect:latest
Image ID: <docker-pullable://prefecthq/prefect@sha256:7a3afaef2864945113cb82d970e8d6127412ece7df184e4b9201338b6e690250>
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
Args:
prefect agent kubernetes start
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 09 Nov 2020 17:50:37 +0000
Finished: Mon, 09 Nov 2020 17:50:45 +0000
Ready: False
Restart Count: 6
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Liveness: http-get http://:8080/api/health delay=40s timeout=1s period=40s #success=1 #failure=2
Environment:
PREFECT__CLOUD__AGENT__AUTH_TOKEN:
PREFECT__CLOUD__API: <https://api.prefect.io>
NAMESPACE: default
IMAGE_PULL_SECRETS:
PREFECT__CLOUD__AGENT__LABELS: []
JOB_MEM_REQUEST:
JOB_MEM_LIMIT:
JOB_CPU_REQUEST:
JOB_CPU_LIMIT:
IMAGE_PULL_POLICY:
SERVICE_ACCOUNT_NAME:
PREFECT__BACKEND: server
PREFECT__CLOUD__AGENT__AGENT_ADDRESS: http://:8080
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p6x2x (ro)
Conditions:
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-p6x2x:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p6x2x
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m32s default-scheduler Successfully assigned default/prefect-agent-dbf98c798-9mskh to gke-prefect-cluster-default-pool-31acb6fa-s7m2
Normal Pulled 6m21s (x4 over 7m31s) kubelet Successfully pulled image "prefecthq/prefect:latest"
Normal Created 6m21s (x4 over 7m31s) kubelet Created container agent
Normal Started 6m21s (x4 over 7m31s) kubelet Started container agent
Normal Pulling 5m33s (x5 over 7m31s) kubelet Pulling image "prefecthq/prefect:latest"
Warning BackOff 2m26s (x23 over 7m13s) kubelet Back-off restarting failed container
Chris White
11/09/2020, 5:53 PMprefect backend server
or set the appropriate config setting on your agent jobM Taufik
11/09/2020, 5:58 PM[2020-11-11 18:03:21,265] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-11 18:03:21,266] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-11 18:03:21,266] INFO - agent | Agent connecting to the Prefect API at <http://10.3.252.38:4200>
[2020-11-11 18:03:21,281] INFO - agent | Waiting for flow runs...
[2020-11-11 18:03:21,443] ERROR - agent | 400 Client Error: Bad Request for url: <http://10.3.252.38:4200>
This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:
query {
mutation($input: get_runs_in_queue_input!) {
get_runs_in_queue(input: $input) {
flow_run_ids
}
}
}
variables {
{"input": {"before": "2020-11-11T18:25:17.297587+00:00", "labels": [], "tenant_id": null}}