Channels
#prefect-server
Title
m
M Taufik
11/09/2020, 5:49 PMHi
I'm trying install prefect server in k8s cluster using helm, and the server is up.
but when kubernetes agent installation i'm facing an issue, error log attached.
thank you for checking, appreciate it
Copy code
kubectl logs prefect-agent-dbf98c798-9mskh
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 418, in _request
json_resp = response.json()
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 426, in _request
raise ClientError("Malformed response received from API.") from exc
prefect.utilities.exceptions.ClientError: Malformed response received from API.pods
Copy code
kubectl describe pods prefect-agent-dbf98c798-9mskh
Name: prefect-agent-dbf98c798-9mskh
Namespace: default
Priority: 0
Node: gke-prefect-cluster-default-pool-31acb6fa-s7m2/10.128.0.7
Start Time: Mon, 09 Nov 2020 17:44:00 +0000
Labels: app=prefect-agent
pod-template-hash=dbf98c798
Annotations: <none>
Status: Running
IP: xx
IPs:
IP: xx
Controlled By: ReplicaSet/prefect-agent-dbf98c798
Containers:
agent:
Container ID: <docker://d799c81d5eecef9a7f10d938d0ceeba3b7adfc57e499b8314a1f7378c5187616>
Image: prefecthq/prefect:latest
Image ID: <docker-pullable://prefecthq/prefect@sha256:7a3afaef2864945113cb82d970e8d6127412ece7df184e4b9201338b6e690250>
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
Args:
prefect agent kubernetes start
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 09 Nov 2020 17:50:37 +0000
Finished: Mon, 09 Nov 2020 17:50:45 +0000
Ready: False
Restart Count: 6
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Liveness: http-get http://:8080/api/health delay=40s timeout=1s period=40s #success=1 #failure=2
Environment:
PREFECT__CLOUD__AGENT__AUTH_TOKEN:
PREFECT__CLOUD__API: <https://api.prefect.io>
NAMESPACE: default
IMAGE_PULL_SECRETS:
PREFECT__CLOUD__AGENT__LABELS: []
JOB_MEM_REQUEST:
JOB_MEM_LIMIT:
JOB_CPU_REQUEST:
JOB_CPU_LIMIT:
IMAGE_PULL_POLICY:
SERVICE_ACCOUNT_NAME:
PREFECT__BACKEND: server
PREFECT__CLOUD__AGENT__AGENT_ADDRESS: http://:8080
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p6x2x (ro)
Conditions:
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-p6x2x:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p6x2x
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m32s default-scheduler Successfully assigned default/prefect-agent-dbf98c798-9mskh to gke-prefect-cluster-default-pool-31acb6fa-s7m2
Normal Pulled 6m21s (x4 over 7m31s) kubelet Successfully pulled image "prefecthq/prefect:latest"
Normal Created 6m21s (x4 over 7m31s) kubelet Created container agent
Normal Started 6m21s (x4 over 7m31s) kubelet Started container agent
Normal Pulling 5m33s (x5 over 7m31s) kubelet Pulling image "prefecthq/prefect:latest"
Warning BackOff 2m26s (x23 over 7m13s) kubelet Back-off restarting failed containerc
Chris White
11/09/2020, 5:53 PM
Hi — this suggests that you are actually hitting the Prefect Cloud API and not your locally hosted API. You’ll need to ensure you run
prefect backend serverAh I see you have that set, maybe try updating the API value?
m
M Taufik
11/09/2020, 5:58 PMyes already changed to prefect backend server
okay let me try, thanks
Hi @Chris White
i'm trying to deploy k8s agent, but the agent can't connect to apollo/graphql, what should i check?
Copy code
[2020-11-11 18:03:21,265] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-11 18:03:21,266] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-11 18:03:21,266] INFO - agent | Agent connecting to the Prefect API at <http://10.3.252.38:4200>
[2020-11-11 18:03:21,281] INFO - agent | Waiting for flow runs...
[2020-11-11 18:03:21,443] ERROR - agent | 400 Client Error: Bad Request for url: <http://10.3.252.38:4200>
This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:
query {
mutation($input: get_runs_in_queue_input!) {
get_runs_in_queue(input: $input) {
flow_run_ids
}
}
}
variables {
{"input": {"before": "2020-11-11T18:25:17.297587+00:00", "labels": [], "tenant_id": null}}from the UI able to connect to graphql
i'm using GKE and deploy prefect server using helm chart https://github.com/PrefectHQ/server/tree/helm-chart/helm/prefect-server
3 Views