https://prefect.io logo
Powered by
Title
j

Josef Trefil

10/28/2020, 11:25 PM
Hi everyone! I got stuck at Scaling Out with Kubernetes tutorial (running Kubernetes on Docker Desktop). When I run my flow from the UI, Prefect Agent gives me 
Deploying flow run ...
and that's it. When I then check 
kubectl get all
, the prefect-agent pod and deployment both show 
READY 0/1
When I check 
kubectl logs deployment/prefect-agent
the very last line of the traceback says: 
prefect.utilities.exceptions.ClientError: Malformed response received from API.
Any idea what I'm doing wrong? 🤷‍♂️ Thank you so much for any clues! 🙂
c

Chris White

10/28/2020, 11:28 PM
Hi @Josef Trefil - the 
Malformed response
error suggests that your agent submitted the flow run to run against Prefect Cloud but did not provide a token to the run in order to communicate; are you attempting to run this with Cloud or a locally setup Server?
j

Josef Trefil

10/28/2020, 11:33 PM
Hello @Chris White , thank you for such a quick response. I run it locally. Server setup.
c

Chris White

10/28/2020, 11:38 PM
so a few notes: - FYI running k8s jobs against Prefect server can be challenging as it requires that the pods / jobs have network access to the API, which is something you’ll need to ensure - it sounds like somehow the agent doesn’t have the appropriate backend configuration; did you run 
prefect backend server
or set 
PREFECT__BACKEND=server
in the agent’s environment? Once set, the agent should then provide that config into the jobs it creates so that they talk to your local API instead of the Cloud API
j

Josef Trefil

10/28/2020, 11:42 PM
Yes, I did run 
prefect backend server
Then i ran: 
prefect server start
prefect agent install kubernetes --rbac | kubectl apply -f -
prefect agent start kubernetes
c

Chris White

10/28/2020, 11:44 PM
so that appears to create two agents: one that you run within the k8s cluster and another that you’re running locally; given the traceback you provided I assume it’s the installed agent that is throwing the error, in which case I recommend inspecting the output of 
prefect agent install kubernetes --rbac
and adding 
PREFECT__BACKEND=server
as an environment variable in the manifest
j

Josef Trefil

10/29/2020, 12:03 AM
So 
PREFECT__BACKEND=server
is there already. 😞 This is the output of 
prefect agent install kubernetes --rbac
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent start kubernetes
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: default
        - name: IMAGE_PULL_SECRETS
          value: ''
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[]'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: ''
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: server
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        image: prefecthq/prefect:0.13.9-python3.6
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-agent-rbac
  namespace: default
rules:
- apiGroups:
  - batch
  - extensions
  resources:
  - jobs
  verbs:
  - '*'
- apiGroups:
  - ''
  resources:
  - pods
  verbs:
  - '*'
---
apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>
kind: RoleBinding
metadata:
  name: prefect-agent-rbac
  namespace: default
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-agent-rbac
subjects:
- kind: ServiceAccount
  name: default
c

Chris White

10/29/2020, 12:19 AM
very interesting; let me confirm that the k8s agent is configured to pass the backend config to the jobs…
AH! This is indeed a bug; it appears that the k8s agent does not pass the backend configuration down. I’ll open an issue, this should be easy to resolve - and just to set expectations, the reason for this is that the k8s agent isn’t fully supported with Prefect Server because of the network requirement that the jobs have access to the API you are serving with Server. However, given that you’re running k8s locally it sounds like it will run just fine once this is addressed 👍 @Marvin open “Kubernetes Agent does not respect backend configuration in the jobs that it submits”
m

Marvin

10/29/2020, 12:22 AM
https://github.com/PrefectHQ/prefect/issues/3586
c

Chris White

10/29/2020, 12:28 AM
In the meantime, if you edit the start command for the agent deployment to: 
prefect agent start kubernetes -e PREFECT__BACKEND=server
i believe it should work as you expect
j

Josef Trefil

10/29/2020, 2:10 PM
Hi @Chris White so I've tried the command but unfortunatelly the behavior is unchanged. I noticed, that when I start server and agent using all the commands mentioned above, the first time I run 
kubectl get all
the pod, deployment and replicaset show READY=1, but when I run the same command right after, they show READY=0. No matter whether I try to run a flow or not... Do I understand correctly, that in my case, it would be easier to use Cloud instead of Server? I can try that...
c

Chris White

10/29/2020, 3:46 PM
Hmmmm that’s interesting; yes that’s correct, it would be much easier to run using Cloud with a kubernetes agent.
j

Josef Trefil

10/29/2020, 4:19 PM
Cloud works. 🙂👍
💯 1
2 Views
#prefect-serverJoin Slack
Powered by