Has anyone ever run into this issue when trying to apply a kubernetes agent?

prefect.exceptions.ClientError: Malformed response received from API.

Here is the full stack trace:

return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
    agent.start()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
    tenant_id=self.tenant_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
    self._tenant_id = self._get_default_server_tenant()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
    response = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.

Here is the .yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  namespace: prefect
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start --job-template "<s3://xxx>"
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: prefect
        - name: IMAGE_PULL_SECRETS
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[''xxx'', ''xxxxx'']'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: Always
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: cloud
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        - name: PREFECT__CLOUD__API_KEY
          value: xxx
        - name: PREFECT__CLOUD__TENANT_ID
          value: xxx
        - name: PREFECT__LOGGING__EXTRA_LOGGERS
          value: "['xxxxx', 'xxxxxxxx']"
        image: prefecthq/prefect:1.2.2-python3.7
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent

Fantastic 😄