Hello, My team is trying to get the k8s prefect agent to run in k8s. Whenever we spin up the agent it fails on

registering agent

. I took a look through the code and it seems that on spin up the agent attempts a query in the db to see if it already exists. If it doesn’t then it registers itself by adding a new record of itself to the db. We are also not currently using the prefect cloud server but rather local instances of apollo, graphql, hasura, etc running in our k8s cluster Here is the stack trace

[2022-01-24 18:15:41,310] INFO - agent | Registering agent...
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 331, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 140, in start_agent
    agent.start()
  File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 189, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 910, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 858, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 2107, in register_agent
    agent_config_id=agent_config_id,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 570, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': '"\'Box\' object has no attribute \'data\'"', 'locations': [{'line': 2, 'column': 5}], 'path': ['register_agent'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '"\'Box\' object has no attribute \'data\'"'}}}]

I can also provide the yaml for the k8s

It could be several things: 1. It could be a Prefect version mismatch - you need to ensure your Server is on Prefect >= your agent and flow registration environment 2. The GraphQL Server endpoint may not have been configured properly in the agent YAML file

- name: PREFECT__CLOUD__API
          value: <http://YOUR_MACHINES_PUBLIC_IP:4200/graphql>

The prefect cloud api should be the apollo server correct?

I understood that you are using Server, sorry. For Cloud it would be:

- name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>

Oh no we are using the server. We have each of the services broken out into their own deployment in k8s. So the graphql hasura, Apollo, ui, towel and agent. These are all backed behind a service and ingress route. In the agent config we set the cloud api to the url for the Apollo server. I'm wondering if it should be set to the Apollo server or to the graphql hasura container?

to the Server

Okay, hmm very odd that it keeps raising an attribute error in python. I’m going to try updating it to the latest version and see if that helps anything. We are currently on 0.15.6.

Sure. And if this doesn’t work, you can try the helm chart from the server repo. most users I’m aware of who run server in production, do it with this helm chart

Unfortunately our internal k8s clusters are locked down pretty tight and don’t allow for the helm chart to run. I would love to use the helm charts though. So if i can’t get this working i’ll most likely figure out some route to allow us to use the helm chart