j

    Jacob Longhurst

    7 months ago
    Hello, My team is trying to get the k8s prefect agent to run in k8s. Whenever we spin up the agent it fails on
    registering agent
    . I took a look through the code and it seems that on spin up the agent attempts a query in the db to see if it already exists. If it doesn’t then it registers itself by adding a new record of itself to the db. We are also not currently using the prefect cloud server but rather local instances of apollo, graphql, hasura, etc running in our k8s cluster Here is the stack trace
    [2022-01-24 18:15:41,310] INFO - agent | Registering agent...
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1053, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 754, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 331, in start
        start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 140, in start_agent
        agent.start()
      File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 189, in start
        self._setup_api_connection()
      File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 910, in _setup_api_connection
        self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
      File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 858, in _register_agent
        agent_config_id=self.agent_config_id,
      File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 2107, in register_agent
        agent_config_id=agent_config_id,
      File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 570, in graphql
        raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'message': '"\'Box\' object has no attribute \'data\'"', 'locations': [{'line': 2, 'column': 5}], 'path': ['register_agent'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '"\'Box\' object has no attribute \'data\'"'}}}]
    I can also provide the yaml for the k8s
    Anna Geller

    Anna Geller

    7 months ago
    It could be several things:1. It could be a Prefect version mismatch - you need to ensure your Server is on Prefect >= your agent and flow registration environment 2. The GraphQL Server endpoint may not have been configured properly in the agent YAML file
    - name: PREFECT__CLOUD__API
              value: <http://YOUR_MACHINES_PUBLIC_IP:4200/graphql>
    j

    Jacob Longhurst

    7 months ago
    The prefect cloud api should be the apollo server correct?
    Anna Geller

    Anna Geller

    7 months ago
    I understood that you are using Server, sorry. For Cloud it would be:
    - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
    j

    Jacob Longhurst

    7 months ago
    Oh no we are using the server. We have each of the services broken out into their own deployment in k8s. So the graphql hasura, Apollo, ui, towel and agent. These are all backed behind a service and ingress route. In the agent config we set the cloud api to the url for the Apollo server. I'm wondering if it should be set to the Apollo server or to the graphql hasura container?
    Anna Geller

    Anna Geller

    7 months ago
    to the Server
    j

    Jacob Longhurst

    7 months ago
    Okay, hmm very odd that it keeps raising an attribute error in python. I’m going to try updating it to the latest version and see if that helps anything. We are currently on 0.15.6.
    Anna Geller

    Anna Geller

    7 months ago
    Sure. And if this doesn’t work, you can try the helm chart from the server repo. most users I’m aware of who run server in production, do it with this helm chart
    j

    Jacob Longhurst

    7 months ago
    Unfortunately our internal k8s clusters are locked down pretty tight and don’t allow for the helm chart to run. I would love to use the helm charts though. So if i can’t get this working i’ll most likely figure out some route to allow us to use the helm chart