Hi i am unable to start my Prefect Server getting below erro Prefect Community #prefect-community

Hi i am unable to start my Prefect Server, getting...

Muddassir Shaikh

07/06/2022, 2:15 PM

Hi i am unable to start my Prefect Server, getting below error:

Copy code

graphql_1   | ValueError: Uniqueness violation.
graphql_1   |
graphql_1   | The above exception was the direct cause of the following exception:
graphql_1   |
graphql_1   | Traceback (most recent call last):
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 674, in await_completed
graphql_1   |     return await completed
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 659, in await_result
graphql_1   |     return_type, field_nodes, info, path, await result
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 733, in complete_value
graphql_1   |     raise result
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
graphql_1   |     return await result
graphql_1   |   File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
graphql_1   |     result = await result
graphql_1   |   File "/prefect-server/src/prefect_server/graphql/tenants.py", line 14, in resolve_create_tenant
graphql_1   |     "id": await api.tenants.create_tenant(name=input["name"], slug=input["slug"])
graphql_1   |   File "/prefect-server/src/prefect_server/api/tenants.py", line 29, in create_tenant
graphql_1   |     tenant_id = await models.Tenant(name=name, slug=slug).insert()
graphql_1   |   File "/prefect-server/src/prefect_server/database/orm.py", line 222, in insert
graphql_1   |     insert_mutation_name=self.__root_fields__.get("insert"),
graphql_1   |   File "/prefect-server/src/prefect_server/database/hasura.py", line 237, in insert
graphql_1   |     result = await self.execute_mutations_in_transaction(mutations=[graphql])
graphql_1   |   File "/prefect-server/src/prefect_server/database/hasura.py", line 165, in execute_mutations_in_transaction
graphql_1   |     as_box=as_box,
graphql_1   |   File "/prefect-server/src/prefect_server/database/hasura.py", line 89, in execute
graphql_1   |     raise ValueError("Uniqueness violation.")
graphql_1   | graphql.error.graphql_error.GraphQLError: Uniqueness violation.
graphql_1   |
graphql_1   | GraphQL request:2:3
graphql_1   | 1 | mutation ($input: create_tenant_input!) {
graphql_1   | 2 |   create_tenant(input: $input) {
graphql_1   |   |   ^
graphql_1   | 3 |     id

Kevin Kho

07/06/2022, 2:33 PM

Is this a fresh install?

Muddassir Shaikh

07/06/2022, 2:44 PM

no, my server was working fine for many months. Crashed today

Kevin Kho

07/06/2022, 3:01 PM

What is your startup command?

Muddassir Shaikh

07/06/2022, 3:06 PM

prefect server start --expose

Kevin Kho

07/06/2022, 3:08 PM

It looks like maybe some containers are still alive? Maybe you can try

prefect server stop

first?

Muddassir Shaikh

07/06/2022, 3:11 PM

okay then?

Kevin Kho

07/06/2022, 3:12 PM

Then try starting again?

Muddassir Shaikh

07/06/2022, 3:12 PM

what if i run

docker system prune -a

Kevin Kho

07/06/2022, 3:13 PM

That removes containers that are off but you need to make sure the prefect related containers are off, which is what the stop does

Muddassir Shaikh

07/06/2022, 3:14 PM

okay

Muddassir Shaikh

07/06/2022, 3:32 PM

one of the docker restarts automatically prefecthq/apollo:core-0.15.11

Copy code

[root@ts-mum1-dsr33 301cd24247730e6518e404be5f20f9bf1c776eedf99ad73ef20f0e37603a2484]# docker ps

CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                        PORTS                            NAMES

12be256528ad   prefecthq/ui:core-0.15.11       "/docker-entrypoint.…"   11 minutes ago   Up 11 minutes (healthy)       80/tcp, 0.0.0.0:8080->8080/tcp   tmp_ui_1

4850e7e29838   prefecthq/apollo:core-0.15.11   "tini -g -- bash -c …"   11 minutes ago   Up About a minute (healthy)   0.0.0.0:4200->4200/tcp           tmp_apollo_1

c1608fa40151   prefecthq/server:core-0.15.11   "tini -g -- python s…"   11 minutes ago   Up 11 minutes                                                  tmp_towel_1

62592addb09f   prefecthq/server:core-0.15.11   "tini -g -- bash -c …"   11 minutes ago   Up 11 minutes (healthy)       127.0.0.1:4201->4201/tcp         tmp_graphql_1

3500079088e3   hasura/graphql-engine:v1.3.3    "graphql-engine serve"   11 minutes ago   Up 11 minutes (healthy)       127.0.0.1:3000->3000/tcp         tmp_hasura_1

36a9cacece03   postgres:11                     "docker-entrypoint.s…"   11 minutes ago   Up 11 minutes (healthy)       127.0.0.1:5432->5432/tcp         tmp_postgres_1

Kevin Kho

07/06/2022, 3:33 PM

I think that might be fine? Does it give an error?

Muddassir Shaikh

07/06/2022, 3:33 PM

Copy code

[9:03 PM] PayloadTooLargeError: request entity too large
    at readStream (/apollo/node_modules/raw-body/index.js:155:17)
    at getRawBody (/apollo/node_modules/raw-body/index.js:108:12)
    at read (/apollo/node_modules/body-parser/lib/read.js:77:3)
    at jsonParser (/apollo/node_modules/body-parser/lib/types/json.js:135:5)
    at Layer.handle [as handle_request] (/apollo/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/apollo/node_modules/express/lib/router/index.js:317:13)
    at /apollo/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/apollo/node_modules/express/lib/router/index.js:335:12)
    at next (/apollo/node_modules/express/lib/router/index.js:275:10)
    at cors (/apollo/node_modules/cors/lib/index.js:188:7)

Kevin Kho

07/06/2022, 3:35 PM

This is upon server start or when running a flow?

Muddassir Shaikh

07/06/2022, 3:36 PM

this is inside prefecthq/apollo:core-0.15.11

Kevin Kho

07/06/2022, 3:37 PM

I know but is it immediately after you start the server? or are you running a flow when this happens?

Muddassir Shaikh

07/06/2022, 3:38 PM

after starting the server

Muddassir Shaikh

07/06/2022, 3:38 PM

even agents are not able to connect

Kevin Kho

07/06/2022, 3:40 PM

Is this log the current time (or close to it) where you are? 9:03 PM?

Muddassir Shaikh

07/06/2022, 3:41 PM

that is slack timing, by copying message

Kevin Kho

07/06/2022, 3:42 PM

Ah ok I see

Kevin Kho

07/06/2022, 3:44 PM

I am pretty confused because this error is normally when you try to send something like a very big result or a lot of logs in a Prefect flow so I don’t know what is hitting the API when you start (aside from tenant creation)

Kevin Kho

07/06/2022, 3:45 PM

So the container restarts and you just get the same error again and again?

Muddassir Shaikh

07/06/2022, 3:47 PM

yes i when try to start the agent i get this error

Copy code

Traceback (most recent call last):
  File "/home/infra/prefect_server/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/home/infra/prefect_server/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/infra/prefect_server/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/infra/prefect_server/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/infra/prefect_server/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/cli/agent.py", line 182, in start
    start_agent(LocalAgent, import_paths=list(import_paths), **kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/cli/agent.py", line 140, in start_agent
    agent.start()
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/agent/agent.py", line 189, in start
    self._setup_api_connection()
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/agent/agent.py", line 910, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/agent/agent.py", line 854, in _register_agent
    agent_id = self.client.register_agent(
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/client/client.py", line 2099, in register_agent
    result = self.graphql(
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/client/client.py", line 549, in graphql
    result = <http://self.post|self.post>(
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/client/client.py", line 452, in post
    response = self._request(
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/client/client.py", line 739, in _request
    response = self._send_request(
  File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/client/client.py", line 603, in _send_request
    response = <http://session.post|session.post>(
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='10.40.1.43', port=4200): Read timed out. (read timeout=15)
Agent Stopped....  Wed Jul  6 21:15:44 IST 2022
[infra@ts-mum1-dsr32 ~]>:

Kevin Kho

07/06/2022, 3:50 PM

That makes sense since it’s saying it can’t connect to the API. It’s the apollo pod that needs to be looked at

Muddassir Shaikh

07/06/2022, 3:52 PM

yup

Kevin Kho

07/06/2022, 3:53 PM

Do you potentially have some flow still running or some agent still running and they are trying to update their state to that API?

Muddassir Shaikh

07/06/2022, 3:55 PM

yes i had 5-6 agents connected and running flows before server crashed.

Kevin Kho

07/06/2022, 3:59 PM

i am not 100% sure but what this looks like is you had a flow that crashed the apollo pod because it was sending a lot of logs or it was sending a huge result. The server went down, and you restart it, but those Flows are still trying to update the result causing it to crash again. So I think you may need to stop those first so apollo can restart

4 Views

Open in Slack

Previous Next