Hi i am unable to start my Prefect Server getting below erro

Question

Hi i am unable to start my Prefect Server getting below error ```graphql 1 | ValueError Uniqueness violation graphql 1 | graphql 1 | The above exception was the direct cause of the following exception graphql 1 | graphql 1 | Traceback most recent call last graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 674 in await completed graphql 1 | return await completed graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 659 in await result graphql 1 | return type field nodes info path await result graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 733 in complete value graphql 1 | raise result graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 628 in await result graphql 1 | return await result graphql 1 | File prefect server src prefect server graphql extensions py line 52 in resolve graphql 1 | result = await result graphql 1 | File prefect server src prefect server graphql tenants py line 14 in resolve create tenant graphql 1 | id await api tenants create tenant name=input name slug=input slug graphql 1 | File prefect server src prefect server api tenants py line 29 in create tenant graphql 1 | tenant id = await models Tenant name=name slug=slug insert graphql 1 | File prefect server src prefect server database orm py line 222 in insert graphql 1 | insert mutation name=self root fields get insert graphql 1 | File prefect server src prefect server database hasura py line 237 in insert graphql 1 | result = await self execute mutations in transaction mutations= graphql graphql 1 | File prefect server src prefect server database hasura py line 165 in execute mutations in transaction graphql 1 | as box=as box graphql 1 | File prefect server src prefect server database hasura py line 89 in execute graphql 1 | raise ValueError Uniqueness violation graphql 1 | graphql error graphql error GraphQLError Uniqueness violation graphql 1 | graphql 1 | GraphQL request 2 3 graphql 1 | 1 | mutation $input create tenant input graphql 1 | 2 | create tenant input $input graphql 1 | | ^ graphql 1 | 3 | id```

Kevin Kho · Answer

Is this a fresh install

Muddassir Shaikh · Answer

no my server was working fine for many months Crashed today

Kevin Kho · Answer

What is your startup command

Muddassir Shaikh · Answer

prefect server start expose

Kevin Kho · Answer

It looks like maybe some containers are still alive Maybe you can try `prefect server stop` first

Muddassir Shaikh · Answer

okay then

Kevin Kho · Answer

Then try starting again

Muddassir Shaikh · Answer

what if i run `docker system prune a`

Kevin Kho · Answer

That removes containers that are off but you need to make sure the prefect related containers are off which is what the stop does

Muddassir Shaikh · Answer

okay

Muddassir Shaikh · Answer

one of the docker restarts automatically prefecthq apollo core 0 15 11 ``` root ts mum1 dsr33 301cd24247730e6518e404be5f20f9bf1c776eedf99ad73ef20f0e37603a2484 docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 12be256528ad prefecthq ui core 0 15 11 docker entrypoint 11 minutes ago Up 11 minutes healthy 80 tcp 0 0 0 0 8080 gt 8080 tcp tmp ui 1 4850e7e29838 prefecthq apollo core 0 15 11 tini g bash c 11 minutes ago Up About a minute healthy 0 0 0 0 4200 gt 4200 tcp tmp apollo 1 c1608fa40151 prefecthq server core 0 15 11 tini g python s 11 minutes ago Up 11 minutes tmp towel 1 62592addb09f prefecthq server core 0 15 11 tini g bash c 11 minutes ago Up 11 minutes healthy 127 0 0 1 4201 gt 4201 tcp tmp graphql 1 3500079088e3 hasura graphql engine v1 3 3 graphql engine serve 11 minutes ago Up 11 minutes healthy 127 0 0 1 3000 gt 3000 tcp tmp hasura 1 36a9cacece03 postgres 11 docker entrypoint s 11 minutes ago Up 11 minutes healthy 127 0 0 1 5432 gt 5432 tcp tmp postgres 1```

Kevin Kho · Answer

I think that might be fine Does it give an error

Muddassir Shaikh · Answer

``` 9 03 PM PayloadTooLargeError request entity too large at readStream apollo node modules raw body index js 155 17 at getRawBody apollo node modules raw body index js 108 12 at read apollo node modules body parser lib read js 77 3 at jsonParser apollo node modules body parser lib types json js 135 5 at Layer handle as handle request apollo node modules express lib router layer js 95 5 at trim prefix apollo node modules express lib router index js 317 13 at apollo node modules express lib router index js 284 7 at Function process params apollo node modules express lib router index js 335 12 at next apollo node modules express lib router index js 275 10 at cors apollo node modules cors lib index js 188 7 ```

Kevin Kho · Answer

This is upon server start or when running a flow

Muddassir Shaikh · Answer

this is inside `prefecthq apollo core 0 15 11`

Kevin Kho · Answer

I know but is it immediately after you start the server or are you running a flow when this happens

Muddassir Shaikh · Answer

after starting the server

Muddassir Shaikh · Answer

even agents are not able to connect

Kevin Kho · Answer

Is this log the current time or close to it where you are 9 03 PM

Muddassir Shaikh · Answer

that is slack timing by copying message

Kevin Kho · Answer

Ah ok I see

Kevin Kho · Answer

I am pretty confused because this error is normally when you try to send something like a very big result or a lot of logs in a Prefect flow so I don t know what is hitting the API when you start aside from tenant creation

Kevin Kho · Answer

So the container restarts and you just get the same error again and again

Muddassir Shaikh · Answer

yes i when try to start the agent i get this error ```Traceback most recent call last File home infra prefect server bin prefect line 8 in lt module gt sys exit cli File home infra prefect server lib python3 8 site packages click core py line 1128 in call return self main args kwargs File home infra prefect server lib python3 8 site packages click core py line 1053 in main rv = self invoke ctx File home infra prefect server lib python3 8 site packages click core py line 1659 in invoke return process result sub ctx command invoke sub ctx File home infra prefect server lib python3 8 site packages click core py line 1659 in invoke return process result sub ctx command invoke sub ctx File home infra prefect server lib python3 8 site packages click core py line 1659 in invoke return process result sub ctx command invoke sub ctx File home infra prefect server lib python3 8 site packages click core py line 1395 in invoke return ctx invoke self callback ctx params File home infra prefect server lib python3 8 site packages click core py line 754 in invoke return callback args kwargs File home infra prefect server lib python3 8 site packages prefect cli agent py line 182 in start start agent LocalAgent import paths=list import paths kwargs File home infra prefect server lib python3 8 site packages prefect cli agent py line 140 in start agent agent start File home infra prefect server lib python3 8 site packages prefect agent agent py line 189 in start self setup api connection File home infra prefect server lib python3 8 site packages prefect agent agent py line 910 in setup api connection self client attach headers X PREFECT AGENT ID self register agent File home infra prefect server lib python3 8 site packages prefect agent agent py line 854 in register agent agent id = self client register agent File home infra prefect server lib python3 8 site packages prefect client client py line 2099 in register agent result = self graphql File home infra prefect server lib python3 8 site packages prefect client client py line 549 in graphql result = File home infra prefect server lib python3 8 site packages prefect client client py line 452 in post response = self request File home infra prefect server lib python3 8 site packages prefect client client py line 739 in request response = self send request File home infra prefect server lib python3 8 site packages prefect client client py line 603 in send request response = File home infra prefect server lib python3 8 site packages requests sessions py line 590 in post return self request POST url data=data json=json kwargs File home infra prefect server lib python3 8 site packages requests sessions py line 542 in request resp = self send prep send kwargs File home infra prefect server lib python3 8 site packages requests sessions py line 655 in send r = adapter send request kwargs File home infra prefect server lib python3 8 site packages requests adapters py line 529 in send raise ReadTimeout e request=request requests exceptions ReadTimeout HTTPConnectionPool host= 10 40 1 43 port=4200 Read timed out read timeout=15 Agent Stopped Wed Jul 6 21 15 44 IST 2022 infra ts mum1 dsr32 ~ gt ```

Kevin Kho · Answer

That makes sense since it s saying it can t connect to the API It s the apollo pod that needs to be looked at

Muddassir Shaikh · Answer

yup

Kevin Kho · Answer

Do you potentially have some flow still running or some agent still running and they are trying to update their state to that API

Muddassir Shaikh · Answer

yes i had 5 6 agents connected and running flows before server crashed

Kevin Kho · Answer

i am not 100 sure but what this looks like is you had a flow that crashed the apollo pod because it was sending a lot of logs or it was sending a huge result The server went down and you restart it but those Flows are still trying to update the result causing it to crash again So I think you may need to stop those first so apollo can restart