ash
06/04/2021, 9:04 AMash
06/04/2021, 9:05 AMError: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1fc4583310>: Failed to establish a new connection: [Errno 111] Connection refused'))
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.25.0.3', 3000)", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n request, auth=auth, timeout=timeout, history=history\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n response = await self.send_single_request(request, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n timeout=timeout.as_dict(),\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n method, url, headers=headers, stream=stream, timeout=timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n self.socket = await self._open_socket(timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n hostname, port, ssl_context, timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n stream_reader=stream_reader, stream_writer=stream_writer\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.25.0.3', 3000)"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.25.0.3', 3000)", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n request, auth=auth, timeout=timeout, history=history\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n response = await self.send_single_request(request, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n timeout=timeout.as_dict(),\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n method, url, headers=headers, stream=stream, timeout=timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n self.socket = await self._open_socket(timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n hostname, port, ssl_context, timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n stream_reader=stream_reader, stream_writer=stream_writer\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.25.0.3', 3000)"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.25.0.3', 3000)", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n request, auth=auth, timeout=timeout, history=history\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n response = await self.send_single_request(request, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n timeout=timeout.as_dict(),\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n method, url, headers=headers, stream=stream, timeout=timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n self.socket = await self._open_socket(timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n hostname, port, ssl_context, timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n stream_reader=stream_reader, stream_writer=stream_writer\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.25.0.3', 3000)"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')"}
nicholas
docker kill $(docker ps | awk 'NR > 1 {print $1}')
to tear down all containers in one go, though NOTE: this will kill non-prefect containers as well). Then try starting prefect server again and let me know if anything changesash
06/04/2021, 2:06 PMash
06/04/2021, 2:06 PMctrl+c
after prefect server start stops all the containersash
06/04/2021, 2:07 PMnicholas
ash
06/05/2021, 9:12 PMdocker ps -a
, so all the containers are actually those starting by prefect server start
therefore i am sure nothing is running on port 3000. Please look at this image ,graphql is not working properly when i inspected on chrome browserash
06/05/2021, 9:13 PMash
06/05/2021, 9:43 PMnicholas
ash
06/08/2021, 10:40 AMash
06/08/2021, 11:31 AM{
"severity": "ERROR",
"name": "prefect-server.HasuraClient",
"message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.19.0.3', 3000)",
"exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n request, auth=auth, timeout=timeout, history=history\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n response = await self.send_single_request(request, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n timeout=timeout.as_dict(),\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n method, url, headers=headers, stream=stream, timeout=timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n self.socket = await self._open_socket(timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n hostname, port, ssl_context, timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n stream_reader=stream_reader, stream_writer=stream_writer\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.19.0.3', 3000)"
}
ash
06/08/2021, 11:32 AMash
06/08/2021, 2:07 PM1.) when graphQl runs : During alembic migration, Could not upgrade the database! error occurs due to (psycopg2.errors.UniqueViolation).
2.) Postgres runs
3.) Hasura runs and gives error
error logs are this
hasura_1 | {"type":"pg-client","timestamp":"2021-06-08T11:39:32.717+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1 | {"type":"pg-client","timestamp":"2021-06-08T11:39:32.717+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1 | {"type":"startup","timestamp":"2021-06-08T11:39:32.717+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.21.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.21.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
4.) Towel runs and since it is dependent on Hasura as per the architecture , it also fails.
nicholas
ash
06/08/2021, 8:54 PMprefect server start --postgres-url postgres://<username>:<password>@hostname:<port>/<dbname>
ash
06/08/2021, 8:54 PMnicholas
ash
06/08/2021, 9:04 PMgraphql_1 | Could not upgrade the database!
graphql_1 | Error: (psycopg2.OperationalError) could not connect to server: No route to host
graphql_1 | Is the server running on host "10.0.1.23" and accepting
graphql_1 | TCP/IP connections on port 5432?
nicholas
ash
06/08/2021, 9:14 PMash
06/08/2021, 9:14 PMpsql -h host_name -U postgres
on terminalnicholas
ash
06/08/2021, 9:20 PMash
06/08/2021, 9:31 PMash
06/08/2021, 9:33 PMnicholas
nicholas
ash
06/08/2021, 9:43 PMnicholas
ash
06/08/2021, 9:44 PMash
06/08/2021, 9:44 PMash
06/08/2021, 9:44 PMnicholas
ash
06/08/2021, 9:46 PMnicholas