https://prefect.io logo
Join Slack
Powered by
Hi my prefect dashboard is working but data is not...
# ask-community
a
Hi my prefect dashboard is working but data is not coming. It keeps loading flows. This is due to some error that came up when prefect server was launched but i am unable to debug it.
error when prefect server is starting
Copy code
Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1fc4583310>: Failed to establish a new connection: [Errno 111] Connection refused'))
towel_1     | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.25.0.3', 3000)", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n    timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n    request, auth=auth, timeout=timeout, history=history\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n    response = await self.send_single_request(request, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n    timeout=timeout.as_dict(),\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n    method, url, headers=headers, stream=stream, timeout=timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n    self.socket = await self._open_socket(timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n    hostname, port, ssl_context, timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n    return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n    stream_reader=stream_reader, stream_writer=stream_writer\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.25.0.3', 3000)"}
towel_1     | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')"}
towel_1     | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.25.0.3', 3000)", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n    timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n    request, auth=auth, timeout=timeout, history=history\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n    response = await self.send_single_request(request, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n    timeout=timeout.as_dict(),\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n    method, url, headers=headers, stream=stream, timeout=timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n    self.socket = await self._open_socket(timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n    hostname, port, ssl_context, timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n    return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n    stream_reader=stream_reader, stream_writer=stream_writer\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.25.0.3', 3000)"}
towel_1     | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')"}
towel_1     | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.25.0.3', 3000)", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n    timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n    request, auth=auth, timeout=timeout, history=history\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n    response = await self.send_single_request(request, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n    timeout=timeout.as_dict(),\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n    method, url, headers=headers, stream=stream, timeout=timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n    self.socket = await self._open_socket(timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n    hostname, port, ssl_context, timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n    return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n    stream_reader=stream_reader, stream_writer=stream_writer\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.25.0.3', 3000)"}
towel_1     | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')"}
n
Hi @ash - it looks like Prefect Server isn't running properly; can you shut down all Prefect-related docker containers (you can use 
docker kill $(docker ps | awk 'NR > 1 {print $1}')
to tear down all containers in one go, though NOTE: this will kill non-prefect containers as well). Then try starting prefect server again and let me know if anything changes
a
i tried this
i was using prefer backend server and 
ctrl+c
after prefect server start stops all the containers
i tried re running it but the same issue prevailed
n
Hm got it - it looks like Hasura is unable to start; can you confirm that port 3000 isn't being used elsewhere?
a
well i am not sure because once i close the server nothing is running on when i view using 
docker ps -a
, so all the containers are actually those starting by 
prefect server start
therefore i am sure nothing is running on port 3000. Please look at this image ,graphql is not working properly when i inspected on chrome browser
n
Right - if Hasura doesn't start properly, metadata won't be applied and the Apollo (GraphQL) server won't start. Which version of Prefect are you on?
a
i am using 0.14.19
@nicholas i just realised that this error have something related to python3.7
Copy code
{
  "severity": "ERROR",
  "name": "prefect-server.HasuraClient",
  "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.19.0.3', 3000)",
  "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n    timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n    request, auth=auth, timeout=timeout, history=history\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n    response = await self.send_single_request(request, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n    timeout=timeout.as_dict(),\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n    method, url, headers=headers, stream=stream, timeout=timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n    self.socket = await self._open_socket(timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n    hostname, port, ssl_context, timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n    return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n    stream_reader=stream_reader, stream_writer=stream_writer\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.19.0.3', 3000)"
}
but my virtualenv is using python3.8 , can there be some issue for towel and hasura api when using prefect == 0.14.19
So far what we know
Copy code
1.) when graphQl runs : During alembic migration, Could not upgrade the database! error occurs due to (psycopg2.errors.UniqueViolation).  
2.) Postgres runs
3.) Hasura runs  and gives error 

error logs are  this 
hasura_1    | {"type":"pg-client","timestamp":"2021-06-08T11:39:32.717+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1    | {"type":"pg-client","timestamp":"2021-06-08T11:39:32.717+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1    | {"type":"startup","timestamp":"2021-06-08T11:39:32.717+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.21.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
hasura_1    | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.21.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}


4.) Towel runs and since it is dependent on Hasura as per the architecture , it also fails.
n
That could be the issue - if the database upgrade fails, the rest of the services won't be able to start. Do you have another postgres database running locally maybe?
a
Hi @nicholas, yes thats exactly what i have thought , just now deployed a postgres on port = 5432 on container and now what i am trying here is using external postgres for a single node machine
Copy code
prefect server start --postgres-url postgres://<username>:<password>@hostname:<port>/<dbname>
is that what you are suggesting?
n
That could definitely work, I was thinking there was an existing database already where an overlap might be occurring - let me know how it goes with your standalone PG instance
a
this time received a the same error with different message
Copy code
graphql_1  | Could not upgrade the database!
graphql_1  | Error: (psycopg2.OperationalError) could not connect to server: No route to host
graphql_1  | 	Is the server running on host "10.0.1.23" and accepting
graphql_1  | 	TCP/IP connections on port 5432?
n
That sounds like there's an issue connecting to the db that you spun up - can you confirm you can connect to it at that address and port independently? And are you running all of this in a container or a remote machine or is this all local?
a
I am using an amazon ec2 server using ubuntu 18.04 and my postgres is running on container
Also i am able to login in psql using 
psql -h host_name -U postgres
on terminal
n
Got it - something's up with how networking is being handled on the machine and unfortunately I don't know how much more help I can be on this piece. There are a few blog posts that can help get Server up and running on a remote machine that might be worth checking out
a
Ok , can you share such blogs , that would be a great favour.Also do you think running postgres directly on machine rather than container will help me getting saved from this network related issue due to containers
Hey Nicholas, i used another db for connecting with prefect and its working now
But dashboard is still coming like this
n
Ah perfect - for reference here's a blog on deploying Server using GCP using traditional CLI tools and another one using Terraform, both should be pretty transferrable to AWS;
@ash if you're deploying to a remote machine, you'll need to set the endpoint in your config before starting Server. To test the connection you can visit the "Getting Started" screen and put your apollo endpoint there. It'll be something like "http://your_host:4200/graphql"
a
OHHHH goshhhhhhh😃 its worked
n
perfect!!! gj 🙂
🙏 1
a
all i needed was to tunnel my 4200 port to my local port so that i could get access to 4200 port of remote machine
Thanks a ton nicholas, that was really amazing
😇😃
n
That's 1 way to do it! Nicely done 👍
a
I will surely go through through the blog you shared to get the understanding of best practice and will get back to you in case of doubts , thanks again.
n
Makes sense to me! Good luck 🙂
16 Views
Open in Slack
PreviousNext
Powered by