I don’t have any errors in my `apollo` pod logs, but I see this in the Agent logs: ```urllib3.except...

Wilson Bilkovich

08/27/2021, 1:19 PM

I don’t have any errors in my

apollo

pod logs, but I see this in the Agent logs:

Copy code

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='prefect-server-initial-apollo.prefect', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))

(prefect is my namespace, prefect-server-initial-apollo is the name of the service)

nicholas

08/27/2021, 3:32 PM

Hi @Wilson Bilkovich - if you ping the apollo pod directly do you get a response?

Wilson Bilkovich

08/27/2021, 3:53 PM

Let me check; thanks for the reply.

Wilson Bilkovich

08/27/2021, 3:55 PM

Copy code

% telnet prefect-server-initial-apollo 4200
Trying 127.1.27.7...
Connected to prefect-server-initial-apollo.
Escape character is '^]'.
GET /graphql

HTTP/1.1 400 Bad Request
X-Powered-By: Express
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Content-Length: 18
ETag: W/"12-7JEJwpG8g89ii7CR/6hhfN27Q+k"
Date: Fri, 27 Aug 2021 15:54:51 GMT
Connection: close

GET query missing.Connection closed by foreign host.

nicholas

08/27/2021, 4:06 PM

And this agent is the kubernetes agent started with the

--set agent.enabled=true

flag?

Wilson Bilkovich

08/27/2021, 4:39 PM

It is, yeah; using the default tenant also created by the Helm chart

nicholas

08/27/2021, 4:52 PM

Are you using any networking policies in your cluster?

nicholas

08/27/2021, 4:52 PM

That might be blocking requests?

Wilson Bilkovich

08/27/2021, 5:15 PM

I haven’t added anything, let me double-check if there are any. This is Amazon EKS and it does a few things by default.

Wilson Bilkovich

08/27/2021, 5:18 PM

Nope, no network policies. Also, I can connect from the agent to apollo when I exec into it:

Copy code

root@prefect-server-initial-agent-5f76b4fd69-vn74p:/# telnet prefect-server-initial-apollo 4200
Trying 172.20.65.234...
Connected to prefect-server-initial-apollo.prefect.svc.cluster.local.
Escape character is '^]'.
GET /graphql?query=foo

HTTP/1.1 400 Bad Request
X-Powered-By: Express
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=utf-8
Content-Length: 146
ETag: W/"92-sfDAzDmlWzD691zaMG6MP1fVCE8"
Date: Fri, 27 Aug 2021 17:17:41 GMT
Connection: close

{"errors":[{"message":"Syntax Error: Unexpected Name \"foo\"","locations":[{"line":1,"column":1}],"extensions":{"code":"GRAPHQL_PARSE_FAILED"}}]}
Connection closed by foreign host.

nicholas

08/27/2021, 6:03 PM

Hm, I'm not entirely sure how to proceed then, if you can query apollo from the rest of the cluster (i'm assuming the UI works) and you can query apollo from the agent container as well I'm at a bit of a loss. Can you start the agent manually ?

Wilson Bilkovich

08/27/2021, 6:20 PM

Hmm, maybe I am just chasing a ghost now, because there’s nothing unhappy in the pod logs for the current instance of the Agent.

nicholas

08/27/2021, 6:22 PM

Ah so it's working now? Maybe a network blip?

Wilson Bilkovich

08/27/2021, 6:25 PM

I’m trying to prove that to myself via Kibana, which I am growing to dislike..

🤔 1

Wilson Bilkovich

08/28/2021, 9:04 PM

Apparently these logs are from some earlier incarnation of my cluster, because I can’t reproduce it. Thanks for the help, I’ll just assume this was something silly I did in Kubernetes.

✅ 1

nicholas

08/30/2021, 7:01 AM

Glad you got it resolved!

50 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.