https://prefect.io logo
Powered by
Title
d

Darren Fleetwood

04/05/2020, 3:49 PM
Hi all, first of all thanks for what looks like a great product so far! I'm trying to use the core server to submit jobs to the docker agent. I can register flows ok and when I run them they are picked up by the agent. But they sit at 'submitted'. Looking at the logs for the docker containers I get this: 
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7739090a90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
. I'm guessing it means the container can't find the graphql api? I'm running under Ubuntu. I'm ssh tunnelling into a jupyter notebook on an Azure VM and running from there - could that be making a difference? thanks!!
As a quick update to this - I can run flows registered with the cloud with a docker agent with this machine fine, so I'm assuming the problem isn't with the docker setup. Will update if/when I figure this out.
j

josh

04/06/2020, 12:12 PM
Hey @Darren Fleetwood any idea what you did differently in order to get it to work?
d

Darren Fleetwood

04/06/2020, 12:25 PM
Hi Josh. I haven't managed to fix it - I'm a bit confused. I'm going to try rebuilding from a blank VM when I get a chance
j

josh

04/06/2020, 12:31 PM
Which version of Docker do you happen to be using on your VM? In the past there have been issues with the 
host.docker.internal
when using Linux (has been supported on Mac and Windows for a while) and I’m thinking that now we need some extra host configuration on the flow run’s container in order to reach back out to the host daemon
d

Darren Fleetwood

04/06/2020, 12:38 PM
Ahh ok, it same flows are fine on my Windows PC so that sounds possible. Docker version 3.0.8, build 2355349d Docker-compose: docker-compose version 1.23.1, build b02f1306
j

josh

04/06/2020, 1:14 PM
I’m attempting to get the Docker API’s 
extra_hosts
config to work but am having trouble so no luck yet on my end
d

Darren Fleetwood

04/06/2020, 1:15 PM
I've tried updating docker to Docker version 19.03.8, build afacb8b7f0 (not sure how I was running such an old version before) but same problem I'm afraid
n

Nate Atkins

04/12/2020, 11:46 PM
Did we ever get an solution to this on Ubuntu? I'm running: 
Docker version 19.03.8, build afacb8b7f0
docker-compose version 1.25.4, build 8d51620a
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f71301eba90>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f71301eba90>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 53, in cloud_flow
    result = client.graphql(query)
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 220, in graphql
    token=token,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 180, in post
    token=token,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 303, in _request
    response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=30)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f71301eba90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
and get the same ConnectionError in the docker logs with --show-flow-logs
j

josh

04/13/2020, 11:36 AM
@Nate Atkins no resolution yet. Would either of you be willing to open an issue with this information so we can better track a fix?
b

Brad

04/13/2020, 11:55 AM
Yep just a +1 for this issue. I can run flows with the Docker agent on my macbook without issue but the same flow fails on a server running Ubuntu
n

Nate Atkins

04/13/2020, 1:59 PM
Opened as https://github.com/PrefectHQ/prefect/issues/2324
j

josh

04/13/2020, 1:59 PM
Thanks Nate!
b

Brad

04/13/2020, 11:07 PM
@josh I’ve started a WIP PR here https://github.com/limx0/prefect/pull/2. I’m going to test it out now
(More just a placeholder until the docker merge gets release, but someone else might find it useful..)
j

josh

04/13/2020, 11:08 PM
Awesome @Brad! Let me know how it works. We should def put this in until Docker finally releases the feature (and keep in past that because people run older docker versions)
b

Brad

04/13/2020, 11:20 PM
Okay - that’s now broken my UI, so no go at this stage
Actually - I take that back it does appear to work
I’ve opened https://github.com/PrefectHQ/prefect/pull/2328
j

josh

04/14/2020, 11:20 AM
Awesome! Taking a look now
#prefect-communityJoin Slack
Powered by