Thread
#prefect-community
    d

    Darren Fleetwood

    2 years ago
    Hi all, first of all thanks for what looks like a great product so far! I'm trying to use the core server to submit jobs to the docker agent. I can register flows ok and when I run them they are picked up by the agent. But they sit at 'submitted'. Looking at the logs for the docker containers I get this:
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7739090a90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    . I'm guessing it means the container can't find the graphql api? I'm running under Ubuntu. I'm ssh tunnelling into a jupyter notebook on an Azure VM and running from there - could that be making a difference? thanks!!
    As a quick update to this - I can run flows registered with the cloud with a docker agent with this machine fine, so I'm assuming the problem isn't with the docker setup. Will update if/when I figure this out.
    j

    josh

    2 years ago
    Hey @Darren Fleetwood any idea what you did differently in order to get it to work?
    d

    Darren Fleetwood

    2 years ago
    Hi Josh. I haven't managed to fix it - I'm a bit confused. I'm going to try rebuilding from a blank VM when I get a chance
    j

    josh

    2 years ago
    Which version of Docker do you happen to be using on your VM? In the past there have been issues with the
    host.docker.internal
    when using Linux (has been supported on Mac and Windows for a while) and I’m thinking that now we need some extra host configuration on the flow run’s container in order to reach back out to the host daemon
    d

    Darren Fleetwood

    2 years ago
    Ahh ok, it same flows are fine on my Windows PC so that sounds possible. Docker version 3.0.8, build 2355349d Docker-compose: docker-compose version 1.23.1, build b02f1306
    j

    josh

    2 years ago
    I’m attempting to get the Docker API’s
    extra_hosts
    config to work but am having trouble so no luck yet on my end
    d

    Darren Fleetwood

    2 years ago
    I've tried updating docker to Docker version 19.03.8, build afacb8b7f0 (not sure how I was running such an old version before) but same problem I'm afraid
    n

    Nate Atkins

    2 years ago
    Did we ever get an solution to this on Ubuntu? I'm running:
    Docker version 19.03.8, build afacb8b7f0
    docker-compose version 1.25.4, build 8d51620a
    and get the same ConnectionError in the docker logs with --show-flow-logs
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
        for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
      File "/usr/local/lib/python3.7/socket.py", line 752, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -2] Name or service not known
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.7/http/client.py", line 1252, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1298, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1247, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.7/http/client.py", line 966, in send
        self.connect()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
        conn = self._new_conn()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f71301eba90>: Failed to establish a new connection: [Errno -2] Name or service not known
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
        timeout=timeout
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f71301eba90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 53, in cloud_flow
        result = client.graphql(query)
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 220, in graphql
        token=token,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 180, in post
        token=token,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 303, in _request
        response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=30)
      File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 578, in post
        return self.request('POST', url, data=data, json=json, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
        raise ConnectionError(e, request=request)
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f71301eba90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    j

    josh

    2 years ago
    @Nate Atkins no resolution yet. Would either of you be willing to open an issue with this information so we can better track a fix?
    Brad

    Brad

    2 years ago
    Yep just a +1 for this issue. I can run flows with the Docker agent on my macbook without issue but the same flow fails on a server running Ubuntu
    n

    Nate Atkins

    2 years ago
    j

    josh

    2 years ago
    Thanks Nate!
    Brad

    Brad

    2 years ago
    @josh I’ve started a WIP PR here https://github.com/limx0/prefect/pull/2. I’m going to test it out now
    (More just a placeholder until the docker merge gets release, but someone else might find it useful..)
    j

    josh

    2 years ago
    Awesome @Brad! Let me know how it works. We should def put this in until Docker finally releases the feature (and keep in past that because people run older docker versions)
    Brad

    Brad

    2 years ago
    Okay - that’s now broken my UI, so no go at this stage
    Actually - I take that back it does appear to work
    j

    josh

    2 years ago
    Awesome! Taking a look now