https://prefect.io logo
Powered by
Title
m

Manuel Mourato

05/22/2020, 1:51 PM
Hello again guys Question: if I run a docker agent like bellow, and run a flow, it works with no issue. 
prefect agent start docker -l docker-execs-test
If I try to set the network to host however, like bellow, the flow is permanently in "Submitted" state. 
prefect agent start docker -l docker-execs-test --network host
Any ideas on why this might be?
j

josh

05/22/2020, 1:55 PM
Hmm 🤔 let me take a look because I am able to reproduce!
🙏 1
Which OS are you on?
The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.
I’m on a Mac and I think this is why it didn’t work for me
m

Manuel Mourato

05/22/2020, 1:58 PM
Linux, Ubuntu 18.04
😢
j

josh

05/22/2020, 1:58 PM
Okay interesting 🤔 would you mind also running it with 
--show-flow-logs
and sometime after the container eventually exits it should spit out an error message back to the agent logs
m

Manuel Mourato

05/22/2020, 2:01 PM
message has been deleted
message has been deleted
j

josh

05/22/2020, 2:10 PM
Is there a reason why you are trying to attach the host network to the container if it works without setting the network?
There already is some host resolution to allow for such behavior: 
if sys.platform.startswith("linux") and self.docker_interface:
            docker_internal_ip = get_docker_ip()
            host_config.update(extra_hosts={"host.docker.internal": docker_internal_ip})
m

Manuel Mourato

05/22/2020, 2:35 PM
A part of my flow involves connecting to a service that runs locally on port 1099, outside containers. Usually what I do in order to connect to a local port from a container, is setting the network to host and set the IP to 127.0.0.1 But this does not work in this case...
j

josh

05/22/2020, 3:04 PM
Just wondering, if instead of attempting to talk to 
127.0.0.1
you try 
host.docker.internal
(without attaching the network host) does it still get an error?
m

Manuel Mourato

05/22/2020, 3:40 PM
If I do that it seems to default to 127.0.0.1 
'Could not connect to Application: JVM exception occurred: Connection refused to host: 127.0.0.1
j

josh

05/22/2020, 3:43 PM
Ah okay. Do you mind opening an issue so we can investigate?
m

Manuel Mourato

05/22/2020, 4:57 PM
Will do. Something else you might find interesting, is that if I start the docker agent inside any network, be it the prefect-server network or any other, like so: 
prefect agent start docker -l docker-execs-test --show-flow-logs --network test-network
OR
prefect agent start docker -l docker-execs-test --show-flow-logs --network prefect-server
OR
prefect agent start docker -l docker-execs-test --show-flow-logs --network host
The flows get stuck in submitted state with this error: 
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fb645a37cf8>, 'Connection to host.docker.internal timed out. (connect timeout=30)')
j

josh

05/22/2020, 5:00 PM
Hmm I’m thinking that it’s because the flow is starting and attempting to reach out to the API to say that it is starting but the api DNS isn’t mapped to inside that network
m

Manuel Mourato

05/22/2020, 5:13 PM
Here is the issue @josh https://github.com/PrefectHQ/prefect/issues/2638
Do you believe it might be a DNS issue? It's interesting that it only works when the network is set to None
j

josh

05/22/2020, 5:17 PM
I’m thinking it could be because that’s what I was seeing when I was looking around https://github.com/moby/moby/issues/26330#issuecomment-246304015 https://stackoverflow.com/questions/44761246/temporary-failure-in-name-resolution-errno-3-with-docker
#prefect-communityJoin Slack
Powered by