https://prefect.io logo
Channels
Powered by
m

Manuel Mourato

05/22/2020, 1:51 PM
Hello again guys Question: if I run a docker agent like bellow, and run a flow, it works with no issue.
Copy code
prefect agent start docker -l docker-execs-test
If I try to set the network to host however, like bellow, the flow is permanently in "Submitted" state.
Copy code
prefect agent start docker -l docker-execs-test --network host
Any ideas on why this might be?
j

josh

05/22/2020, 1:55 PM
Hmm 🤔 let me take a look because I am able to reproduce!
🙏 1
Which OS are you on?
The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.
I’m on a Mac and I think this is why it didn’t work for me
m

Manuel Mourato

05/22/2020, 1:58 PM
Linux, Ubuntu 18.04
😢
j

josh

05/22/2020, 1:58 PM
Okay interesting 🤔 would you mind also running it with 
--show-flow-logs
and sometime after the container eventually exits it should spit out an error message back to the agent logs
m

Manuel Mourato

05/22/2020, 2:01 PM
message has been deleted
message has been deleted
j

josh

05/22/2020, 2:10 PM
Is there a reason why you are trying to attach the host network to the container if it works without setting the network?
There already is some host resolution to allow for such behavior:
Copy code
if sys.platform.startswith("linux") and self.docker_interface:
            docker_internal_ip = get_docker_ip()
            host_config.update(extra_hosts={"host.docker.internal": docker_internal_ip})
m

Manuel Mourato

05/22/2020, 2:35 PM
A part of my flow involves connecting to a service that runs locally on port 1099, outside containers. Usually what I do in order to connect to a local port from a container, is setting the network to host and set the IP to 127.0.0.1 But this does not work in this case...
j

josh

05/22/2020, 3:04 PM
Just wondering, if instead of attempting to talk to 
127.0.0.1
you try 
host.docker.internal
(without attaching the network host) does it still get an error?
m

Manuel Mourato

05/22/2020, 3:40 PM
If I do that it seems to default to 127.0.0.1
Copy code
'Could not connect to Application: JVM exception occurred: Connection refused to host: 127.0.0.1
j

josh

05/22/2020, 3:43 PM
Ah okay. Do you mind opening an issue so we can investigate?
m

Manuel Mourato

05/22/2020, 4:57 PM
Will do. Something else you might find interesting, is that if I start the docker agent inside any network, be it the prefect-server network or any other, like so:
Copy code
prefect agent start docker -l docker-execs-test --show-flow-logs --network test-network
OR
prefect agent start docker -l docker-execs-test --show-flow-logs --network prefect-server
OR
prefect agent start docker -l docker-execs-test --show-flow-logs --network host
The flows get stuck in submitted state with this error:
Copy code
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fb645a37cf8>, 'Connection to host.docker.internal timed out. (connect timeout=30)')
j

josh

05/22/2020, 5:00 PM
Hmm I’m thinking that it’s because the flow is starting and attempting to reach out to the API to say that it is starting but the api DNS isn’t mapped to inside that network
m

Manuel Mourato

05/22/2020, 5:13 PM
Do you believe it might be a DNS issue? It's interesting that it only works when the network is set to None
j

josh

05/22/2020, 5:17 PM
Powered by