https://prefect.io logo
Join Slack
Powered by
Hello again guys Question: if I run a docker agent...
# ask-community
m
Hello again guys Question: if I run a docker agent like bellow, and run a flow, it works with no issue.
Copy code
prefect agent start docker -l docker-execs-test
If I try to set the network to host however, like bellow, the flow is permanently in "Submitted" state.
Copy code
prefect agent start docker -l docker-execs-test --network host
Any ideas on why this might be?
j
Hmm 🤔 let me take a look because I am able to reproduce!
🙏 1
Which OS are you on?
The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.
I’m on a Mac and I think this is why it didn’t work for me
m
Linux, Ubuntu 18.04
😢
j
Okay interesting 🤔 would you mind also running it with 
--show-flow-logs
and sometime after the container eventually exits it should spit out an error message back to the agent logs
m
message has been deleted
message has been deleted
j
Is there a reason why you are trying to attach the host network to the container if it works without setting the network?
There already is some host resolution to allow for such behavior:
Copy code
if sys.platform.startswith("linux") and self.docker_interface:
            docker_internal_ip = get_docker_ip()
            host_config.update(extra_hosts={"host.docker.internal": docker_internal_ip})
m
A part of my flow involves connecting to a service that runs locally on port 1099, outside containers. Usually what I do in order to connect to a local port from a container, is setting the network to host and set the IP to 127.0.0.1 But this does not work in this case...
j
Just wondering, if instead of attempting to talk to 
127.0.0.1
you try 
host.docker.internal
(without attaching the network host) does it still get an error?
m
If I do that it seems to default to 127.0.0.1
Copy code
'Could not connect to Application: JVM exception occurred: Connection refused to host: 127.0.0.1
j
Ah okay. Do you mind opening an issue so we can investigate?
m
Will do. Something else you might find interesting, is that if I start the docker agent inside any network, be it the prefect-server network or any other, like so:
Copy code
prefect agent start docker -l docker-execs-test --show-flow-logs --network test-network
OR
prefect agent start docker -l docker-execs-test --show-flow-logs --network prefect-server
OR
prefect agent start docker -l docker-execs-test --show-flow-logs --network host
The flows get stuck in submitted state with this error:
Copy code
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fb645a37cf8>, 'Connection to host.docker.internal timed out. (connect timeout=30)')
j
Hmm I’m thinking that it’s because the flow is starting and attempting to reach out to the API to say that it is starting but the api DNS isn’t mapped to inside that network
m
Do you believe it might be a DNS issue? It's interesting that it only works when the network is set to None
j
Open in Slack
PreviousNext
Powered by