https://prefect.io logo
Title
m

Marwan Sarieddine

05/19/2020, 5:29 PM
Hi everyone - new to prefect, I got a the server running on prefect cloud, along with a kuberenetes agent that is polling from the server - I registered a flow that shows up on the server and I am trying to run it but I am getting an error. I am following the [Docker Sidecar on Kubernetes deployment recipe](https://docs.prefect.io/orchestration/recipes/k8s_docker_sidecar.html) I did not do any code changes and used the flow script as is - I am running prefect version
0.10.7
and I am getting the below error in the logs
19 May 2020,01:03:00 	agent	INFO	Submitted for execution: Job prefect-job-d7e49926
19 May 2020,01:03:26 	prefect.CloudFlowRunner	INFO	Beginning Flow run for 'Run a Prefect Flow in Docker'
19 May 2020,01:03:27 	prefect.CloudFlowRunner	INFO	Starting flow run.
19 May 2020,01:03:27 	prefect.CloudTaskRunner	INFO	Task 'PullImage': Starting task run...
19 May 2020,01:03:27 	prefect.CloudTaskRunner	ERROR	Unexpected error: DockerException("Error while fetching server API version: HTTPConnectionPool(host='localhost', port=2375): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8768ed7c10>: Failed to establish a new connection: [Errno 111] Connection refused'))")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
It seems that
localhost:2375
(docker daemon address is not reachable for some reason) - anyone faced this issue before ?
z

Zachary Hughes

05/19/2020, 5:39 PM
Hi @Marwan Sarieddine, asking the team now! Will let you know as soon as we've got an answer for you.
m

Marwan Sarieddine

05/19/2020, 5:47 PM
@Zachary Hughes thank you for the prompt response
j

josh

05/19/2020, 5:52 PM
I am able to reproduce the error, wonder if something to do with the docker API has changed 🤔 looking into it
m

Marwan Sarieddine

05/19/2020, 6:02 PM
@josh thanks for taking the time to look into it
j

josh

05/19/2020, 6:51 PM
Alright, I was able to get it working. Looks like they have changed how TLS is handled in DinD images now. It now defaults to being enabled which is not what we want for this recipe (would require you to provide a cert) and it causes the API to server on port
2376
instead of the
2375
that we want! All we need to do is disable serving the TLS encrypted API by adding the following env var to the
dind-daemon
container:
env:
    - name: DOCKER_TLS_CERTDIR
      value: ""
I’ll make the adjustment in the recipe! Thanks for pointing this out 🙂
:upvote: 1
m

Marwan Sarieddine

05/19/2020, 9:24 PM
@josh - thank you - it works now ! Also thanks for explaining the details concerning TLS handling