https://prefect.io logo
Join Slack
Powered by
Hey, We use CircleCI to test and register our flo...
# ask-community
k
Hey, We use CircleCI to test and register our flows (Prefect v0.14.12) which we use Docker storage for. In our recent attempts to register a new flow we have been hit with this error and failing to build:
Copy code
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
(full trace in this thread) From reading around is looks like a potential docker issue, an inability to connect to a socket?! Has anyone else been hit with this?
Copy code
Removing intermediate container ffc26cbf530e
 ---> 720f6f321752
Step 6/10 : RUN mkdir -p /opt/prefect/
 ---> Running in 679b82f2fde9
Removing intermediate container 679b82f2fde9
 ---> 9d78552ba1d9
Step 7/10 : COPY bordereau.flow /opt/prefect/flows/bordereau.prefect
 ---> f13939593f26
Step 8/10 : COPY healthcheck.py /opt/prefect/healthcheck.py
 ---> 04b9344f7e76
Step 9/10 : COPY tasks /opt/prefect/flows/tasks
 ---> 50ac16185624
Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
 ---> Running in f78c279cb415
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 152, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
  File "/opt/prefect/flows/tasks/commands/extract_and_load.py", line 9, in <module>
    from flows.tasks.configuration.config import (DEFAULT_DESTINATION_SCHEMA,
  File "/opt/prefect/flows/tasks/configuration/config.py", line 22, in <module>
    base_url=docker.from_env().api.base_url,
  File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 101, in from_env
    **kwargs_from_env(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
z
Hi! This is an error where the docker client cannot find the docker engine server. The "file" that's missing is a socket.
k
@Zanie thanks for the quick reply!
z
There's a workaround there 🙂
k
omg thank you @Zanie, I will try that work around now!
@Zanie just so I fully understand the fix -- currently, I am using the following:
Copy code
default_client = docker.from_env()

dockerised_storage = Docker(
    base_url=default_client.api.base_url,
    tls_config=docker.TLSConfig(default_client.api.cert),
    registry_url=''
    python_dependencies=python_deps
)
The workaround checks whether the .from_env().api.base_url method is a remote host and then sets it to the variable 
default_client
. Is that not how I currently have it set up? Or have I been really stupid and mis-read the workaround?
z
What you're doing should work in CircleCI, I think. It does look like the same thing the workaround does (except the workaround will also work on your local machine)
I may have misread your traceback actually, on a second look this is failing during the healthcheck
k
Sorry, the above configuration is how I currently have things set up and they don't work in CircleCI. Sorry, I should have been clearer.
z
Ah so because you're calling 
docker.from_env()
in your flow script, when we try to run the healthcheck it is trying to contact docker from within the docker image it has just built and is failing because there's no docker in docker setup.
Can you try using my workaround instead which is local-compatible? It might still fail on the 
from_env
call but I'm not sure.
k
When should we be calling 
docker.from_env()
if not when registering a flow? I am not sure if I understand the workaround but this is what I am currently testing:
Copy code
def DockerCircleCI(**kwargs: Any) -> Any:
    import docker

    default_client = docker.from_env()

    # This will break local builds / is only needed for remote docker hosts
    if default_client.api.base_url != "<http+docker://localhost>":
        kwargs.setdefault("base_url", default_client.api.base_url)

    kwargs.setdefault("tls_config", docker.TLSConfig(default_client.api.cert))

    return Docker(**kwargs)


dockerised_storage = DockerCircleCI(
    registry_url="",
    python_dependencies=python_deps
)
I am then passing 
dockerised_storage
into the 
storage
parameter of 
Flow()
.
z
Yep, that is how you'd use it
The issue is that 
from_env()
is being called when registering the flow then again when the flow script is run within the built storage for a healthcheck.
(I think this is what is happening)
k
Thanks, whilst I am testing this do you know what introduced this bug? Was it some sub-dependency that has changed?
Afraid no luck, here is the traceback:
Copy code
Step 1/10 : FROM prefecthq/prefect:0.14.15-python3.6
 ---> cece2ab0a690
Step 2/10 : ENV PYTHONPATH='$PYTHONPATH:/opt/prefect/:****************************/flows/tasks'     PREFECT__USER_CONFIG_PATH='/opt/prefect/config.toml'
 ---> Using cache
 ---> 97d69ce7f159
Step 3/10 : RUN pip install pip --upgrade
 ---> Using cache
 ---> 3f1a0cea00fa
Step 4/10 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.14.15#egg=prefect[all_orchestration_extras]>
 ---> Using cache
 ---> 5b2e98a043c8
Step 5/10 : RUN pip install docker==4.4.1 requests==2.25.1 pandas==1.1.5 sqlalchemy==1.3.22 psycopg2-binary==2.8.6 pygit2==1.5.0 pact-python==1.3.0 awscli==1.19.46 gspread==3.7.0 oauth2client==4.1.3 wheel
 ---> Using cache
 ---> 720f6f321752
Step 6/10 : RUN mkdir -p /opt/prefect/
 ---> Using cache
 ---> 9d78552ba1d9
Step 7/10 : COPY bordereau.flow /opt/prefect/flows/bordereau.prefect
 ---> ea6f3b54fe04
Step 8/10 : COPY healthcheck.py /opt/prefect/healthcheck.py
 ---> 93747758af17
Step 9/10 : COPY tasks /opt/prefect/flows/tasks
 ---> 31dd22e4b1c1
Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
 ---> Running in 780c3747c0f6
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.excep
tions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 152, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
  File "/usr/local/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 562, in subimport
    __import__(name)
  File "/opt/prefect/flows/tasks/configuration/config.py", line 43, in <module>
    "PYTHONPATH": "$PYTHONPATH:/opt/prefect/:****************************/flows/tasks"
  File "/opt/prefect/flows/tasks/configuration/config.py", line 25, in DockerCircleCI
    default_client = docker.from_env()
  File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 101, in from_env
    **kwargs_from_env(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Removing intermediate container 780c3747c0f6
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'' returned a non-zero code: 1
****************************/.venv/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host '35.224.236.246'. Adding certificate verification is strongly advised. See: <https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings>
  InsecureRequestWarning,
Traceback (most recent call last):
  File "flows/flows.etl.bordereau.py", line 87, in <module>
    serialise.serialise(flow, "ExtractLoad", config.FLOW_LABEL)
  File "****************************/flows/tasks/configuration/serialise.py", line 12, in serialise
    add_default_labels=False
  File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1710, in register
    idempotency_key=idempotency_key,
  File "****************************/.venv/lib/python3.6/site-packages/prefect/client/client.py", line 797, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1485, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 303, in build
    self._build_image(push=push)
  File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 371, in _build_image
    "Your docker image failed to build!  Your flow might have "
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.

Exited with code exit status 1
z
Nothing should have changed for this on our end. Did you even switch Prefect versions?
k
No, nothing has changed from our perspective...
z
And your cloudpickle/dockerpy versions are the same?
The deserialization healthcheck is failing
k
We haven't been pinning those packages. They are being used by Prefect under the hood, right?
z
Yeah and we pin to an upper bound to stop breaking changes from getting in, but that's my best theory right now.
Can you do
Copy code
if not prefect.context.loading_flow:
    flow.storage = Docker(...)
else:
    flow.storage = None
We set a context variable so you can avoid doing things that are sensitive like this
k
Sure, I am just trying to resolve dependencies and pin cloudpickle down + add that conditional to the flow.
Copy code
prefect = { version = "0.14.12", extras = ["aws"] }
docker = "==4.4.1"
cloudpickle = "==1.5.0"
^ for reference.
@Zanie are you sure on the context method name? I am getting:
Copy code
AttributeError: 'Context' object has no attribute 'loading_flow'
Copy code
if not prefect.context.get("loading_flow", False):
        flow.storage = config.default_flow_storage
    else:
        flow.storage = None
^ I think that is how to pull it out.
z
Sorry yeah a 
get
is necessary.
k
@Michael Hadorn, no luck I am afraid. Still the same error on the healthcheck:
Copy code
Step 1/10 : FROM prefecthq/prefect:0.15.4-python3.6
 ---> 41e4970e0048
Step 2/10 : ENV PYTHONPATH='$PYTHONPATH:/opt/prefect/:****************************/flows/tasks'     PREFECT__USER_CONFIG_PATH='/opt/prefect/config.toml'
 ---> Using cache
 ---> c7b74b0f1a4f
Step 3/10 : RUN pip install pip --upgrade
 ---> Using cache
 ---> 28f093a33f7e
Step 4/10 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.15.4#egg=prefect[all_orchestration_extras]>
 ---> Using cache
 ---> 5fc1e26879a9
Step 5/10 : RUN pip install docker==4.4.1 requests==2.25.1 pandas==1.1.5 sqlalchemy==1.3.22 psycopg2-binary==2.8.6 pygit2==1.5.0 pact-python==1.3.0 awscli==1.20.33 gspread==3.7.0 oauth2client==4.1.3 cloudpickle==1.5.0 types-requests==2.25.6 types-pyyaml==5.4.10 wheel
 ---> Using cache
 ---> 33cbba1c8169
Step 6/10 : RUN mkdir -p /opt/prefect/
 ---> Using cache
 ---> 3c7332eead15
Step 7/10 : COPY bordereau.flow /opt/prefect/flows/bordereau.prefect
 ---> a42ba277e1dd
Step 8/10 : COPY healthcheck.py /opt/prefect/healthcheck.py
 ---> 823ad6db6f40
Step 9/10 : COPY tasks /opt/prefect/flows/tasks
 ---> ecf269287600
Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
 ---> Running in e75a88b55e15
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1291, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1337, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1286, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1046, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 984, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1291, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1337, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1286, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1046, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 984, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 152, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
  File "/opt/prefect/flows/tasks/commands/extract_and_load.py", line 9, in <module>
    from flows.tasks.configuration.config import (DEFAULT_DESTINATION_SCHEMA,
  File "/opt/prefect/flows/tasks/configuration/config.py", line 43, in <module>
    "PYTHONPATH": "$PYTHONPATH:/opt/prefect/:****************************/flows/tasks"
  File "/opt/prefect/flows/tasks/configuration/config.py", line 25, in DockerCircleCI
    default_client = docker.from_env()
  File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 101, in from_env
    **kwargs_from_env(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'' returned a non-zero code: 1
****************************/.venv/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host '34.72.24.230'. Adding certificate verification is strongly advised. See: <https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings>
  InsecureRequestWarning,
Traceback (most recent call last):
  File "flows/flows.etl.bordereau.py", line 92, in <module>
    serialise.serialise(flow, "ExtractLoad", config.FLOW_LABEL)
  File "****************************/flows/tasks/configuration/serialise.py", line 12, in serialise
    add_default_labels=False
  File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1734, in register
    idempotency_key=idempotency_key,
  File "****************************/.venv/lib/python3.6/site-packages/prefect/client/client.py", line 1127, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1497, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 303, in build
    self._build_image(push=push)
  File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 370, in _build_image
    "Your docker image failed to build!  Your flow might have "
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
So, we think the issue is due to calling 
docker.from_env()
when the healthchecks are run, right? At that point docker is not available to the client as it is within a CI container. So, we want to call it when registering but not when running a healthcheck -- have I got that right?
What is a confusing me is how my other Flows (in the same repo) are able to be registered, built and pushed to ECR...
z
I believe that's what's happening yeah.
I'm a bit perplexed from over here though 😄 if you disable healthchecks does your flow run without issue?
You can also put a 
try/except
around the 
docker.from_env()
call and just return a vanilla 
Docker
storage since we're not actually going to do anything with it on load
k
Ha same! I have just found 
disable healthchecks
and that did build the image all fine. I am testing the run soon. Once I have got this running (albeit circumventing health checks) I will add a try catch to help debug this. This behaviour seems to have come out of nowhere from our POV. Thanks @Zanie
So it looks like this Flow does not get unpickled correctly:
Copy code
[1 September 2021 9:27pm]: Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n DockerException("Error while fetching server API version: (\'Connection aborted.\', FileNotFoundError(2, \'No such file or directory\'))",)\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with \'3.6.5\', currently running with \'3.6.14\')',)
Now the 3.6.5 vs 3.6.14 is an obvious call out. We were previously pinning to 3.6, which could explain this drift. I am going to pin it now and see if it makes a difference.
z
Makes sense that it fails to unpickle, it's the same thing the healthcheck is catching.
A try/except on 
from_env
feels like the best bet for now.
k
I am adding the try/except to see if that is the same issue. This might be a totally stupid question but if I am using Prefect Cloud how is Python version of the execution controlled? I have seemingly pinned everywhere to 3.6.5 but I get the same issue in the Prefect UI.
The agent, possibly.
z
Since you're using docker storage, it should be the version of Python in the docker image
From your error, sounds like 3.6.5 in CI where 
prefect register
is being called and 3.6.14 in the container
k
hmm I am going to ssh in, but I am 99% sure 3.6.5 is used everywhere and pinned. (clearly something is using 3.6.14...) What is baffling is as I register my current flows they all build and push to ECR and the Flows run in Prefect (a part from this problematic one). It is this single Flow which does not build (without ignoring the healthcheck) and complains about not having access to Docker. I am trying to test now with try/except to see if there is some masked error going on.
Ha, ok well adding a try except to the function from earlier:
Copy code
def DockerCircleCI(**kwargs: Any) -> Any:
    try:
        default_client = docker.from_env()
        # This will break local builds / is only needed for remote docker hosts
        if default_client.api.base_url != "<http+docker://localhost>":
            kwargs.setdefault("base_url", default_client.api.base_url)

        kwargs.setdefault("tls_config", docker.TLSConfig(
            default_client.api.cert))

        return Docker(**kwargs)
    except Exception as e:
        print('THIS IS THE EXCEPTION CAUGHT')
        print(e)
        return Docker(**kwargs)
printed this in the logs:
Copy code
Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
 ---> Running in 697db9831d6e
Beginning health checks...
System Version check: OK
THIS IS THE EXCEPTION CAUGHT
Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Cloudpickle serialization check: OK
Result check: OK
Environment dependency check: OK
All health checks passed.
The image built and ran in Prefect (without randomly complaining about 3.6.14...) 🤷‍♂️ I literally have no words?! ha
z
I think we only show the version mismatch if there's an exception to help track down root causes
k
ah ok, that explains that. Thanks @Zanie I am not really sure what to do tbh. I cant isolate the issue or get the images to build (without handling the exception like above). We have quite a few business critical Flows which we have been relying on for quite a long time now. I am also none the wiser as to what caused this change in the first place tbh. It looks like (regardless of whether I ignore the healthchecks or handle them) the images being built do not read the environment variables from CirlceCI, which hold our secrets and credentials. So, is the fix here to move all of our config out of CircleCi and into Prefect Secrets??
My last remaining theory is to try figure out I can get the docker container to connect with the docker server. Something like this https://link.medium.com/voIP4qcUcjb to try and solve the docker in a docker issue ...
@Zanie do you know any way of passing any flags into Prefect for when it builds the Flow image? I am trying to think about how I could instruct that container to mount its socket. Also, I noticed that the python image being used by Prefect is:
Copy code
Step 1/10 : FROM prefecthq/prefect:0.15.4-python3.6
How is that controlled? As that could create an image with 3.6.14 running.
z
That will indeed pull the latest 3.6 release since it's based on the python 3.6 images
What's wrong with the try/except? It'll fail to build your flow if it errors in a meaningful way and it will not throw the error when you don't care
15 Views
Open in Slack
PreviousNext
Powered by