Thread
#prefect-community
    k

    Kieran

    1 year ago
    Hey, We use CircleCI to test and register our flows (Prefect v0.14.12) which we use Docker storage for. In our recent attempts to register a new flow we have been hit with this error and failing to build:
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    (full trace in this thread) From reading around is looks like a potential docker issue, an inability to connect to a socket?! Has anyone else been hit with this?
    Removing intermediate container ffc26cbf530e
     ---> 720f6f321752
    Step 6/10 : RUN mkdir -p /opt/prefect/
     ---> Running in 679b82f2fde9
    Removing intermediate container 679b82f2fde9
     ---> 9d78552ba1d9
    Step 7/10 : COPY bordereau.flow /opt/prefect/flows/bordereau.prefect
     ---> f13939593f26
    Step 8/10 : COPY healthcheck.py /opt/prefect/healthcheck.py
     ---> 04b9344f7e76
    Step 9/10 : COPY tasks /opt/prefect/flows/tasks
     ---> 50ac16185624
    Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
     ---> Running in f78c279cb415
    Beginning health checks...
    System Version check: OK
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.6/http/client.py", line 980, in send
        self.connect()
      File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
        sock.connect(self.unix_socket)
    FileNotFoundError: [Errno 2] No such file or directory
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
        timeout=timeout
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
        raise value.with_traceback(tb)
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.6/http/client.py", line 980, in send
        self.connect()
      File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
        sock.connect(self.unix_socket)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
        return self.version(api_version=False)["ApiVersion"]
      File "/usr/local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
        return self._result(self._get(url), json=True)
      File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
        return f(self, *args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 237, in _get
        return self.get(url, **self._set_request_timeout(kwargs))
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
        return self.request('GET', url, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
        raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 152, in <module>
        flows = cloudpickle_deserialization_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
        flows.append(cloudpickle.loads(flow_bytes))
      File "/opt/prefect/flows/tasks/commands/extract_and_load.py", line 9, in <module>
        from flows.tasks.configuration.config import (DEFAULT_DESTINATION_SCHEMA,
      File "/opt/prefect/flows/tasks/configuration/config.py", line 22, in <module>
        base_url=docker.from_env().api.base_url,
      File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 101, in from_env
        **kwargs_from_env(**kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 45, in __init__
        self.api = APIClient(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 197, in __init__
        self._version = self._retrieve_server_version()
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
        'Error while fetching server API version: {0}'.format(e)
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! This is an error where the docker client cannot find the docker engine server. The "file" that's missing is a socket.
    k

    Kieran

    1 year ago
    @Michael Adkins thanks for the quick reply!
    Michael Adkins

    Michael Adkins

    1 year ago
    There's a workaround there 🙂
    k

    Kieran

    1 year ago
    omg thank you @Michael Adkins, I will try that work around now!
    @Michael Adkins just so I fully understand the fix -- currently, I am using the following:
    default_client = docker.from_env()
    
    dockerised_storage = Docker(
        base_url=default_client.api.base_url,
        tls_config=docker.TLSConfig(default_client.api.cert),
        registry_url=''
        python_dependencies=python_deps
    )
    The workaround checks whether the .from_env().api.base_url method is a remote host and then sets it to the variable
    default_client
    . Is that not how I currently have it set up? Or have I been really stupid and mis-read the workaround?
    Michael Adkins

    Michael Adkins

    1 year ago
    What you're doing should work in CircleCI, I think. It does look like the same thing the workaround does (except the workaround will also work on your local machine)
    I may have misread your traceback actually, on a second look this is failing during the healthcheck
    k

    Kieran

    1 year ago
    Sorry, the above configuration is how I currently have things set up and they don't work in CircleCI. Sorry, I should have been clearer.
    Michael Adkins

    Michael Adkins

    1 year ago
    Ah so because you're calling
    docker.from_env()
    in your flow script, when we try to run the healthcheck it is trying to contact docker from within the docker image it has just built and is failing because there's no docker in docker setup.
    Can you try using my workaround instead which is local-compatible? It might still fail on the
    from_env
    call but I'm not sure.
    k

    Kieran

    1 year ago
    When should we be calling
    docker.from_env()
    if not when registering a flow? I am not sure if I understand the workaround but this is what I am currently testing:
    def DockerCircleCI(**kwargs: Any) -> Any:
        import docker
    
        default_client = docker.from_env()
    
        # This will break local builds / is only needed for remote docker hosts
        if default_client.api.base_url != "<http+docker://localhost>":
            kwargs.setdefault("base_url", default_client.api.base_url)
    
        kwargs.setdefault("tls_config", docker.TLSConfig(default_client.api.cert))
    
        return Docker(**kwargs)
    
    
    dockerised_storage = DockerCircleCI(
        registry_url="",
        python_dependencies=python_deps
    )
    I am then passing
    dockerised_storage
    into the
    storage
    parameter of
    Flow()
    .
    Michael Adkins

    Michael Adkins

    1 year ago
    Yep, that is how you'd use it
    The issue is that
    from_env()
    is being called when registering the flow then again when the flow script is run within the built storage for a healthcheck.
    (I think this is what is happening)
    k

    Kieran

    1 year ago
    Thanks, whilst I am testing this do you know what introduced this bug? Was it some sub-dependency that has changed?
    Afraid no luck, here is the traceback:
    Step 1/10 : FROM prefecthq/prefect:0.14.15-python3.6
     ---> cece2ab0a690
    Step 2/10 : ENV PYTHONPATH='$PYTHONPATH:/opt/prefect/:****************************/flows/tasks'     PREFECT__USER_CONFIG_PATH='/opt/prefect/config.toml'
     ---> Using cache
     ---> 97d69ce7f159
    Step 3/10 : RUN pip install pip --upgrade
     ---> Using cache
     ---> 3f1a0cea00fa
    Step 4/10 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.14.15#egg=prefect[all_orchestration_extras]>
     ---> Using cache
     ---> 5b2e98a043c8
    Step 5/10 : RUN pip install docker==4.4.1 requests==2.25.1 pandas==1.1.5 sqlalchemy==1.3.22 psycopg2-binary==2.8.6 pygit2==1.5.0 pact-python==1.3.0 awscli==1.19.46 gspread==3.7.0 oauth2client==4.1.3 wheel
     ---> Using cache
     ---> 720f6f321752
    Step 6/10 : RUN mkdir -p /opt/prefect/
     ---> Using cache
     ---> 9d78552ba1d9
    Step 7/10 : COPY bordereau.flow /opt/prefect/flows/bordereau.prefect
     ---> ea6f3b54fe04
    Step 8/10 : COPY healthcheck.py /opt/prefect/healthcheck.py
     ---> 93747758af17
    Step 9/10 : COPY tasks /opt/prefect/flows/tasks
     ---> 31dd22e4b1c1
    Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
     ---> Running in 780c3747c0f6
    Beginning health checks...
    System Version check: OK
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.6/http/client.py", line 980, in send
        self.connect()
      File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
        sock.connect(self.unix_socket)
    FileNotFoundError: [Errno 2] No such file or directory
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
        timeout=timeout
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
        raise value.with_traceback(tb)
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.6/http/client.py", line 980, in send
        self.connect()
      File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
        sock.connect(self.unix_socket)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
        return self.version(api_version=False)["ApiVersion"]
      File "/usr/local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
        return self._result(self._get(url), json=True)
      File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
        return f(self, *args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 237, in _get
        return self.get(url, **self._set_request_timeout(kwargs))
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
        return self.request('GET', url, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
        raise ConnectionError(err, request=request)
    requests.excep
    tions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 152, in <module>
        flows = cloudpickle_deserialization_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
        flows.append(cloudpickle.loads(flow_bytes))
      File "/usr/local/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 562, in subimport
        __import__(name)
      File "/opt/prefect/flows/tasks/configuration/config.py", line 43, in <module>
        "PYTHONPATH": "$PYTHONPATH:/opt/prefect/:****************************/flows/tasks"
      File "/opt/prefect/flows/tasks/configuration/config.py", line 25, in DockerCircleCI
        default_client = docker.from_env()
      File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 101, in from_env
        **kwargs_from_env(**kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 45, in __init__
        self.api = APIClient(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 197, in __init__
        self._version = self._retrieve_server_version()
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
        'Error while fetching server API version: {0}'.format(e)
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    Removing intermediate container 780c3747c0f6
    The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'' returned a non-zero code: 1
    ****************************/.venv/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host '35.224.236.246'. Adding certificate verification is strongly advised. See: <https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings>
      InsecureRequestWarning,
    Traceback (most recent call last):
      File "flows/flows.etl.bordereau.py", line 87, in <module>
        serialise.serialise(flow, "ExtractLoad", config.FLOW_LABEL)
      File "****************************/flows/tasks/configuration/serialise.py", line 12, in serialise
        add_default_labels=False
      File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1710, in register
        idempotency_key=idempotency_key,
      File "****************************/.venv/lib/python3.6/site-packages/prefect/client/client.py", line 797, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1485, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 303, in build
        self._build_image(push=push)
      File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 371, in _build_image
        "Your docker image failed to build!  Your flow might have "
    ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
    
    Exited with code exit status 1
    Michael Adkins

    Michael Adkins

    1 year ago
    Nothing should have changed for this on our end. Did you even switch Prefect versions?
    k

    Kieran

    1 year ago
    No, nothing has changed from our perspective...
    Michael Adkins

    Michael Adkins

    1 year ago
    And your cloudpickle/dockerpy versions are the same?
    The deserialization healthcheck is failing
    k

    Kieran

    1 year ago
    We haven't been pinning those packages. They are being used by Prefect under the hood, right?
    Michael Adkins

    Michael Adkins

    1 year ago
    Yeah and we pin to an upper bound to stop breaking changes from getting in, but that's my best theory right now.
    Can you do
    if not prefect.context.loading_flow:
        flow.storage = Docker(...)
    else:
        flow.storage = None
    We set a context variable so you can avoid doing things that are sensitive like this
    k

    Kieran

    1 year ago
    Sure, I am just trying to resolve dependencies and pin cloudpickle down + add that conditional to the flow.
    prefect = { version = "0.14.12", extras = ["aws"] }
    docker = "==4.4.1"
    cloudpickle = "==1.5.0"
    ^ for reference.
    @Michael Adkins are you sure on the context method name? I am getting:
    AttributeError: 'Context' object has no attribute 'loading_flow'
    if not prefect.context.get("loading_flow", False):
            flow.storage = config.default_flow_storage
        else:
            flow.storage = None
    ^ I think that is how to pull it out.
    Michael Adkins

    Michael Adkins

    1 year ago
    Sorry yeah a
    get
    is necessary.
    k

    Kieran

    1 year ago
    @Michael Hadorn, no luck I am afraid. Still the same error on the healthcheck:
    Step 1/10 : FROM prefecthq/prefect:0.15.4-python3.6
     ---> 41e4970e0048
    Step 2/10 : ENV PYTHONPATH='$PYTHONPATH:/opt/prefect/:****************************/flows/tasks'     PREFECT__USER_CONFIG_PATH='/opt/prefect/config.toml'
     ---> Using cache
     ---> c7b74b0f1a4f
    Step 3/10 : RUN pip install pip --upgrade
     ---> Using cache
     ---> 28f093a33f7e
    Step 4/10 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.15.4#egg=prefect[all_orchestration_extras]>
     ---> Using cache
     ---> 5fc1e26879a9
    Step 5/10 : RUN pip install docker==4.4.1 requests==2.25.1 pandas==1.1.5 sqlalchemy==1.3.22 psycopg2-binary==2.8.6 pygit2==1.5.0 pact-python==1.3.0 awscli==1.20.33 gspread==3.7.0 oauth2client==4.1.3 cloudpickle==1.5.0 types-requests==2.25.6 types-pyyaml==5.4.10 wheel
     ---> Using cache
     ---> 33cbba1c8169
    Step 6/10 : RUN mkdir -p /opt/prefect/
     ---> Using cache
     ---> 3c7332eead15
    Step 7/10 : COPY bordereau.flow /opt/prefect/flows/bordereau.prefect
     ---> a42ba277e1dd
    Step 8/10 : COPY healthcheck.py /opt/prefect/healthcheck.py
     ---> 823ad6db6f40
    Step 9/10 : COPY tasks /opt/prefect/flows/tasks
     ---> ecf269287600
    Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
     ---> Running in e75a88b55e15
    Beginning health checks...
    System Version check: OK
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.6/http/client.py", line 1291, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1337, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1286, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1046, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.6/http/client.py", line 984, in send
        self.connect()
      File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
        sock.connect(self.unix_socket)
    FileNotFoundError: [Errno 2] No such file or directory
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
        timeout=timeout
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 769, in reraise
        raise value.with_traceback(tb)
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.6/http/client.py", line 1291, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1337, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1286, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.6/http/client.py", line 1046, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.6/http/client.py", line 984, in send
        self.connect()
      File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
        sock.connect(self.unix_socket)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
        return self.version(api_version=False)["ApiVersion"]
      File "/usr/local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
        return self._result(self._get(url), json=True)
      File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
        return f(self, *args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 237, in _get
        return self.get(url, **self._set_request_timeout(kwargs))
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
        return self.request('GET', url, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
        raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 152, in <module>
        flows = cloudpickle_deserialization_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
        flows.append(cloudpickle.loads(flow_bytes))
      File "/opt/prefect/flows/tasks/commands/extract_and_load.py", line 9, in <module>
        from flows.tasks.configuration.config import (DEFAULT_DESTINATION_SCHEMA,
      File "/opt/prefect/flows/tasks/configuration/config.py", line 43, in <module>
        "PYTHONPATH": "$PYTHONPATH:/opt/prefect/:****************************/flows/tasks"
      File "/opt/prefect/flows/tasks/configuration/config.py", line 25, in DockerCircleCI
        default_client = docker.from_env()
      File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 101, in from_env
        **kwargs_from_env(**kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/client.py", line 45, in __init__
        self.api = APIClient(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 197, in __init__
        self._version = self._retrieve_server_version()
      File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
        'Error while fetching server API version: {0}'.format(e)
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    
    The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'' returned a non-zero code: 1
    ****************************/.venv/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host '34.72.24.230'. Adding certificate verification is strongly advised. See: <https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings>
      InsecureRequestWarning,
    Traceback (most recent call last):
      File "flows/flows.etl.bordereau.py", line 92, in <module>
        serialise.serialise(flow, "ExtractLoad", config.FLOW_LABEL)
      File "****************************/flows/tasks/configuration/serialise.py", line 12, in serialise
        add_default_labels=False
      File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1734, in register
        idempotency_key=idempotency_key,
      File "****************************/.venv/lib/python3.6/site-packages/prefect/client/client.py", line 1127, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "****************************/.venv/lib/python3.6/site-packages/prefect/core/flow.py", line 1497, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 303, in build
        self._build_image(push=push)
      File "****************************/.venv/lib/python3.6/site-packages/prefect/storage/docker.py", line 370, in _build_image
        "Your docker image failed to build!  Your flow might have "
    ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
    So, we think the issue is due to calling
    docker.from_env()
    when the healthchecks are run, right? At that point docker is not available to the client as it is within a CI container. So, we want to call it when registering but not when running a healthcheck -- have I got that right?
    What is a confusing me is how my other Flows (in the same repo) are able to be registered, built and pushed to ECR...
    Michael Adkins

    Michael Adkins

    1 year ago
    I believe that's what's happening yeah.
    I'm a bit perplexed from over here though 😄 if you disable healthchecks does your flow run without issue?
    You can also put a
    try/except
    around the
    docker.from_env()
    call and just return a vanilla
    Docker
    storage since we're not actually going to do anything with it on load
    k

    Kieran

    1 year ago
    Ha same! I have just found
    disable healthchecks
    and that did build the image all fine. I am testing the run soon. Once I have got this running (albeit circumventing health checks) I will add a try catch to help debug this. This behaviour seems to have come out of nowhere from our POV. Thanks @Michael Adkins
    So it looks like this Flow does not get unpickled correctly:
    [1 September 2021 9:27pm]: Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n DockerException("Error while fetching server API version: (\'Connection aborted.\', FileNotFoundError(2, \'No such file or directory\'))",)\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with \'3.6.5\', currently running with \'3.6.14\')',)
    Now the 3.6.5 vs 3.6.14 is an obvious call out. We were previously pinning to 3.6, which could explain this drift. I am going to pin it now and see if it makes a difference.
    Michael Adkins

    Michael Adkins

    1 year ago
    Makes sense that it fails to unpickle, it's the same thing the healthcheck is catching.
    A try/except on
    from_env
    feels like the best bet for now.
    k

    Kieran

    1 year ago
    I am adding the try/except to see if that is the same issue. This might be a totally stupid question but if I am using Prefect Cloud how is Python version of the execution controlled? I have seemingly pinned everywhere to 3.6.5 but I get the same issue in the Prefect UI.
    The agent, possibly.
    Michael Adkins

    Michael Adkins

    1 year ago
    Since you're using docker storage, it should be the version of Python in the docker image
    From your error, sounds like 3.6.5 in CI where
    prefect register
    is being called and 3.6.14 in the container
    k

    Kieran

    1 year ago
    hmm I am going to ssh in, but I am 99% sure 3.6.5 is used everywhere and pinned. (clearly something is using 3.6.14...) What is baffling is as I register my current flows they all build and push to ECR and the Flows run in Prefect (a part from this problematic one). It is this single Flow which does not build (without ignoring the healthcheck) and complains about not having access to Docker. I am trying to test now with try/except to see if there is some masked error going on.
    Ha, ok well adding a try except to the function from earlier:
    def DockerCircleCI(**kwargs: Any) -> Any:
        try:
            default_client = docker.from_env()
            # This will break local builds / is only needed for remote docker hosts
            if default_client.api.base_url != "<http+docker://localhost>":
                kwargs.setdefault("base_url", default_client.api.base_url)
    
            kwargs.setdefault("tls_config", docker.TLSConfig(
                default_client.api.cert))
    
            return Docker(**kwargs)
        except Exception as e:
            print('THIS IS THE EXCEPTION CAUGHT')
            print(e)
            return Docker(**kwargs)
    printed this in the logs:
    Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/bordereau.prefect"]' '(3, 6)'
     ---> Running in 697db9831d6e
    Beginning health checks...
    System Version check: OK
    THIS IS THE EXCEPTION CAUGHT
    Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    Cloudpickle serialization check: OK
    Result check: OK
    Environment dependency check: OK
    All health checks passed.
    The image built and ran in Prefect (without randomly complaining about 3.6.14...) 🤷‍♂️ I literally have no words?! ha
    Michael Adkins

    Michael Adkins

    1 year ago
    I think we only show the version mismatch if there's an exception to help track down root causes
    k

    Kieran

    1 year ago
    ah ok, that explains that. Thanks @Michael Adkins I am not really sure what to do tbh. I cant isolate the issue or get the images to build (without handling the exception like above). We have quite a few business critical Flows which we have been relying on for quite a long time now. I am also none the wiser as to what caused this change in the first place tbh. It looks like (regardless of whether I ignore the healthchecks or handle them) the images being built do not read the environment variables from CirlceCI, which hold our secrets and credentials. So, is the fix here to move all of our config out of CircleCi and into Prefect Secrets??
    My last remaining theory is to try figure out I can get the docker container to connect with the docker server. Something like this https://link.medium.com/voIP4qcUcjb to try and solve the docker in a docker issue ...
    @Michael Adkins do you know any way of passing any flags into Prefect for when it builds the Flow image? I am trying to think about how I could instruct that container to mount its socket. Also, I noticed that the python image being used by Prefect is:
    Step 1/10 : FROM prefecthq/prefect:0.15.4-python3.6
    How is that controlled? As that could create an image with 3.6.14 running.
    Michael Adkins

    Michael Adkins

    1 year ago
    That will indeed pull the latest 3.6 release since it's based on the python 3.6 images
    What's wrong with the try/except? It'll fail to build your flow if it errors in a meaningful way and it will not throw the error when you don't care