I m running into an issue where I cannot build my flow using Prefect Community #ask-community

I'm running into an issue where I cannot build my ...

Marc Lipoff

04/21/2021, 2:49 PM

I'm running into an issue where I cannot build my flow (using docker storage). I am running:

Copy code

prefect build -p /home/circleci/project/data_flows/flows/vaccine_cdc.py

(log in thread)

Zanie

04/21/2021, 2:50 PM

Hey!

Copy code

sock.connect(self.unix_socket)
      FileNotFoundError: [Errno 2] No such file or directory

Usually indicates that there is no Docker engine running (it's trying to connect to a socket, not a typical file)

Marc Lipoff

04/21/2021, 2:51 PM

ah ok. that sort of makes sense (would be nice if docker was a little more explicit)

Kevin Kho

04/21/2021, 3:01 PM

Hi @Marc Lipoff! Just a friendly reminder if you could move the log into a thread next time to not crowd the main channel.

Marc Lipoff

04/21/2021, 3:12 PM

Copy code

Building `Docker` storage...
    Error building storage:
      Traceback (most recent call last):
        File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
        File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 950, in send
    self.connect()
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
      PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/home/circleci/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
        File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
        File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
        File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
        File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
        File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
        File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/http/client.py", line 950, in send
    self.connect()
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
      urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
        File "/home/circleci/.local/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
        File "/home/circleci/.local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
        File "/home/circleci/.local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
        File "/home/circleci/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
      requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/cli/build_register.py", line 873, in build
    storage.build()
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 303, in build
    self._build_image(push=push)
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 333, in _build_image
    self.pull_image()
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 559, in pull_image
    client = self._get_client()
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 545, in _get_client
    return docker.APIClient(
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
        File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
      docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

  Building 'vaccine-cdc'... Error

Marc Lipoff

04/21/2021, 3:18 PM

I "believe" the problem that I am facing is that the prefect docker commands are not finding the the circleci "remote_docker". That remote_docker container is a step in the job, via:

Copy code

{
                 setup_remote_docker: {
                   docker_layer_caching: true,
                 },
               }

My circleci knowledge is fairly limited, so any help would be appreciated

Zanie

04/21/2021, 3:20 PM

Ah yeah this is kind of a pain -- there's an open issue to fix the default setting detection on CircleCI

Zanie

04/21/2021, 3:20 PM

https://github.com/PrefectHQ/prefect/issues/3987

Marc Lipoff

04/21/2021, 3:22 PM

ah ok great.

Zanie

04/21/2021, 3:22 PM

If you just drop that function in place of

Docker

you should be good to go

Marc Lipoff

05/26/2021, 3:27 PM

Hi @Zanie I hate to bring this up a month later... I've still having similiar issues. My storage looks something like this:

Copy code

"storage": DockerCircleCI(
            registry_url=ecr_registry_url,  # type: ignore
            image_name=_create_repo_name(flow_name, environment),
            python_dependencies=python_dependencies,
            files=files_to_move_over,
            env_vars={"PYTHONPATH": "$PYTHONPATH:/:data_flows"},
            stored_as_script=True,
            path=os.path.join("data_flows/flows/", file_name),
        )

But when the healthchecks are run as part of the

prefect build

, I get this:

Copy code

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.r
equest('GET', url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 150, in <module>
    flows = import_flow_from_script_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 63, in import_flow_from_script_check
    flows.append(extract_flow_from_file(file_path=flow_file_path))
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/storage.py", line 86, in extract_flow_from_file
    exec(contents, exec_vals)
  File "<string>", line 129, in <module>
  File "/data_flows/utils/deployment.py", line 237, in get_configs
    "storage": DockerCircleCI(
  File "/data_flows/utils/deployment.py", line 40, in DockerCircleCI
    default_client = docker.from_env()
  File "/usr/local/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/usr/local/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Removing intermediate container be8e964f1a0f
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["data_flows/flows/vaccine_cdc.py"]' '(3, 8)'' returned a non-zero code: 1
    Error building storage:
      Traceback (most recent call last):
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/cli/build_register.py", line 901, in build
    storage.build()
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 303, in build
    self._build_image(push=push)
        File "/home/circleci/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 370, in _build_image
    raise ValueError(
      ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.

  Building 'vaccine-cdc'... Error

To reiterate, I'm trying to run this as a job within CircleCI. My job looks something like this:

Copy code

run('Set python version', |||
          echo "Must remove .python-version as that does not match the python version installed"
          rm .python-version && pyenv global 3.8.5 && python --version && which python
        |||),
        restore_cache('pip-cache-{{ checksum "data_flows/requirements.txt" }}', ['./']),
        run('Install AWS CLI', |||
          python -m pip install awscli --user
        |||),
        run('Install Base Python Reqs', |||
          python -m pip install -r ${BASE_PATH}/requirements.txt --user
        |||),
        save_cache('pip-cache-{{ checksum "data_flows/requirements.txt" }}', ['./']),
      ] + [
        {
          'aws-cli/configure': {
            'aws-access-key-id': '$AWS_ACCESS_KEY_ID',
            'aws-region': '$AWS_DEFAULT_REGION',
          },
        },
        {
          setup_remote_docker: {
            docker_layer_caching: true,
          },
        },
      ] + [
        run('Install Dependencies, for Build Only', |||
          echo "Install basic python deps so that we can build. Being exact is not so important."
          python ${BASE_PATH}/utils/install_dependencies.py ${BASE_PATH}/flows/vaccine_cdc.py
        |||),
        run('Build Prefect Image', |||
          export PYTHONPATH=$PYTHONPATH:$CIRCLE_WORKING_DIRECTORY
          export PYTHONPATH=$PYTHONPATH:$CIRCLE_WORKING_DIRECTORY/data_flows
          prefect build -p ${BASE_PATH}/flows/vaccine_cdc.py. #### <------ THIS IS THE PROBLEM LINE
        |||),

Zanie

05/26/2021, 3:34 PM

It looks like your

DockerCircleCI

storage is failing to connect to the Docker Engine during the healthcheck. It seems like it should not need to do that.. ~~Could you try setting your storage in an if main block? Like~~

Zanie

05/26/2021, 3:35 PM

Nevermind I don't think that will work with the build command.

Zanie

05/26/2021, 3:37 PM

Perhaps try:

Copy code

if prefect.context.get("loading_flow"):
    flow.storage = DockerCircleCI(...)

Zanie

05/26/2021, 3:39 PM

I think you could also place this block in your

get_configs

function and just return

None

for storage if it's not being loaded for registration. I imagine there is a better fix here but I want to make sure this is the issue first.

Marc Lipoff

05/26/2021, 3:41 PM

... let me give that a whirl

Marc Lipoff

05/26/2021, 6:17 PM

thanks. that worked

Zanie

05/26/2021, 6:18 PM

That's good to know! I'm not sure how to fix this in the long run yet

Zanie

05/26/2021, 6:19 PM

You could put a

try/except

around the CI specific portion of your custom docker storage

Zanie

05/26/2021, 6:19 PM

We want to have a general solution for that eventually as well

Marc Lipoff

06/02/2021, 1:59 PM

Hi @Zanie thanks for the above. That makes sense. But after playing with it more, that solution wont work, for 2 reasons: 1. I eventually do need to build it as a docker storage, for the registration process. Using the

prefect register...

command yields this same error 2. I do want to build it as a docker storage, even before registering. The reason is that the build process uses the "healthcheck" which is really helpful to find issues earlier rather than later. This is all to say: is there a better way to still use the docker storage and make it work with circleci?

Zanie

06/02/2021, 2:15 PM

I guess I'm a bit confused because we use docker storage in circle CI with the function I provided and we have not encountered any healthcheck issues

Zanie

06/02/2021, 2:19 PM

So perhaps this actually has something to do with your docker image

Zanie

06/02/2021, 2:19 PM

I feel like doing

Copy code

def DockerCircleCI(**kwargs):
    import docker
    import warnings

    try:
        default_client = docker.from_env()

        # This will break local builds / is only needed for remote docker hosts
        if default_client.api.base_url != "<http+docker://localhost>":
            kwargs.setdefault("base_url", default_client.api.base_url)

        kwargs.setdefault("tls_config", docker.TLSConfig(default_client.api.cert))
        
    except Exception:
        warnings.warn("Failed to load docker settings from the environment! Continuing with defaults...")
    
    return Docker(**kwargs)

Zanie

06/02/2021, 2:20 PM

Should do it; the issue seems to be that during the healthcheck the Docker storage class is being instantiated and

docker.from_env()

is failing, but you don't really need the storage class to load during the healthcheck.

Marc Lipoff

06/02/2021, 3:44 PM

Hm ya i hear what you are saying.

Marc Lipoff

06/02/2021, 3:45 PM

Do you use the

setup_remote_docker

from docker?

Zanie

06/02/2021, 3:59 PM

Yeah

Marc Lipoff

06/07/2021, 1:40 PM

Hi @Zanie I'm back 😉 So, I've got it mostly working, but I'm getting intermittent SSL errors. It's intermittent in that some flow register/build just fine. But others encounter an SSL error. The stack trace is below. Any ideas what up there (I've also tried wrapping the

docker.from_env()

in a retired loop with no success)

Copy code

Traceback (most recent call last):
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
    File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
    File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
    File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
  ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)
  
During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/circleci/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
    File "/home/circleci/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
  urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='35.196.50.158', port=2376): Max retries exceeded with url: /version (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))
  
During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
    File "/home/circleci/.local/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
    File "/home/circleci/.local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
    File "/home/circleci/.local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
    File "/home/circleci/.local/lib/python3.8/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
  requests.exceptions.SSLError: HTTPSConnectionPool(host='35.196.50.158', port=2376): Max retries exceeded with url: /version (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))
  
During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/circleci/.local/lib/python3.8/site-packages/prefect/cli/build_register.py", line 126, in load_flows_from_script
    namespace = runpy.run_path(path, run_name="<flow>")
    File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
    File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    File "/home/circleci/.pyenv/versions/3.8.5/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
    File "data_flows/flows/shipping_forecast.py", line 673, in <module>
    **get_configs(
    File "/home/circleci/project/data_flows/utils/deployment.py", line 253, in get_configs
    storage = DockerCircleCI(
    File "/home/circleci/project/data_flows/utils/deployment.py", line 45, in DockerCircleCI
    default_client = docker.from_env()
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
    return cls(
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
    File "/home/circleci/.local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
  docker.errors.DockerException: Error while fetching server API version: HTTPSConnectionPool(host='35.196.50.158', port=2376): Max retries exceeded with url: /version (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

Marc Lipoff

06/09/2021, 4:01 PM

@Zanie just want to follow up on this, and see if you had any thoughts?

Zanie

06/09/2021, 4:09 PM

Weird

Zanie

06/09/2021, 4:10 PM

I'm not really sure what's going on here; this is a

docker.from_env()

bug with CircleCI -- not a Prefect thing

Marc Lipoff

06/09/2021, 6:43 PM

Ok thanks. I am trying to use the "parallelism" functionality in circelci. I am able to get it to run smoothly without it. But when parallelism is on, I get this. I'm going to continue debugging.

Zanie

06/09/2021, 6:44 PM

Ah that's tricky. You may be better off opening an issue in the CirlceCI or the Docker Client repos

4 Views

Open in Slack

Previous Next