Andy Dyer

    Andy Dyer

    1 year ago
    Hey all we are just about to finish our first prefect flow and have enjoyed the dev experience. I am trying to register my flow in gitlabs CI and push its Docker storage to google container registry, but I am getting two errors I assume revolving around Docker in Docker issues. Please see below
    With DOCKER_HOST = <tcp://docker:2376>
    
        raise DockerException(
    docker.errors.DockerException: Error while fetching server API version: HTTPConnectionPool(host='docker', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb67baf4760>: Failed to establish a new connection: [Errno 111] Connection refused'))
    
    
    without DOCKER_HOST set 
    
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    nicholas

    nicholas

    1 year ago
    Hi @Andy Dyer - when you say register your flow in gitlabs CI, what do you mean? Register it with Prefect Cloud/Server in a gitlab CI pipeline? Is that when the docker image push step is failing?
    Andy Dyer

    Andy Dyer

    1 year ago
    Register it with Prefect Cloud/Server in a gitlab CI pipeline? Is that when the docker image push step is failing?
    yeah we are using a cloud backend and the registration works locally. I dont think its a prefect issue. Probably more of a connection to the docker daemon running in gitlab ci
    ci script looks like
    pip3 install -r requirements.txt;
          printf "%s" "$SERVICE_KEY_JSON" > "keyfile.json";
          gcloud auth activate-service-account --key-file keyfile.json;
          prefect backend cloud;
          prefect auth login --token $PREFECT_TENANT_TOKEN;
          python3 main.py;
    nicholas

    nicholas

    1 year ago
    Yeah that sounds correct, it seems like there's a daemon connection problem external to Prefect. Is there a way to ensure your CI machine has an accessible docker daemon of a recent version maybe?
    Andy Dyer

    Andy Dyer

    1 year ago
    kind of a DevOps noob can you give me some info on how to find that out?
    like the CI machine itself is running a gcloud docker image that is running the above script
    nicholas

    nicholas

    1 year ago
    Ah ok, then it sounds like the issue is that the container you're running (gcloud in this case) probably doesn't have docker installed. Try installing docker in your CI script ahead of time with whatever commands make sense for that image. You can find the installation instructions in the docker docs here: https://docs.docker.com/engine/install/
    Andy Dyer

    Andy Dyer

    1 year ago
    so it does have docker installed because
    which docker
    does point to
    /usr/local/bin/docker
    nicholas

    nicholas

    1 year ago
    Can you access the docker daemon from within the container?
    Andy Dyer

    Andy Dyer

    1 year ago
    like just
    ps aux | grep dockerd
    ?
    nicholas

    nicholas

    1 year ago
    Perhaps just
    dockerd
    Andy Dyer

    Andy Dyer

    1 year ago
    /bin/bash: line 120: dockerd: command not found
    do you have an image in mind that would have it preinstalled. I can just switch to installing the gcloud cli probably faster as a whole
    nicholas

    nicholas

    1 year ago
    Yeah that's what I'd suggest, that's usually easiest. I don't have one in mind though, that'll be specific more to your pipeline
    Andy Dyer

    Andy Dyer

    1 year ago
    does the prefect image have it preinstalled?
    nicholas

    nicholas

    1 year ago
    it does!
    Andy Dyer

    Andy Dyer

    1 year ago
    so I have the prefect image, installed gcloud and am authed but still
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='docker', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f724125ae80>: Failed to establish a new connection: [Errno 111] Connection refused'))
    docker.errors.DockerException: Error while fetching server API version: HTTPConnectionPool(host='docker', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f724125ae80>: Failed to establish a new connection: [Errno 111] Connection refused'))
    nicholas

    nicholas

    1 year ago
    What's the value of
    $DOCKER_HOST
    in your machine's environment?
    Andy Dyer

    Andy Dyer

    1 year ago
    DOCKER_HOST=<tcp://docker:2376>