https://prefect.io logo
Join Slack
Powered by
Question for folks doing CI/CD automation to build...
# prefect-community
j
Question for folks doing CI/CD automation to build & register Prefect Flows. Building Docker storage for one of our flows from our CI/CD system (Screwdriver.io) fails with the exception below. Docker appears to be installed and working properly because 
docker run hello-world
succeeds. (There is special "docker-in-docker" config we use with Screwdriver and this works well for many of our other projects.) I can also build Docker storage successfully on my laptop from inside the same image that our CI/CD system is using. I'm sure it's some simple config that I'm missing -- anybody run into this?
Copy code
15:25:50 Traceback (most recent call last):
15:25:50   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
15:25:50     "__main__", mod_spec)
15:25:50   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
15:25:50     exec(code, run_globals)
15:25:50   File "/usr/local/lib/python3.7/site-packages/srm_datasci-0.1.0-py3.7.egg/srm/datasci/flows/util/test_flow.py", line 35, in <module>
15:25:50     flow.register("Utilities")
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1412, in register
15:25:50     no_url=no_url,
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 616, in register
15:25:50     serialized_flow = flow.serialize(build=build)  # type: Any
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1221, in serialize
15:25:50     storage = self.storage.build()  # type: Optional[Storage]
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 263, in build
15:25:50     self._build_image(push=push)
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 293, in _build_image
15:25:50     self.pull_image()
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 483, in pull_image
15:25:50     client = self._get_client()
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 471, in _get_client
15:25:50     return docker.APIClient(base_url=self.base_url, version="auto")
15:25:50   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 190, in __init__
15:25:50     self._version = self._retrieve_server_version()
15:25:50   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 215, in _retrieve_server_version
15:25:50     'Error while fetching server API version: {0}'.format(e)
15:25:50 docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
I suspect it's related to this line in the Prefect Docker storage code: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/storage/docker.py#L96)
Was able to work through this. Updating here in case anyone runs into something similar. Screwdriver configures their "docker-in-docker" setup with DOCKER_HOST set to 
<tcp://localhost:2375>
So for us the change was:
Copy code
from prefect.environments.storage import Docker

storage = Docker(
    registry_url="<redacted>",
    image_name="<redacted>",
    image_tag=IMAGE_TAG,
    python_dependencies=["<redacted>"],
    base_url=environ.get("DOCKER_HOST"),
)
It might be good to bubble this change down into Prefect, e.g:
Copy code
default_url = environ.get("DOCKER_HOST", "<unix://var/run/docker.sock>")
Here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/storage/docker.py#L96
l
ah, you beat me (though my only contribution was to note about base_url being an option for you to configure once you knew what your CI’s docker api endpoint really was)
I like your idea, reading docker docs briefly it looks like DOCKER_HOST is an env var natively supported by the docker client if not dockerd itself
j
Thanks @Laura Lorenz (she/her), that all makes sense!
l
I’m going to try out the issue bot 🙂
marvin 1
👍 1
hmm its not that
@Marvin archive “set default_url for Docker storage off DOCKER_HOST env var if it exists”
m
Open in Slack
PreviousNext
Powered by