Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
pacc-london-2023
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
j
Joe Schmid
03/13/2020, 7:34 PMQuestion for folks doing CI/CD automation to build & register Prefect Flows. Building Docker storage for one of our flows from our CI/CD system (Screwdriver.io) fails with the exception below. Docker appears to be installed and working properly because
docker run hello-world15:25:50 Traceback (most recent call last):
15:25:50 File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
15:25:50 "__main__", mod_spec)
15:25:50 File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
15:25:50 exec(code, run_globals)
15:25:50 File "/usr/local/lib/python3.7/site-packages/srm_datasci-0.1.0-py3.7.egg/srm/datasci/flows/util/test_flow.py", line 35, in <module>
15:25:50 flow.register("Utilities")
15:25:50 File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1412, in register
15:25:50 no_url=no_url,
15:25:50 File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 616, in register
15:25:50 serialized_flow = flow.serialize(build=build) # type: Any
15:25:50 File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1221, in serialize
15:25:50 storage = self.storage.build() # type: Optional[Storage]
15:25:50 File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 263, in build
15:25:50 self._build_image(push=push)
15:25:50 File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 293, in _build_image
15:25:50 self.pull_image()
15:25:50 File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 483, in pull_image
15:25:50 client = self._get_client()
15:25:50 File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 471, in _get_client
15:25:50 return docker.APIClient(base_url=self.base_url, version="auto")
15:25:50 File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 190, in __init__
15:25:50 self._version = self._retrieve_server_version()
15:25:50 File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 215, in _retrieve_server_version
15:25:50 'Error while fetching server API version: {0}'.format(e)
15:25:50 docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))I suspect it's related to this line in the Prefect Docker storage code: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/storage/docker.py#L96)
Was able to work through this. Updating here in case anyone runs into something similar. Screwdriver configures their "docker-in-docker" setup with DOCKER_HOST set to
<tcp://localhost:2375>from prefect.environments.storage import Docker
storage = Docker(
registry_url="<redacted>",
image_name="<redacted>",
image_tag=IMAGE_TAG,
python_dependencies=["<redacted>"],
base_url=environ.get("DOCKER_HOST"),
)default_url = environ.get("DOCKER_HOST", "<unix://var/run/docker.sock>")l
Laura Lorenz (she/her)
03/13/2020, 8:49 PMah, you beat me (though my only contribution was to note about base_url being an option for you to configure once you knew what your CI’s docker api endpoint really was)
I like your idea, reading docker docs briefly it looks like DOCKER_HOST is an env var natively supported by the docker client if not dockerd itself
j
Joe Schmid
03/13/2020, 8:53 PMThanks @Laura Lorenz (she/her), that all makes sense!
l
Laura Lorenz (she/her)
03/13/2020, 8:54 PMI’m going to try out the issue bot 🙂
:marvin: 1
👍 1
hmm its not that
@Marvin archive “set default_url for Docker storage off DOCKER_HOST env var if it exists”
m
Marvin
03/13/2020, 9:40 PM