https://prefect.io logo
Join Slack
Powered by
hello, i'm trying to do a docker deploy. It recomm...
# ask-community
s
hello, i'm trying to do a docker deploy. It recommends serving from the root directory (but i already have a docker image there), so i am getting the error
Copy code
RuntimeError: Failed to generate Dockerfile. Dockerfile already exists in the current directory.
i notice that
Copy code
def generate_default_dockerfile(context: Optional[Path] = None):
optionally can take a path context, but i don't see a way to specify that from my simple flow:
Copy code
if __name__ == "__main__":
    test_processing_flow.deploy(
        name="test-processor-flow",
        work_pool_name="test-docker-pool",
        image="test-processor-flow:latest",
        push=False
    )
am i missing something?
n
hi @Steven - if you don't want 
.deploy()
to build an image for you (because you already have one) then you can set 
build=False
alongside your 
push=False
s
the dockerfile in the root directory is unrelated, but one for the overall project
just trying to do this simple local docker deploy following the docs as they are and keep hitting snags
n
you can specify a 
Dockerfile
if you want 
.deploy()
to use it for the build like this
what other snags are you hitting?
s
oh i was trying to use the auto dockerfile generation but since we already had one at the root folder, it seems like there was no way to config where that temp dockerfile was configured from the flow.deploy
n
correct, it’s in a tmpdir so it gets wiped out. but you’re free to provide your own dockerfile as i mention
s
right, but there is no way to allow the autodocker config default path so it doesn't conflict with an already existing one in the root? just trying to make sure i didnt miss a config option. and thanks a lot for all of your replies
n
hmm i see what you mean, as a workaround i bet you could add a suffix to your dockerfile but yeah maybe the cli should warn that “hey, looks like you’ve got one already, if you want to use it do so and so..” and have the cli write instead to a suffixed Dockerfile.temp or something
s
@Nate i encountered a similar error yesterday with the auto dockerfile generation.
Copy code
`  File "/Users/user/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/utilities/dockerutils.py", line 114, in docker_client
    raise RuntimeError(
RuntimeError: This error is often thrown because Docker is not running. Please ensure Docker is running.
Copy code
from prefect.deployments import DeploymentImage

from neo4j_python.prefect.flows.malware_processor_flow import malware_processing_flow

if __name__ == "__main__":
    malware_processing_flow.deploy(
        name="malware-processor-flow",
        work_pool_name="stevens-docker-pool",
        image=DeploymentImage(
            name="malware-processor-image",
            tag="latest",
            dockerfile="Dockerfile.malware_processor_flow"
        ),
        push=False
    )
Copy code
# We're using the latest version of Prefect with Python 3.10
FROM prefecthq/prefect:2-python3.11

# Add our requirements.txt file to the image and install dependencies
# poetry export --format=requirements.txt --output=requirements.txt
COPY requirements.txt .
RUN pip install -r requirements.txt --trusted-host <http://pypi.python.org|pypi.python.org> --no-cache-dir

# Add our flow code to the image
COPY ../flows /opt/prefect/flows

# Run our flow script when the container starts
CMD ["python", "flows/malware_processor_flow.py"]
i definitely have docker desktop running on my machine
n
hmm im not sure this is the same thing from what you've provided here do you have more of the trace?
s
Copy code
Traceback (most recent call last):
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/connectionpool.py", line 497, in _make_request
    conn.request(
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/connection.py", line 395, in request
    self.endheaders()
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/http/client.py", line 1281, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/http/client.py", line 1041, in _send_output
    self.send(msg)
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/http/client.py", line 979, in send
    self.connect()
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/connectionpool.py", line 497, in _make_request
    conn.request(
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/urllib3/connection.py", line 395, in request
    self.endheaders()
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/http/client.py", line 1281, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/http/client.py", line 1041, in _send_output
    self.send(msg)
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/http/client.py", line 979, in send
    self.connect()
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/utilities/dockerutils.py", line 110, in docker_client
    client = docker.DockerClient.from_env()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/shanna/code/intel-graph/src/neo4j_python/prefect/deployments/malware_processor_deploy.py", line 6, in <module>
    malware_processing_flow.deploy(
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 296, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/flows.py", line 1098, in deploy
    deployment_ids = await deploy(
                     ^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/deployments/runner.py", line 980, in deploy
    image.build()
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/deployments/runner.py", line 835, in build
    build_image(**build_kwargs)
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/utilities/dockerutils.py", line 164, in build_image
    with docker_client() as client:
  File "/Users/shanna/.pyenv/versions/3.11.4/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/shanna/.pyenv/versions/3.11.4/envs/intel-graph/lib/python3.11/site-packages/prefect/utilities/dockerutils.py", line 114, in docker_client
    raise RuntimeError(
RuntimeError: This error is often thrown because Docker is not running. Please ensure Docker is running.
n
yeah this seems about not being able to get a docker client
Copy code
packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
can you do this?
Copy code
» python -c "import docker; print(docker.DockerClient.from_env())"
<docker.client.DockerClient object at 0x100950a40>
s
same error
n
right, so this has something to do with your docker, not so much the .deploy() method
s
i've tried setting DOCKER_HOST with a wide variety of options, but none of them seem to pickup my local docker
n
are you using podman or something?
s
i am not, docker desktop, and all my other docker containers from various projects work fine, etc
Copy code
shanna-mba:~ shanna$ docker ps
CONTAINER ID   IMAGE         COMMAND                  CREATED       STATUS                  PORTS                    NAMES
80ab7ffc5a95   postgres:14   "docker-entrypoint.s…"   7 weeks ago   Up 24 hours (healthy)   0.0.0.0:5432->5432/tcp   intel-graph-api-db-1
n
hm i also use docker desktop and never have to set the host maybe 
unset DOCKER_HOST
?
s
i previously didn't have it set and only looking through the code did i attempt to try it
it is currently unset
👍 1
(i am on a mac m1 for reference)
n
hmm not sure
is your docker socket file at the normal place?
Copy code
ls /var/run/ | grep docker.sock
do you need 
sudo
for other docker stuff? maybe a permissions thing somehow
s
no, i havent ever used sudo with docker
Copy code
ls /var/run/ | grep docker.sock
didnt return anything
n
! that seems related to the issue
from the docker sdk
Copy code
class DockerClient:
    """
    A client for communicating with a Docker server.

    Example:

        >>> import docker
        >>> client = docker.DockerClient(base_url='<unix://var/run/docker.sock>')

    Args:
        base_url (str): URL to the Docker server. For example,
            ``unix:///var/run/docker.sock`` or ``<tcp://127.0.0.1:1234>``.
        version (str): The version of the API to use. Set to ``auto`` to
            automatically detect the server's version. Default: ``1.35``
        timeout (int): Default timeout for API calls, in seconds.
        tls (bool or :py:class:`~docker.tls.TLSConfig`): Enable TLS. Pass
            ``True`` to enable it with default options, or pass a
            :py:class:`~docker.tls.TLSConfig` object to use custom
            configuration.
        user_agent (str): Set a custom user agent for requests to the server.
        credstore_env (dict): Override environment variables when calling the
            credential store process.
        use_ssh_client (bool): If set to `True`, an ssh connection is made
            via shelling out to the ssh client. Ensure the ssh client is
            installed and configured on the host.
        max_pool_size (int): The maximum number of connections
            to save in the pool.
    """
    def __init__(self, *args, **kwargs):
        self.api = APIClient(*args, **kwargs)
s
currently running a
Copy code
find / -name "docker.sock" 2> /dev/null
not sure if mac stores it in a different place so i'm searching everything
yeah, so it was found in
Copy code
/System/Volumes/Data/Users/shanna/.docker/run/docker.sock
n
so you could 
set DOCKER_HOST
to that - but personally i would explore reinstalling / having it at the standard place
s
i don't believe that is an option given the way this company laptop is provisioned, etc
n
ahh
🫡
s
and this still gives the same error unfortunately
Copy code
DOCKER_HOST=/System/Volumes/Data/Users/shanna/.docker/run/docker.sock  PYTHONPATH=src/ python src/neo4j_python/prefect/deployments/malware_processor_deploy.py
163 Views
Open in Slack
PreviousNext
Powered by