Thread
#prefect-community
    e

    Elio

    4 months ago
    Hi, we are using prefect Local Agent wrapped in a docker container, we would like to switch to Docker Agent. Does someone knows if it's possible to setup a Docker Agent with Docker IN Docker (dind) ? Thanks !
    Kevin Kho

    Kevin Kho

    4 months ago
    You could by mounting the sock:
    prefect agent docker start --volume //var/run/docker.sock:/var/run/docker.sock
    but this is normally not a good pattern. Why do you need to do this?
    e

    Elio

    4 months ago
    Because I don't want to install a python package on my environment and I prefer to run it in a docker container
    Anna Geller

    Anna Geller

    4 months ago
    I guess what Kevin meant is that docker-in-docker is sometimes hard to manage. Are you on Prefect Cloud or Server? on Prefect 1.0 or 2.0? if the problem you try to solve is managing code dependencies across flow runs, then it's enough if flow runs are packaged as containers, the agent itself can run as a subprocess. The Prefect agent is a lightweight process that doesn't require any custom dependencies other than Prefect and it can deploy flow runs with custom dependencies as docker containers
    e

    Elio

    4 months ago
    We are using Prefect Cloud, we are actually using prefect 1 and we can migrate to 2.0 I'm not trying to solve managing code dependencies, we just don't want anything installed outside Docker. So using Docker Agent, we don't want to pip install on our servers and would like to use it in it's own container.
    Anna Geller

    Anna Geller

    4 months ago
    we just don't want anything installed outside Docker
    As our CTO Chris White often says: "Never say just" 🙂 I understand the sentiment toward containerization but you need to keep in mind that there is no one-size-fits-all way of packaging dependencies. For Prefect Docker agent specifically, it needs to run as an independent process on your machine in order to communicate with your Docker client to spin up Docker containers for a flow run. Think of it this way: you can't install a Docker client (e.g. Docker Desktop) as a Docker container, would you agree? It must be installed and running in order to spin up dockerized workloads. You can think of the Prefect Docker agent as a similar process that needs direct access to your system in order to spin up containerized workloads. To minimize any headaches with dependency management, many Prefect users who use a Docker agent spin up the agent process within a virtual environment.
    e

    Elio

    4 months ago
    Think of it this way: you can't install a Docker client (e.g. Docker Desktop) as a Docker container, would you agree?
    For example, cadvisor (monitoring) talk to the docker daemon via the socket file. So we can run cadvisor to monitor docker containers from a cadvisor container where the docker .sock file is mounted as a volume
    I don't understand the technical problem to do this, what's the difference between talking to the socketfile from the host system and talking to the socketfile from a container where the socketfile is mounted
    Anna Geller

    Anna Geller

    4 months ago
    Maybe I'm not a Docker expert enough to explain it well but there is a difference