Has anyone had success with running prefect docker agents within another docker image? (docker within docker). I’ve been able to do this just fine using an older docker daemon (v 1.13 on Centos) launching the container with the --privileged flag. But, I can’t seem to get this to work on a newer daemon version (1.40). [2020-06-10 23:02:59,073] INFO - agent | Successfully pulled image <ecr_image_name>... [2020-06-10 23:02:59,180] ERROR - agent | Logging platform error for flow run ee3bb337-ad78-466a-a774-6e67f467aedc [2020-06-10 23:02:59,500] ERROR - agent | Error while deploying flow: APIError(HTTPError(’400 Client Error: Bad Request for url: http+docker://localhost/v1.40/containers/create'))

Hi! I’m digging around a bit in the source code and the docker docs, the latter since it seems like its the docker API that doesn’t like something along the way. Out of curiosity, what makes you want to deploy docker-in-docker? I see that around sometimes

Thanks Laura! Basically, for the organization of agents. We’ve been deploying multiple prefect agents on the same machine with docker. The servers we’re using a have a lot of resources and also have other important jobs running that I don’t want to interfere with. We’ve been building flows with the docker storage type in prep for using fargate/kubernetes in the future.

Gotcha thanks! fwiw that is the latest for the python sdk so I would expect it shouldn’t have any funky business with the newest daemon version…. I’ll try to reproduce

Thanks Laura! for reference - I’m launching that first container with this - “docker run -v /var/run/docker.sock:/var/run/docker.sock --privileged $image”

I’m having a pretty hard time getting docker-in-docker setup going haha even with your tips 😅 If you don’t mind can you open an issue so we can get some more eyes on it? In addition we are curious if you can run the following in your docker container (the one you are trying to start an agent in), for example just in a python shell, to see if we can isolate the problem to the docker client or something else:

import docker
c = docker.APIClient(version="auto")
container = c.create_container("their_image", command="echo 1")
c.start(container=container.get("Id"))

This would be just to check if in your docker container, can the docker client create a container under normal/vanilla conditions, without all the extra stuff we tack on when we start a flow

Ah, that may have pointed me to the real error, thank you! The container launches and then shuts down a few seconds later. I’ll follow up once I can narrow this down a bit.