https://prefect.io logo
c

Chris Vrooman

06/11/2020, 7:03 PM
Has anyone had success with running prefect docker agents within another docker image? (docker within docker). I’ve been able to do this just fine using an older docker daemon (v 1.13 on Centos) launching the container with the --privileged flag. But, I can’t seem to get this to work on a newer daemon version (1.40). [2020-06-10 230259,073] INFO - agent | Successfully pulled image <ecr_image_name>... [2020-06-10 230259,180] ERROR - agent | Logging platform error for flow run ee3bb337-ad78-466a-a774-6e67f467aedc [2020-06-10 230259,500] ERROR - agent | Error while deploying flow: APIError(HTTPError(’400 Client Error: Bad Request for url: http+docker://localhost/v1.40/containers/create'))
👀 1
l

Laura Lorenz (she/her)

06/11/2020, 7:20 PM
Hi! I’m digging around a bit in the source code and the docker docs, the latter since it seems like its the docker API that doesn’t like something along the way. Out of curiosity, what makes you want to deploy docker-in-docker? I see that around sometimes
Can you get the version of the python package
docker
that is in your docker container you are trying to run the agent in?
c

Chris Vrooman

06/11/2020, 7:34 PM
Thanks Laura! Basically, for the organization of agents. We’ve been deploying multiple prefect agents on the same machine with docker. The servers we’re using a have a lot of resources and also have other important jobs running that I don’t want to interfere with. We’ve been building flows with the docker storage type in prep for using fargate/kubernetes in the future.
yep, let me take a look
👍 1
docker==4.2.1 for the docker container running the agent
l

Laura Lorenz (she/her)

06/11/2020, 7:42 PM
Gotcha thanks! fwiw that is the latest for the python sdk so I would expect it shouldn’t have any funky business with the newest daemon version…. I’ll try to reproduce
c

Chris Vrooman

06/11/2020, 7:51 PM
Thanks Laura! for reference - I’m launching that first container with this - “docker run -v /var/run/docker.sock:/var/run/docker.sock --privileged $image”
which launches the prefect agent via - “prefect agent start docker --show-flow-logs -l $affinity_label -t <token>” in the entrypoint
l

Laura Lorenz (she/her)

06/11/2020, 8:07 PM
I’m having a pretty hard time getting docker-in-docker setup going haha even with your tips 😅 If you don’t mind can you open an issue so we can get some more eyes on it? In addition we are curious if you can run the following in your docker container (the one you are trying to start an agent in), for example just in a python shell, to see if we can isolate the problem to the docker client or something else:
Copy code
import docker
c = docker.APIClient(version="auto")
container = c.create_container("their_image", command="echo 1")
c.start(container=container.get("Id"))
This would be just to check if in your docker container, can the docker client create a container under normal/vanilla conditions, without all the extra stuff we tack on when we start a flow
c

Chris Vrooman

06/11/2020, 8:34 PM
Ah, that may have pointed me to the real error, thank you! The container launches and then shuts down a few seconds later. I’ll follow up once I can narrow this down a bit.
marvin 1
Heyy laura, I was able to deploy another container okay with that script. I wasn’t testing with the right docker image initially. But, I did log an issue for this - https://github.com/PrefectHQ/prefect/issues/2781
❤️ 1