Has anyone managed to get Prefect installed into a Docker container for the purpose of building Flows into Docker? I keep running around in circles with the D-in-D problem, my brain is creaking…

d-in-d is always a headache, but this pattern works for me (on debian based images):

docker run -v $PWD:$PWD -w $PWD -v /var/run/docker.sock:/var/run/docker.sock IMAGE_NAME command

By mapping the $PWD volume, and docker.sock, it allows the Docker instance inside the container to behave as if it was running natively on the Docker host.

if you’re relying on the DOCKER_HOST variable to use a remote docker daemon, there’s a fix for that coming that I believe is unreleased yet

Thanks for that guys, should have been clearer - the container that I’m installing prefect to is running inside a GitLab container, and whileI can specify it should use the dind service, I can’t get it to actually build the flow into docker. Latest issue is the following: File “/usr/lib/python3.8/site-packages/docker/transport/unixconn.py”, line 43, in connect sock.connect(self.unix_socket) 385 FileNotFoundError: [Errno 2] No such file or directory

I believe this could be fixed by that change we made for DOCKER_HOST

I really hope so cos I’d love to stop beating my head off it 😂

i’m not sure what “i can specify it should use the dind service” means, though

Ah apologies, yeah let me grab a link that will explain it better than I can…

if you could link me to this feature you’re enabling in gitlab, i could look further into it

in this container, are you able to start any other docker images?

Basically yeah…. my next attempt was to see if I could run any docker commands inside the build/CI/CD container, but in theory I know that using the instructiosn from that link above does allow for building docker images inside a docker container

can i see your config?

Interesting, I didn’t think I had to do that…. you want the config for my Gitlab job?

ah ok ok sorry, you’re doing it a different way than I assumed--that is helpful context, thank you

Yeah I had a feeling we were looking from different points 🙂 AFAIK, running docker builds inside the type of job config I supplied there should work. But for this specific problem I can’t get it to work…

I’m back to my original theory that this is related to DOCKER_HOST

Could well be rigth… running a quick test to see if I can do ANY docker build stuff…

can you echo out $DOCKER_HOST?

echo $DOCKER_HOST tcp://docker:2375

yep!

Ok cool! I’ll try that now - both items or just one?

good question

Option 1 works, builds like a charm!! Many thanks to you Tyler

awesome!

I'll dig into it some more and get back to you 😁 I did environment variable substitution to get the config.toml to register with a prefect I stance I deployed to aws, and the rest of the substitutions seem to have worked, just need to see why this specific one didn't work 👍

if you register a flow to Server, then yes you should see a Flow in the Server UI

Yeah that’s what I see when I register it locally. Guessing it didn’t register onto the remote one then…

do you have a cloud account you might have accidentally registered to?

Nope, nothing accessible from there. Curious!

is your UI local?

No

ah ok cool

Any idea why I get that funky looking flow output above? The one missing the first part of the URL?

that would appear to be the location of the published flow

But using that same flow and building locally, it registers to the remote service, and I get a usable url for it.. I think I probably have an issue in my toml, as you mentioned. I’ll go away and look again, but it’ll be tomorrow cos it’s late here 🙂 Thanks again for all your help!

yes exactly

Nah man, thanks for yours, you helped me out of a big freakin hole 😂