Has anyone managed to get Prefect installed into a Docker co

Question

Has anyone managed to get Prefect installed into a Docker container for the purpose of building Flows into Docker I keep running around in circles with the D in D problem my brain is creaking

Scott Zelenka · Answer

d in d is always a headache but this pattern works for me on debian based images ```docker run v $PWD $PWD w $PWD v var run docker sock var run docker sock IMAGE NAME command``` By mapping the $PWD volume and docker sock it allows the Docker instance inside the container to behave as if it was running natively on the Docker host

Tyler Wanner · Answer

if you re relying on the DOCKER HOST variable to use a remote docker daemon there s a fix for that coming that I believe is unreleased yet

Darragh · Answer

Thanks for that guys should have been clearer the container that I m installing prefect to is running inside a GitLab container and whileI can specify it should use the dind service I can t get it to actually build the flow into docker Latest issue is the following File usr lib python3 8 site packages docker transport unixconn py line 43 in connect sock connect self unix socket <https gitlab com cygnus data data factory orchestration jobs 550641513 L385|385> FileNotFoundError Errno 2 No such file or directory

Darragh · Answer

So I think I m doing something fundamentally ass ways

Tyler Wanner · Answer

I believe this could be fixed by that change we made for DOCKER HOST

Darragh · Answer

I really hope so cos I d love to stop beating my head off it joy

Tyler Wanner · Answer

i m not sure what i can specify it should use the dind service means though

Tyler Wanner · Answer

does that mean you re enabling the container to have access to a remote docker daemon

Darragh · Answer

Ah apologies yeah let me grab a link that will explain it better than I can

Tyler Wanner · Answer

if you could link me to this feature you re enabling in gitlab i could look further into it

Tyler Wanner · Answer

ah yes perfect thanks

Darragh · Answer

Tyler Wanner · Answer

in this container are you able to start any other docker images

Tyler Wanner · Answer

are you running a docker gitlab container that is spinning up a prefect container that is then building your flow into another docker image

Darragh · Answer

Basically yeah my next attempt was to see if I could run any docker commands inside the build CI CD container but in theory I know that using the instructiosn from that link above does allow for building docker images inside a docker container

Tyler Wanner · Answer

can i see your config

Tyler Wanner · Answer

if you re using a similar configuration to those docs it would appear you should be using that container to spin up a prefect container with a mounted docker daemon

Tyler Wanner · Answer

v var run docker sock var run docker sock

Darragh · Answer

Interesting I didn t think I had to do that you want the config for my Gitlab job

Darragh · Answer

Job Description

Tyler Wanner · Answer

ah ok ok sorry you re doing it a different way than I assumed that is helpful context thank you

Darragh · Answer

Yeah I had a feeling we were looking from different points slightly smiling face AFAIK running docker builds inside the type of job config I supplied there should work But for this specific problem I can t get it to work

Tyler Wanner · Answer

I m back to my original theory that this is related to DOCKER HOST

Darragh · Answer

Could well be rigth running a quick test to see if I can do ANY docker build stuff

Darragh · Answer

Progress at last The following job config absolute barebones docker build works

Darragh · Answer

Untitled

Tyler Wanner · Answer

can you echo out $DOCKER HOST

Tyler Wanner · Answer

if i m right i should have a workaround for you

Darragh · Answer

echo $DOCKER HOST <tcp docker 2375>

Tyler Wanner · Answer

yep

Tyler Wanner · Answer

we re right

Tyler Wanner · Answer

so here are two things you can do

Tyler Wanner · Answer

1 try running this with a pip install of prefect master this will give you the fix which hopefully works to resolve this issue but 0 11 0 is not ready for production yet 2 set base url in your docker storage to the DOCKER HOST env var

Tyler Wanner · Answer

`base url = os environ get DOCKER HOST None ` should do the trick in your docker storage

Tyler Wanner · Answer

btw that s an either or you shouldn t need to do both

Darragh · Answer

Ok cool I ll try that now both items or just one

Darragh · Answer

Ah great minds slightly smiling face

Darragh · Answer

Not to sound stupid but is there a pre release I need to enable to get that master `pip install prefect master` some other syntax

Tyler Wanner · Answer

good question

Tyler Wanner · Answer

`pip install git+<https github com PrefectHQ prefect git master>`

Tyler Wanner · Answer

i believe this should do the trick

Darragh · Answer

Option 1 works builds like a charm Many thanks to you Tyler

Darragh · Answer

I m having one further problem where it s trying to find `graphql = $ cloud api graphql alpha ` and getting an empty value after some variable subsititon but I think that s my problem

Darragh · Answer

Thanks again

Tyler Wanner · Answer

awesome

Tyler Wanner · Answer

is that a problem i can also help with

Darragh · Answer

I ll dig into it some more and get back to you grin I did environment variable substitution to get the config toml to register with a prefect I stance I deployed to aws and the rest of the substitutions seem to have worked just need to see why this specific one didn t work +1

Darragh · Answer

Worked around the graphql problem something weird going on with the variable substituion New problem though As far as I know once my config toml is pointed at the right server my instance on AWS when it builds and registers I should see it in the UI right

Darragh · Answer

Untitled

Darragh · Answer

And probably related I get this as output from the build `Flow flow 8e6b2c22 f8a2 4de1 a98f 38e998ba1bfc`

Tyler Wanner · Answer

if you register a flow to Server then yes you should see a Flow in the Server UI

Darragh · Answer

Yeah that s what I see when I register it locally Guessing it didn t register onto the remote one then

Tyler Wanner · Answer

do you have a cloud account you might have accidentally registered to

Tyler Wanner · Answer

or another server a local server

Darragh · Answer

Nope nothing accessible from there Curious

Darragh · Answer

I did a local build register and THAT one shows up on the remove server So the server is running and accepting flows which is good

Tyler Wanner · Answer

is your UI local

Darragh · Answer

No

Tyler Wanner · Answer

ah ok cool

Tyler Wanner · Answer

so maybe your CI container s toml is incorrect and is trying to publish to a local server it is running

Darragh · Answer

Any idea why I get that funky looking flow output above The one missing the first part of the URL

Darragh · Answer

I d guess something like that yea

Tyler Wanner · Answer

that would appear to be the location of the published flow

Tyler Wanner · Answer

since you used a local env i think it s saving to that dir

Darragh · Answer

But using that same flow and building locally it registers to the remote service and I get a usable url for it I think I probably have an issue in my toml as you mentioned I ll go away and look again but it ll be tomorrow cos it s late here slightly smiling face Thanks again for all your help

Tyler Wanner · Answer

yes exactly

Tyler Wanner · Answer

i ll be here if you need me

Tyler Wanner · Answer

thanks for ur time today

Darragh · Answer

Nah man thanks for yours you helped me out of a big freakin hole joy