https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Maria

03/30/2022, 6:35 AM
Hi, I have a flow that contains a few docker related tasks (amongst many others) - PullImage, StartContainer, etc Works fine locally but proved to be tricky to run as a DockerStorage on a test vm. After some trial and error the solution I found how I can start a container from a container (in case someone else is stuck too):
flow.run_config = DockerRun(image="my_image", host_config={"binds":["/var/run/docker.sock:/var/run/docker.sock"]})
But I also need to pull an image when its not available and this step fails since I am not authenticated from inside my flow container. I can probably create a shell task that does docker login for me before image pull, but I am wondering if there are better options?
a

Anna Geller

03/30/2022, 8:32 AM
To assess whether there is a better option, could you explain why are you using those Docker tasks - is it because you have some non-Python dependencies?
If you are doing this because e.g. each step of your workflow requires different Python dependencies, you could use the flow of flows pattern as shown in this example https://discourse.prefect.io/t/can-prefect-run-each-task-in-a-different-docker-container/434
m

Maria

03/30/2022, 10:56 AM
So as a part of data workflow I need to run ML model developed by a data scientist in R. We agreed on an input and output and to me his docker is a “black box” I run to get the outcomes I need for a next step. So it is pull - start - get results. To me it makes sense to run as a set of tasks inside my main flow
So yes it is non-python dependencies but I just want to execute them natively as it was 3 very simple tasks when run locally
k

Kevin Kho

03/30/2022, 1:52 PM
Hey @Maria, wondering if you got this to work? I see you tried to mount the Docker sock but you need to attach is as a volume on the agent as well for it to apply to the Flow
m

Maria

03/30/2022, 8:33 PM
Hey @Kevin Kho yes that instruction above allows me to start ML container from a Flow container (when image available locally), I haven't made any other changes. I still don't know what to do to be able to pull an image as I need to be authenticated inside the flow container to do so.
k

Kevin Kho

03/30/2022, 8:35 PM
Ah i see what you mean what the shell task isnt a bad option
m

Maria

03/30/2022, 8:36 PM
I will try it today
@Kevin Kho hmm trying it out and it is not gonna work. I don't have docker installed inside the docker so I cannot just execute
docker login
in a ShellTask. But then I don't understand how come tasks
CreateContainer()
,
StartContainer()
, etc work fine and produce expected result but
PullImage()
doesn't? They all should talk to an outside Docker API which is bind mounted to the Flow container
I also tried mounting /.docker directory that contains auth but that didn't help :)
k

Kevin Kho

03/30/2022, 11:04 PM
Ah I see what is the specific error you get though?
m

Maria

03/30/2022, 11:07 PM
Copy code
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=[...url...]>: Internal Server Error ("Head [...url...]/manifests/latest": no basic auth credentials")
k

Kevin Kho

03/30/2022, 11:17 PM
Yeah that weird though I don’t know how to authenticate it. In general, we discourage docker in docker because of unpredictability like this. Could you use a local agent or kubernetes agent with the k8s
ReadNamespacedJob
which creates a container. Also! you can try breaking this us into flow of flows and using
StartFlowRun
and
create_flow_run
m

Maria

03/30/2022, 11:23 PM
Just so I understand flow of flows - is the idea that second flow is not using Docker Storage?
k

Kevin Kho

03/30/2022, 11:25 PM
A couple of ways but you can also do using DockerRun + ShellTask to run what you want to?
m

Maria

03/30/2022, 11:27 PM
hmm okay I will try splitting it. Thanks 🙂
5 Views