Hi, I have a flow that contains a few docker related tasks (amongst many others) - PullImage, StartContainer, etc Works fine locally but proved to be tricky to run as a DockerStorage on a test vm. After some trial and error the solution I found how I can start a container from a container (in case someone else is stuck too):

flow.run_config = DockerRun(image="my_image", host_config={"binds":["/var/run/docker.sock:/var/run/docker.sock"]})

But I also need to pull an image when its not available and this step fails since I am not authenticated from inside my flow container. I can probably create a shell task that does docker login for me before image pull, but I am wondering if there are better options?

To assess whether there is a better option, could you explain why are you using those Docker tasks - is it because you have some non-Python dependencies?

So as a part of data workflow I need to run ML model developed by a data scientist in R. We agreed on an input and output and to me his docker is a “black box” I run to get the outcomes I need for a next step. So it is pull - start - get results. To me it makes sense to run as a set of tasks inside my main flow

Hey @Maria, wondering if you got this to work? I see you tried to mount the Docker sock but you need to attach is as a volume on the agent as well for it to apply to the Flow

Hey @Kevin Kho yes that instruction above allows me to start ML container from a Flow container (when image available locally), I haven't made any other changes. I still don't know what to do to be able to pull an image as I need to be authenticated inside the flow container to do so.

Ah i see what you mean what the shell task isnt a bad option

I will try it today

Ah I see what is the specific error you get though?

Yeah that weird though I don’t know how to authenticate it. In general, we discourage docker in docker because of unpredictability like this. Could you use a local agent or kubernetes agent with the k8s

ReadNamespacedJob

which creates a container. Also! you can try breaking this us into flow of flows and using

StartFlowRun

and

create_flow_run

Just so I understand flow of flows - is the idea that second flow is not using Docker Storage?

A couple of ways but you can also do using DockerRun + ShellTask to run what you want to?

hmm okay I will try splitting it. Thanks 🙂