Thread
#prefect-community
    m

    Maria

    5 months ago
    Hi, I have a flow that contains a few docker related tasks (amongst many others) - PullImage, StartContainer, etc Works fine locally but proved to be tricky to run as a DockerStorage on a test vm. After some trial and error the solution I found how I can start a container from a container (in case someone else is stuck too):
    flow.run_config = DockerRun(image="my_image", host_config={"binds":["/var/run/docker.sock:/var/run/docker.sock"]})
    But I also need to pull an image when its not available and this step fails since I am not authenticated from inside my flow container. I can probably create a shell task that does docker login for me before image pull, but I am wondering if there are better options?
    Anna Geller

    Anna Geller

    5 months ago
    To assess whether there is a better option, could you explain why are you using those Docker tasks - is it because you have some non-Python dependencies?
    If you are doing this because e.g. each step of your workflow requires different Python dependencies, you could use the flow of flows pattern as shown in this example https://discourse.prefect.io/t/can-prefect-run-each-task-in-a-different-docker-container/434
    m

    Maria

    5 months ago
    So as a part of data workflow I need to run ML model developed by a data scientist in R. We agreed on an input and output and to me his docker is a “black box” I run to get the outcomes I need for a next step. So it is pull - start - get results. To me it makes sense to run as a set of tasks inside my main flow
    So yes it is non-python dependencies but I just want to execute them natively as it was 3 very simple tasks when run locally
    Kevin Kho

    Kevin Kho

    5 months ago
    Hey @Maria, wondering if you got this to work? I see you tried to mount the Docker sock but you need to attach is as a volume on the agent as well for it to apply to the Flow
    m

    Maria

    5 months ago
    Hey @Kevin Kho yes that instruction above allows me to start ML container from a Flow container (when image available locally), I haven't made any other changes. I still don't know what to do to be able to pull an image as I need to be authenticated inside the flow container to do so.
    Kevin Kho

    Kevin Kho

    5 months ago
    Ah i see what you mean what the shell task isnt a bad option
    m

    Maria

    5 months ago
    I will try it today
    @Kevin Kho hmm trying it out and it is not gonna work. I don't have docker installed inside the docker so I cannot just execute
    docker login
    in a ShellTask. But then I don't understand how come tasks
    CreateContainer()
    ,
    StartContainer()
    , etc work fine and produce expected result but
    PullImage()
    doesn't? They all should talk to an outside Docker API which is bind mounted to the Flow container
    I also tried mounting /.docker directory that contains auth but that didn't help 😃
    Kevin Kho

    Kevin Kho

    5 months ago
    Ah I see what is the specific error you get though?
    m

    Maria

    5 months ago
    docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=[...url...]>: Internal Server Error ("Head [...url...]/manifests/latest": no basic auth credentials")
    Kevin Kho

    Kevin Kho

    5 months ago
    Yeah that weird though I don’t know how to authenticate it. In general, we discourage docker in docker because of unpredictability like this. Could you use a local agent or kubernetes agent with the k8s
    ReadNamespacedJob
    which creates a container. Also! you can try breaking this us into flow of flows and using
    StartFlowRun
    and
    create_flow_run
    m

    Maria

    5 months ago
    Just so I understand flow of flows - is the idea that second flow is not using Docker Storage?
    Kevin Kho

    Kevin Kho

    5 months ago
    A couple of ways but you can also do using DockerRun + ShellTask to run what you want to?
    m

    Maria

    5 months ago
    hmm okay I will try splitting it. Thanks 🙂