Hi there, I am trying to deploy a flow and I am running into some issues... I have some complex dependencies and so want to use a custom base docker image. I've tried to create a simple example illustrating what I want to do. - In this example, I have a base docker image which has sqlalchemy installed (this is a placeholder for my own library). - I always develop inside docker containers and don’t want to install this library locally. This is also useful so I can easily deploy in CI. I’ve tried two approaches: - Running the main.py script on my host (but this obviously fails due to an import error). - Use the base image to run main.py, mounting the docker socket. This fails with the message “ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included”, even though in the output I can see that all the health checks pass. A few other thoughts I had: - Perhaps I could just move the imports inside the task functions - but this doesn’t seem nice to me. - Perhaps I should use a docker task instead - but this seems overkill and i believe this is the exact use case of the base image. Dockerfile:

FROM python:3.8
RUN python -m pip install sqlalchemy prefect

main.py

from prefect import Flow, task
from prefect.environments.storage import Docker
import sqlalchemy


@task
def some_task():
    print(sqlalchemy.__version__)


with Flow("Some Flow") as flow:
    some_task()

if __name__ == "__main__":
    flow.storage = Docker(
        registry_url="<http://docker.io|docker.io>",
        image_name="storage_image",
        image_tag="latest",
        base_image="test-base",
        local_image=True,
    )
    flow.register(project_name="Development")

Test script:

#!/usr/bin/env bash
set -x

docker build . -t test-base

#: Approach 1
python main.py

#: Approach 2
docker run -v /var/run/docker.sock:/var/run/docker.sock -v ~/.docker:/root/.docker -e PREFECT__CLOUD__AUTH_TOKEN="$PREFECT__CLOUD__AUTH_TOKEN" -v $(pwd)/main.py:/main.py test-base bash -c 'python /main.py'

Please could you guide me with the best approach? Many thanks!

Hey @Sam Luen-English, I’m going to get some more insight for best practices here.

Thanks @Kyle Moon-Wright 👍

Yep, and I think I can relay the consensus on this. Essentially, yes you can but it is not recommended and may introduce further complexity - you can certainly import a library within a task or utilize some deferred import. However, this docker-in-docker pattern can be needlessly problematic, for many issues unrelated to Prefect. Our recommendation is to run/register your script on your host machine with your dependencies installed locally and avoid developing within Docker. Of course, if anyone else has opinions on this topic, I’d love to hear further insight/experiences/recommendations!

This is interesting and a problem we’re working through too. Adds a lot of complexity if you intend to run multiple flows with different dependencies within the same host/execution layer.

For us the install-deps-on-host pattern doesn't work well since we're running in a DaskKubernetesEnvironment - we looked at ways of scripting worker setup but that wasn't obvious how to do well, so we've been doing the Docker-on-Docker approach. Not a fan, but it is working.

Thank you all for you thoughts. I have implemented the docker-on-docker approach for now. I think it would be great to have a solution to this in Prefect. It seems preferable to develop in the same container as will run the flow; and as @Walt Wells pointed out, it can get messy if you are developing multiple flows with all dependencies on your host.

@Sam Luen-English - Hi - it seems that we have a similar restriction and cannot follow install-deps-on-host pattern. Would you please refer to the docker-on-docker solution or share details. Thanks in advance.

Hi @Joachim Zaspel - sorry I missed this, here is my solution in summary if it useful to you (or anyone else in future)..

version: "3"

services:
  prefect:
    build: .
    image: my-project-base
    env_file:
      - docker.env
    volumes:
      - ./project-dir/:/project-dir
      - /var/run/docker.sock:/var/run/docker.sock
      - ~/.docker:/root/.docker

So my image is built and tagged as "my-project-base"

flow.storage = Docker(
        registry_url="my-registry",
        image_name="my-project",
        base_image="my-project-base",
        local_image=True,
    )

The above code is contained in the "project-dir" shared with the container, and run from within the container. I think that is all the logic required. Let me know if you have any issues :)