Hello everyone ! I am a data scientist who is currently testing the Prefect library. I am a little bit stuck on using prefect flows with Docker. Do you have any example on flows configured with DockerRun and ran with a Local DockerAgent ? My flow is registered but is stuck in 'Submitted for execution'. Thanks for your help ! 🙂

register_flow.py

Kevin built a couple of examples as well: this article, or demos like this one.

docker_agent_py.py

Thanks a lot for the replies ! i'll test the scripts !

Hey, that is just an example of packaging a Python package into a Docker container. You can then run attach your image through the

DockerRun

. It looks like it got your flow. What do you see when you say it gets stuck? Any logs?

no more logs out of the container?

# in the agent
show_flow_logs=True

also maybe try using prefect to build the image ?

storage = Docker(
        registry_url=registry_url, # set to none for keeping image locally
        image_name=image_name,
        image_tag=image_tag,
        dockerfile='Dockerfile', # path to your dockerfile
    )
storage.build(push=push) # set false for not pushing the image to a registry and keep only local

@Kevin Kho: Thanks for the response ! I might be missing something but i thought that running a dockerized flow witha server backend means registering your flow with a dockerrun config. staring a local docker agent and then running the flow (using quickrun) . In my case the docker agent does get my flow but nothing happens, i might be wrong but does the docker agent start a container to run the flow using the image name ? @Andreas Tsangarides: Thanks for the -show-flow-logs !! I'm having more output, basically a connection refused :

│[2021-10-15 08:17:30,735] INFO - agent | Deploying flow run 84bb1c3f-90dc-4633-b83f-f4c057cd1
│938 to execution environment...                                                              
 │[2021-10-15 08:17:31,128] INFO - agent | Completed deployment of flow run 84bb1c3f-90dc-4633-
│b83f-f4c057cd1938                                                                            
│Traceback (most recent call last):                                                           
│  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn
│    (self._dns_host, self.port), self.timeout, **extra_kw                                    
│  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in creat
e_connection                                                                                 
│    raise err                                                                                
│  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in creat
e_connection                                                                                 
    sock.connect(sa)                                                                         
│ConnectionRefusedError: [Errno 111] Connection refused

This looks like it can’t connect to Docker. What happens when you do

docker run hello-world

in the command line on the machine you have the Docker agent on?

Everything works fine with docker run / build ... I have a group permissions for docker , do i need to provide prefect with higher permissions even though it is run with my session ?

use registry_url=None. The build should be able to complete and just store the image locally. Do you get any steps in the build or it fails immediately?

It fails immediately, by default the registry_url is set to None, here is my .py

import prefect
from prefect import task, Flow, context
from prefect.run_configs import DockerRun
from prefect.storage import Local
from prefect.storage.docker import Docker


@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello task")


# Configure storage as docker file
docker_storage = Docker(
    registry_url=None, dockerfile="Dockerfile", image_name="test:latest"
)
docker_storage.build()

with Flow(
    "docker_example",
    storage=docker_storage,
    # storage=Local(path="/app/test.py", stored_as_script=True),
    run_config=DockerRun(image="test:latest", labels=["docker"]),
) as flow:
    hello_task()


if __name__ == "__main__":
    flow.register("docker_test")

btw... what i do is i attach the storage and run config at registration... the flow.py file does not contain those... only the flow/tasks I attach the run_config and storage to the flow in a separate script where i register the flow so my dockerfile only has the flow logic if that makes sense.. • flow.py contains the (with Flow(.....) as flow) • register.py imports the flow from flow,py, attaches storage and run_config and registers the flow. So the docker image only has the flow logic

Ok thx i'll try this setup but do you think it changes from the flow.register in _main___ ?

Thanks for helping @Andreas Tsangarides! @Amine Dirhoussi to check whether the error comes from Docker client and network, or some issues with Python dependencies, you could try to run this simple flow that uses nothing but base Prefect image with no extra dependencies:

"""
prefect agent docker start --label docker
"""
from prefect import task, Flow
from prefect.run_configs import DockerRun
from prefect.storage import Docker


@task(log_stdout=True)
def hello_world():
    print("hello world")


with Flow(
    "dockerrun-test-flow",
    storage=Docker(),
    run_config=DockerRun(image="prefecthq/prefect:latest", labels=["docker"]),
) as flow:
    hello_world()

Does this flow execute successfully?

@Anna Geller thx for your help . Registering the flow worked !! I did tweak the setup a little bit though, according to the issue : https://github.com/PrefectHQ/prefect/issues/4963 . I added

prefect start server --expose

, apparently it is a Ubuntu issue. I 'll try without to check if it works without . Thanks a lot

nice work! 👍 Another thing we need to check in this setup is whether the environment from which you register your flow has the same Python and prefect version e.g. base image in Dockerfile:

FROM prefecthq/prefect:0.15.6-python3.8

if you use Python 3.8 and Prefect 0.15.6 on the machine from which you deploy your flow to ensure that the serialized flow is using the same version. LMK if there is any other issue.

I checked and i'm using the same conda env for registering the flow and for running the docker agent. I also tested without

--expose

flag and the basic flow does not work. I am still having connection issue, when trying to build my image using the

Docker storage

and a custom Dockerfile. 😕

@Amine Dirhoussi Do you have any special reason for using Server rather than Prefect Cloud? I’m asking because Server adds a lot of complexity and Cloud is much easier and free to use for the first 10000 successful task runs each month. You probably know it that when you run

prefect server start

, it spins up all components as docker containers, and if not set up properly, you may end up running docker in docker.

@Anna Geller The main reason is that i work for a very secure company and we need to deploy flows within the existing architecture. The on premise solution is the only one we have available for the team

@Amine Dirhoussi, In that case, I have great news for you 🙂 Prefect Cloud is based on a hybrid execution model that keeps your code and data private while still taking full advantage of a managed workflow orchestration service. This means that Prefect Cloud never receives your code or data. The code gets executed entirely on your infrastructure (can be on premise, as you mentioned), and orchestration happens through metadata. You can read more about it here and here.

Thanks for the infos i'll try pushing toward this direction