Hi all, I am trying to follow this medium post to create a dockerized flow, and I am now able to dockerize the flow and upload it to our image repository. https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a But I don't know how to register it to a docker agent? I have a docker agent running using

prefect agent docker start --show-flow-logs

I have tried to run

prefect register --project test-project --path hello_word_dockerized.py

, but the registration failed because I don't have pandas installed on my docker agent's host (and I am not intended to because I want to have isolated docker images per flows).

import prefect
from prefect import task, Flow
from prefect.run_configs import DockerRun
from prefect.storage import Local
import pandas

def get_logger():
    from prefect.utilities.logging import get_logger
    import logging
    import sys

    logger = get_logger()
    logger.setLevel('INFO')
    log_stream = logging.StreamHandler(sys.stdout)
    logger.addHandler(log_stream)

    return logger

logger = get_logger()

@task(log_stdout=True)
def hello_world():
    <http://logger.info|logger.info>("Hello, World (Dockerized)")

with Flow("Hello World Dockerized") as flow:
    hello_world()

IMAGE_NAME = "prefect-hello-world-dockerized"
IMAGE_TAG = "latest"

import os
print(os.path.basename(__file__))
flow.storage = Local(path="/opt/prefect/flows/hello_word_dockerized.py", stored_as_script=True)

flow.run_config = DockerRun(image=f"{IMAGE_NAME}:{IMAGE_TAG}", labels=["uat", "docker"])

Hey @Ray Tang, have you need to local packages installed to register because it evaluates the Python script. Something you can do though to bypass this is run the import inside a task so that’s it deferred to execution. You need to register this, and then you’ll be able to run it on the Docker agent

@Kevin Kho I can do that. But in some cases, some tasks are in a shared module, and I need to import them at the top level (which I can't defer the import), what should we do?

Then you really need the registration environment to have them installed or wrap them in a try-except

we don't want to install any package globally, because we don't want to deal with the dependency conflict messes, and we want to have the flexibility of upgrading some dependencies in some flows for POC.

Yes you do need the registration environment, unless you use

try-except

instead because

prefect register …

opens the Python file and evaluates it to create that Flow object

so in this case, will prefect register the flow correctly?

with Flow("Hello World Dockerized") as flow:
    try:
        from flow_common import hello_workld_task
        hello_workld_task()
    except:
        pass

Ah this will not work because you really need that task code to exist to register.

you could make a flow to register your flows. That way the registration inside that flow will always run in a base image that can contain your shared modules. Something like this

from prefect.utilities.storage import extract_flow_from_file

@task(checkpoint=False,log_stdout=True,max_retries=3,retry_delay=dt.timedelta(seconds=10))
def register_flow(path, project_name):
    flow = extract_flow_from_file(path)
    flow.register(project_name=project_name)

with Flow("Register Flow", run_config=DockerRun(image='your-image-here')) as flow:
    flow_path = Parameter("flow_path")
    project_name = Parameter("project_name")
    register_flow(flow_path, project_name)

Thanks @Anurag Bajpai, we are looking into this.

When you go to the UI, I think you should be able to see the registered RunConfig

I have modified @Anurag Bajpai a bit, aiming to make it a reusable flow for future deployment use. https://gist.github.com/raytangRT/3ee77bdfccac80e317e45d3c430402cd but the problem is that when call this by

python register_flow.py --project_name demo-project-dockerized/hello-world-dockerized --flow_path /opt/prefect/build/hello_world_dockerized/flows/ --image_name prefect-hello-world-dockerized:latested/flows/ --image_name aprefect-hello-world-dockerized:latest

it doesn't throw an error with a incorrect image_name

i don't think flow.run() uses the flow's run_config by default - @Kevin Kho can correct me if i'm wrong.

Yes that’s right.

flow.run()

does not respect RunConfig. Only agent backed runs do. But I think Ray is registering?