I was wondering if is any pattern to expose modules to be containerized in a python way. The project I'm working on has the following structure:

.my-package
├── __init__.py
├── _config.yml
├── flows
    ├── __init__.py
    ├── flow1.py
    ├── flow2.py
    └── flow3.py
└── utils.py

So I can expose my flows via storage object like this:

from prefect.storage import Module

storage = Module("my-package.flows")

Each of the flows that belong to

my-package

have the following structure:

from prefect import Parameter, task
from prefect.utilities import logging as logging

def core_function(**args)-> prefect.flow:
   # process flow
   return flow

flow = core_function()

Seem legit to me, since I haven't spotted any downside, do you have any recommended pattern or advise?

Hey @Miguel Angel, this looks good to me. You would have this in a container right?

Yes, that package was built as containerized flow. However, I got the following error during flow exec on Prefect CloudÇ

Failed to load and execute Flow's environment: ValueError("Failed to find flow 'flow1' in 'my-package.flows'")

So, what I'm trying to do is import them right away on my

flows.___init__.py_

so can be instantiated on creation.

Will put together a working example. Do you have a setup.py?

Thanks @Kevin Kho, I'm still trying to catch up with all the prefect's functionalities. We have a Dask Cluster configuration file. Is that what you mean by setup?

No. The

setup.py

to actually package the Python directory into a module. I have an example up there. You need it to install the module with

pip

I see, that's a nice approach. Our dockerfile was inspired by prefect docs where using conda env to install dependencies and running the main package.

You should be good then. Just make sure that your Flow uses the Docker container so you need to specify it by using

DockerRun

and the

Docker

agent so that when it pulls the Flow, it will run it on top of that image.

However, that worked pretty good inside of

my-package

container, Once it's deployed I could get further logs, but keeps pointing out to

Failed to load and execute Flow's environment: ValueError("Failed to find flow 'flow1' in 'my-package.flows'")

What agent are you using?

I'm using a prefect ECSAgent for that purpose.

Gotcha. Thanks for the info. You good now? Looks like you just had to specify where the Flow is stored for the agent to pick it up

Apparently It solved my problem, thanks for the assistance.