Hi All, is anyone using Module storage option?

Milly gupta

04/19/2021, 1:24 PM

Hi All, is anyone using Module storage option?

Kevin Kho

04/19/2021, 1:28 PM

Hi @Milly gupta! What question do you have about model storage?

Kevin Kho

04/19/2021, 1:30 PM

I read you other thread asking for a working example. I provided one here

Milly gupta

04/19/2021, 1:33 PM

Thanks @Kevin Kho. I am looking for working examples as documentation doesn't explain much

Milly gupta

04/19/2021, 1:48 PM

Copy code

So I have created test_module.py from prefect import task, Flow
from prefect.storage import Module
@task
def say_hello():
    print("hello world")
flow = Flow("hello-flow")
flow.storage = Module("test_module")
flow.storage.add_flow(flow)
flow.add_task(say_hello)

Milly gupta

04/19/2021, 1:48 PM

and a separate script to register

Milly gupta

04/19/2021, 1:49 PM

Copy code

from test_module import flow
flow.run_config = LocalRun(labels=["mg-test"])
flow.register(project_name="CDP")

Milly gupta

04/19/2021, 1:49 PM

So this work but wondering if it's the right way

Florian Kühnlenz

04/19/2021, 1:52 PM

biggest problem I ran into was that the module storage needs the code to be installed as a package in the environment.

Kevin Kho

04/19/2021, 1:52 PM

I believe this will work but just a couple of comments. First is that while this is right, we recommend the functional API.

with Flow("") as flow:

and then putting the tasks underneath. Second is that you can just have

flow.register

at the end of the file, but I think for ModuleStorage, it normally makes sense to have Flows inside a folder and register those flows all at once with the command line like in the previous linked thread. I believe this will all work though.

Kevin Kho

04/19/2021, 1:52 PM

Thank you for the reminder @Florian Kühnlenz!

Kevin Kho

04/19/2021, 1:53 PM

@Marvin open “Docs around ModuleStorage Need More Details on Usage”

Marvin

04/19/2021, 1:53 PM

https://github.com/PrefectHQ/prefect/issues/4415

Milly gupta

04/19/2021, 1:58 PM

yeah thanks @Florian Kühnlenz. That's the intention to use packages eventually

Milly gupta

04/19/2021, 1:58 PM

@Kevin Kho Yeah we use functional API, this is just a test example to understand module storage

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.