https://prefect.io
Join Slack
Hello, I'm trying to run a prefect flow locally where the flow is within a module. The path to the f...
# ask-community
i
Hello, I'm trying to run a prefect flow locally where the flow is within a module. The path to the flow file is: 
repo_name/projects/npa_project/npa_project_module/npa_flow.py
. The module is 
npa_project_module
. When i run the following in the CLI 
prefect run -m npa_project_module.npa_flow
I get an error from prefect saying: 
No module named "npa_project_module"
. This happens both when I cd to the 
npa_project
directory and the 
npa_project_module
directory before running the cmd. Any ideas what could be wrong?
k
Did you pip install the module?
i
No I didn't. It's a module I created
k
You need to still package it as a module and pip install it so that it’s importable by the Flow run. Do you know how to package the directory into a Python module? I have an example that might help
i
Kindly share an example
k
This shows an example. You can ignore the Docker stuff. There is Python packaging there as well and then you install it with 
pip install -e .
i
I'm going through that article but it's unclear to me at what point I have to run the 
pip install -e .
Also in what directory should it be run
k
in the folder that contains your module. 
pip install -e .
the 
.
means currently directory and it uses the 
setup.py
file to create the module.
pip install -e .
you can run after making the 
setup.py
i
I have done this but 
prefect run --module npa_project_module
still gives the same error
k
I think you need to give a name when you do that also with 
prefect run --module … --name …
. Can I see what your setup.py looks like?
i
#!/usr/bin/env python
# -*- coding: utf-8 -*-
""" setup.py - Install this package but not any of its dependencies
This assumes that other tools will install the dependencies from the pipfile
"""
from setuptools import setup, find_packages
setup(
name="npa_segmentation",
version="0.1.0",
description="Pipeline that splits out NPA forecast",
long_description="",
author="****** ******",
packages=find_packages(),
include_package_data=True,
zip_safe=False,
)
k
i think your module name is 
npa_segmentation
then, not 
npa_project_module
i
Yes, I have used that
Still doesn't work
k
If you have a python script with:
Copy code
import npa_segmentation
does it work?
i
yes it does
k
Does it work if you run the script from other directories?
i
Now prefect recognizes the module. I had to run the 
pip install -e .
again.
I'm now stuck with another error which says flow not found
Copy code
Found no flows at npa_segmentation.dags.npa_segmentation_predict.
Copy code
def run_flow():    
    with Flow(
        "npa-segmentation-predict-flow", storage=storage, executor=executor, schedule=dag_schedule
    ) as flow:

        prediction_window = Parameter(name="pred_window", default=30, required=False)

        pspd = pull_snowflake_data_pred(bucket_base, database, schema, prediction_window)
        prepare_predictions_table(database, schema, bucket_base, bucket_base, upstream_tasks=[pspd])

if __name__ == "__main__":
    run_flow()
k
The flow will only exist in this file if you run it as main. It won’t if you import so you need to run the 
run_flow()
i
Copy code
def run_flow():    
    with Flow(
        "npa-segmentation-predict-flow", storage=storage, executor=executor, schedule=dag_schedule
    ) as flow:

        prediction_window = Parameter(name="pred_window", default=30, required=False)

        pspd = pull_snowflake_data_pred(bucket_base, database, schema, prediction_window)
        prepare_predictions_table(database, schema, bucket_base, bucket_base, upstream_tasks=[pspd])

run_flow()
This still gives the same error
k
What storage are you adding to that Flow?
i
Copy code
from prefect.storage import Docker
storage = Docker(
  dockerfile="Dockerfile",
  registry_url=f"<http://us.gcr.io/{PROJECT}|us.gcr.io/{PROJECT}>",
  image_name="test_prefect_cloud",
k
Let me try to make a quick example here
I have an example here. You can do can put these files in a folder.
Copy code
pip install -e .
to install. This installs a package named flows.
Copy code
prefect run --module flows --name module_test2
will give:
Copy code
Running flow locally...
└── 13:47:39 | INFO    | Beginning Flow run for 'module_test2'
└── 13:47:39 | INFO    | Task 'abc': Starting task run...
└── 13:47:39 | INFO    | 2
└── 13:47:40 | INFO    | Task 'abc': Finished task run for task with final state: 'Success'
└── 13:47:40 | INFO    | Flow run SUCCESS: all reference tasks succeeded
Flow run succeeded!
You need to put the flows in the init file so they get imported along with the module
i
what does this do? 
from prefect.storage import Module
I don't the the imported Module being used in the flows
k
oh you dont need it. i had some other code i repurposed so i just didnt remove it
i
Thank you very much. The flow runs locally now.
Is there anything I should take note of when I want to run this flow on cloud?
k
Yeah a while ago you showed me your storage was Docker storage
You need to register these with Module storage
and then make sure the Module is installed on the execution environment
i
Could you please share a sample code where you used the Module storage
and I have the 
pip install -e .
in my dockerfile
k
I am not sure if it should be 
flow.storage=Module("mymodule.flows.flow1")
or 
flow.storage=Module("flows.flow1")
but I think it should be one of the two
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by