Rio McMahon
02/23/2022, 11:23 PMsrc/
directory) what is the best way to import it? I tried following similar logic to this: https://docs.prefect.io/orchestration/flow_config/storage.html#loading-additional-files-with-git-storage but adding to the import path:
import pathlib, sys
file_path = pathlib.Path(__file__).resolve().parent
sys.path.append(file_path)
But keep getting this error:
[23 February 2022 4:22pm]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'src'")
Is there a best practice for importing external python code into a flow?Kevin Kho
02/23/2022, 11:23 PMRio McMahon
02/23/2022, 11:26 PM# general prefect imports
import prefect
from prefect import task, Flow
from prefect.storage import Git
from prefect.run_configs import ECSRun
from prefect.client import Secret
# specific imports to load files from src/
import pathlib, sys
file_path = pathlib.Path(__file__).resolve().parent
sys.path.append(file_path)
from src.seasonality_index_builder_dynamic_agg import run_seasonality_index_builder_dynamic_agg
# define a wrapper task to expose logging
@task(log_stdout=True, checkpoint=False)
def run_script():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Running script...")
run_seasonality_index_builder_dynamic_agg()
# instantiate the flow - we store the flow definition in gitlab
with Flow("seasonality_index_builder",
storage=Git(
[git info]
),
run_config=ECSRun(
[ECS stuff]
)
) as flow:
run_script()
# Register the flow under the "tutorial" project
flow.register(project_name="Testing",
labels=['ds']
)
Kevin Kho
02/23/2022, 11:28 PMRio McMahon
02/23/2022, 11:47 PMCOPY src /home/mambauser/src
in the dockerfile, then
import pathlib, sys, os
sys.path.append(pathlib.Path(os.environ["HOME"]).resolve())
in the flow. In this case os.environ["HOME"]
should resolve to /home/mambauser
Kevin Kho
02/23/2022, 11:48 PMsrc
as a Python package so it’s accessible wherever the Flow runs. Are you familiar with how to do that?Rio McMahon
02/23/2022, 11:50 PMsrc
into a module then install via pip within my docker container?Kevin Kho
02/23/2022, 11:51 PMsetup.py
?Rio McMahon
02/23/2022, 11:54 PMKevin Kho
02/23/2022, 11:58 PM