Mike Lev
11/04/2021, 2:59 PMLocalRun
with LocalExecutor
how can I do the equivalent of sys.path.append
to the run config working dir
currently flows are working without a backend but then when I start to run on my server I get a an error with ModuleNotFound
currently my structure is as such
MainProject/
|-coreLogicModule/
|-PrefectStack/
flows/ -need access to coreLogic
runflows.py
Kevin Kho
11/04/2021, 3:00 PMsys.path.append
inside a task if you still need it?Mike Lev
11/04/2021, 3:10 PMAnna Geller
11/04/2021, 3:29 PMcoreLogicModule
• to run this, you cd
into the MainProject dir and run: pip install -e .
• as long as this package is installed in the environment from which you start the agent, and from the environment from which you register your flows, this should work without modifying the path or even adding the path to the agent
• here is an example folder structure and setup.py: https://github.com/anna-geller/packaging-prefect-flows
here is an example setup.py that you could use:
from setuptools import setup, find_packages
with open('requirements.txt') as f:
requirements = f.read().splitlines()
setup(
name="coreLogicModule",
version='0.01', # version this as you like
packages=find_packages(),
install_requires=requirements
)
LMK if you have questions to this.MainProject/
|-coreLogicModule/
|-PrefectStack/
flows/ -need access to coreLogic
runflows.py # can now use: from <http://coreLogicModule.xyz|coreLogicModule.xyz> import abc
requirements.txt
setup.py
sharing example dir structure in case the explanation wasn’t clearKevin Kho
11/04/2021, 4:05 PMimport sys
sys.path.append("../..")
import thing_in_directory_above
@task
def abc():
return thing_in_directory_above()
with Flow(...) as flow:
abc()
this will not work because the import happens during “build” time when the flow is registered, but you want it to be deferred and happen during run time when the flow is executed. The way to do this would be to use a task to defer appending the path. For example:
@task
def abc():
import sys
sys.path.append("../..")
import thing_in_directory_above
return thing_in_directory_above()
with Flow(...) as flow:
abc()
and now the appending is done when the Flow is running.Mike Lev
11/04/2021, 5:12 PMKevin Kho
11/04/2021, 5:15 PM