https://prefect.io logo
Title
m

Mike Lev

11/04/2021, 2:59 PM
hey once again when executing a
LocalRun
with
LocalExecutor
how can I do the equivalent of
sys.path.append
to the run config working dir currently flows are working without a backend but then when I start to run on my server I get a an error with
ModuleNotFound
currently my structure is as such
MainProject/
|-coreLogicModule/
|-PrefectStack/
    flows/    -need access to coreLogic
    runflows.py
k

Kevin Kho

11/04/2021, 3:00 PM
You can set the LocalRun working dir. Example here . And then you can maybe do
sys.path.append
inside a task if you still need it?
m

Mike Lev

11/04/2021, 3:10 PM
hey @Kevin Kho - not quite following… just updated my question maybe you could recommend a better dir structure ?
if you could elaborate on what happens to the flow after flow registration and how that metadata is stored that would be crucial because I couldnt find that in the docs
a

Anna Geller

11/04/2021, 3:29 PM
@Mike Lev the easiest way to solve it would be to: • add setup.py at the “MainProject” level, • this setup.py would reference the
coreLogicModule
• to run this, you
cd
into the MainProject dir and run:
pip install -e .
• as long as this package is installed in the environment from which you start the agent, and from the environment from which you register your flows, this should work without modifying the path or even adding the path to the agent • here is an example folder structure and setup.py: https://github.com/anna-geller/packaging-prefect-flows here is an example setup.py that you could use:
from setuptools import setup, find_packages

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name="coreLogicModule",
    version='0.01', # version this as you like
    packages=find_packages(),
    install_requires=requirements
)
LMK if you have questions to this.
👍 1
MainProject/
|-coreLogicModule/
|-PrefectStack/
    flows/    -need access to coreLogic
    runflows.py # can now use: from <http://coreLogicModule.xyz|coreLogicModule.xyz> import abc
requirements.txt
setup.py
sharing example dir structure in case the explanation wasn’t clear
👍 1
k

Kevin Kho

11/04/2021, 4:05 PM
Anna’s suggestion is definitely more proper but I was thinking that if you did:
import sys
sys.path.append("../..")

import thing_in_directory_above

@task
def abc():
   return thing_in_directory_above()

with Flow(...) as flow:
    abc()
this will not work because the import happens during “build” time when the flow is registered, but you want it to be deferred and happen during run time when the flow is executed. The way to do this would be to use a task to defer appending the path. For example:
@task
def abc():
   import sys
   sys.path.append("../..")
   import thing_in_directory_above
   return thing_in_directory_above()

with Flow(...) as flow:
    abc()
and now the appending is done when the Flow is running.
👍 1
Or you can just add that directory to you Python path and I think the import will resolve during execution
👍 1
m

Mike Lev

11/04/2021, 5:12 PM
amazing ... its working slowly also ... any straightforward approaches to config serialization behavior for python objs and numpy arrays so I dont get the obnoxious json dumps errors?
k

Kevin Kho

11/04/2021, 5:15 PM
Think of it this way, everything that you are transmitting to Prefect happens through an API so it has to be JSONSerializeable. So you can’t pass stuff like Pandas DataFrames or Numpy Arrays through any API.
Registering happens through an API call