Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

hey once again

when executing a `LocalRun` with `LocalExecutor` how can I do the equivalent of `sys.path.append` to the run config working dir

currently flows are working without a backend but then when I start to run on my server I get a an error with `ModuleNotFound`

currently my structure is as such
```MainProject/
|-coreLogicModule/
|-PrefectStack/
    flows/    -need access to coreLogic
    runflows.py```


You can set the LocalRun working dir. Example <https://docs.prefect.io/orchestration/flow_config/run_configs.html#localrun|here> . And then you can maybe do `sys.path.append` inside a task if you still need it?

hey <@U01QEJ9PP53> - not quite following… just updated my question

maybe you could recommend a better dir structure ?


if you could elaborate on what happens to the flow after flow registration and how that metadata is stored that would be crucial because I couldnt find that in the docs

<@U02CMGZNL8N> the easiest way to solve it would be to:
• add setup.py at the “MainProject” level, 
• this setup.py would reference the `coreLogicModule` 
• to run this, you `cd` into the MainProject dir and run: `pip install -e .`
• as long as this package is installed in the environment from which you start the agent, and from the environment from which you register your flows, this should work without modifying the path or even adding the path to the agent 
• here is an example folder structure and setup.py: <https://github.com/anna-geller/packaging-prefect-flows>
here is an example setup.py that you could use:
```from setuptools import setup, find_packages

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name="coreLogicModule",
    version='0.01', # version this as you like
    packages=find_packages(),
    install_requires=requirements
)```
LMK if you have questions to this.

```MainProject/
|-coreLogicModule/
|-PrefectStack/
    flows/    -need access to coreLogic
    runflows.py # can now use: from <http://coreLogicModule.xyz|coreLogicModule.xyz> import abc
requirements.txt
setup.py```
sharing example dir structure in case the explanation wasn’t clear

Anna’s suggestion is definitely more proper but I was thinking that if you did:
```import sys
sys.path.append("../..")

import thing_in_directory_above

@task
def abc():
   return thing_in_directory_above()

with Flow(...) as flow:
    abc()```
this will not work because the import happens during “build” time when the flow is registered, but you want it to be deferred and happen during run time when the flow is executed. The way to do this would be to use a task to defer appending the path. For example:
```@task
def abc():
   import sys
   sys.path.append("../..")
   import thing_in_directory_above
   return thing_in_directory_above()

with Flow(...) as flow:
    abc()```
and now the appending is done when the Flow is running.

Or you can just add that directory to you Python path and I think the import will resolve during execution

amazing ... its working slowly

also ... any straightforward approaches to config serialization behavior for python objs and numpy arrays so I dont get the obnoxious json dumps errors?

Think of it this way, everything that you are transmitting to Prefect happens through an API so it has to be JSONSerializeable. So you can’t pass stuff like Pandas DataFrames or Numpy Arrays through any API.