John Mizerany
09/07/2022, 10:03 PMsys.path.append
to include the module in our PYTHONPATH but that did not work. We are using Git Remote storage but it seems the agent we are using is not able to pick up on the custom files/modules we wrote in the subdirectory (we are still using prefect cloud 1.0 and the UI when we create a run gives us Failed to load and execute flow run: ModuleNotFoundError
)Nate
09/07/2022, 10:16 PMJohn Mizerany
09/07/2022, 10:27 PMNate
09/07/2022, 10:49 PMJohn Mizerany
09/07/2022, 11:07 PMNate
09/07/2022, 11:15 PMJohn Mizerany
09/07/2022, 11:18 PMNate
09/07/2022, 11:27 PMJohn Mizerany
09/07/2022, 11:33 PMimport sys
sys.path.append("./helpers")
import helpers.aws as aws
import helpers.helpers as helpers
import helpers.snowflake_helpers as snowflake_helpers
And the folder structure looks like this
folder/
|_ flow_example.py
|_ helpers/
|_ __init__.py
|_ helpers.py
|_ aws.py
|_ snowflake_helpers.py
So it seems that it is messing up on that first helpers.aws line where I am importingNate
09/08/2022, 3:47 PMsys.path.append("./helpers")
will work, since this code will run at registration time and it seems you'd be appending your local path to your EC2's sys path
I would try something more like the example you linked, where you get the file_path = Path(__file__).resolve().parent
which will update according to where it was loaded from, and then you should be* able to add files from that file_path
to your system path
(* I'm not positive whether this works. I'm currently on vacation so I would have to try and reproduce myself once I'm back - feel free to send a message from this thread to the channel if you're still stuck - but again you can always install your helpers where you run the agent or use docker to include your deps via Dockerfile for a DockerRun)John Mizerany
09/08/2022, 3:48 PM