https://prefect.io logo
Title
d

Dylan Lim

02/06/2023, 3:51 AM
Hi all, how do you define
import_paths
in Orion like how you would do so in Prefect 1.0? My flows need to import functions from another folder in my source code and it’s unable to find it when I run the agent.
w

Walter Cavinaw

02/06/2023, 3:55 AM
the flow is unable to find the code in another folder? What storage block are you using and what is the flow running on (Process, Docker, KubernetesJob)?
d

Dylan Lim

02/06/2023, 4:02 AM
Thanks for pointing me to storage blocks - I haven’t implemented that yet 🤦‍♂️ I’m running the agent on a remote server using supervisord without docker/kubernetes. Would a local file system storage block work with me just defining the basepath as the folder where my flows are stored? Will try it out soon based on what’s here: https://docs.prefect.io/concepts/storage/
w

Walter Cavinaw

02/06/2023, 4:08 AM
if the code is already on the machine then you can just define the absolute path to the working directory in the deployment with the path parameter. It will use localfilesystem storage by default, I think?
d

Dylan Lim

02/07/2023, 3:29 AM
Thanks so much @Walter Cavinaw! So I tried this and it didn’t seem to work… I ran a simple flow that looks a bit like this:
import os

from prefect import flow, get_run_logger

@flow
def log_location():
    logger = get_run_logger()
    <http://logger.info|logger.info>(os.getcwd())

if __name__ == '__main__':
    log_location()
Deployed it on my machine and checked the .yaml file for the path and everything looked ok. Checked the logs and got this:
INFO: Downloading flow code from storage at '/home/dylan/prefect/flows'
INFO: /tmp/tmpiwcz1bmlprefect
The second line is the line produced by
os.getcwd()
. I’m not sure why it’s downloading flow code from the path but executing it from tmp folder - how can I change that? I think this is why my relative imports are failing… so for context I’m trying to import functions from a folder on the same level as ‘flows’. This is the bit of code in my flow code that fails:
import sys 
sys.path.append("../")
from utilities.functions import get_sql_file
My file structure looks like this:
/prefect
  /flows
    test_log.py
  /utilities
    __init__.py
    functions.py
Any help is appreciated!
w

Walter Cavinaw

02/07/2023, 3:33 AM
what does your deployment look like? you can specify Process(working_dir="<the wd you want>") as the infrastructure parameter to your deployment
I do this in code. Below is an example for one of our process flows that runs on a remote machine.
deployment = Deployment.build_from_flow(
                flow=flow_deployment['flow'],
                name=flow_deployment['name'],
                infrastructure=Process(working_dir="<censor>"),
                storage=GitHub.load('<censor>'),
                skip_upload=True,
                path="<censor>",
                work_queue_name='<censor>',
                schedule=schedule,
                apply=True,
                parameters=parameters,
            )
👀 1
d

Dylan Lim

02/07/2023, 3:47 AM
Wow it works now! Thanks so much @Walter Cavinaw appreciate it tons 🙏
1