Hey guys, can someone please point me in the direc...
# ask-communityj
Joseph Loss
04/05/2021, 6:54 PMHey guys, can someone please point me in the direction of importing user-created functions from other python files? We have a large library of common functions that are used in almost every script, I'm developing a use-case for using Prefect over VisualCron but I'm running into some issues here
k
Kevin Kho
04/05/2021, 6:56 PM
I assume this works when running on local but gives an error when running elsewhere?
j
Joseph Loss
04/05/2021, 6:58 PMyessir, we use sys.path.appends like so:
Copy code
sys.path.append('D:\\_Common\\python')
from fnLibrary import fnODBConnect, setLoggingk
Kevin Kho
04/05/2021, 7:01 PM
I think the best practice here is to package that library with the common tools and include that in a Docker image. If you package it as a wheel, you can pip install it. and then import
fnLibraryj
Joseph Loss
04/05/2021, 7:02 PMI have used celery automation with these type of imports in the past, but perhaps due to running Prefect Cloud, I need to pass in the imported module metadata somehow.
k
Kevin Kho
04/05/2021, 7:04 PM
What storage and run config are you using for Prefect?
Kevin Kho
04/05/2021, 7:06 PM
The local run works because it sees those files on the local machine, but you need the agent to be able to see the files as well when it runs it.
Kevin Kho
04/05/2021, 7:07 PM
This link may be helpful for you: https://docs.prefect.io/orchestration/recipes/configuring_storage.html#including-other-python-scripts
j
Joseph Loss
04/05/2021, 7:10 PMRight now, I'm using the Prefect python package on a Windows Server 2019 VM.
Believe me, I would much rather use the docker image, but at present our scripts are run from python venvs (trust me, this is alarming).
I did manage to hackathon my way into using Docker in process-isolation mode, as enabling nested virtualization is too risky for the financial market work until I'm able to bug test it as much as possible
Joseph Loss
04/05/2021, 7:12 PMand at present, we're only able to run docker in windows containers (missing out on 80% of the powerful docker images built in linux containers). This is due to WSL2 not being supported and/or broken in WS2019 at the moment 😕
Joseph Loss
04/05/2021, 7:12 PMI'll look into the docker storage docs and try to get something going using Windows containers though
k
Kevin Kho
04/05/2021, 7:15 PM
Are the flows that will be orchestrated by Prefect going to run on the same Windows Server 2019 VM?
j
Joseph Loss
04/05/2021, 7:16 PMyes, and everything that those flows need to run (environment, filepaths for output, imported modules) are all located on the same drive within that VM
k
Kevin Kho
04/05/2021, 7:19 PM
To circumvent Docker, you can package that library and install it wherever your flows will be running. There are just a couple of stuff to keep track of (making sure the latest version is installed). Versioning that set of helper scripts inside the module would probably be the way to go.
j
Joseph Loss
04/05/2021, 8:23 PMwell I'll be damned, this worked BEAUTIFULLY Kevin!! Thank you so much!
k
Kevin Kho
04/05/2021, 8:28 PM
Didn’t do much here. Happy you figured it out! 🙂
🙌 1
2 Views