https://prefect.io logo
#prefect-community
Title
# prefect-community
b

Braun Reyes

11/23/2022, 9:01 PM
In v2, is anyone packaging dependencies in storage along with the flow files? Seems like this could allow for dependencies per flow in a mono-repo without needing a separate docker image per flow. Kind of how aws lambda zip files work.
r

redsquare

11/23/2022, 9:30 PM
we have a base image with common deps then install anything else on demand with extra_pip_packages (k8s)
f

Fred Birchall

11/24/2022, 12:32 PM
Also running on k8s, we have a central
functions
library that we have stored in git and use kubernetes/git-sync to sync the library into the pods and then have a symbolic link to mirror this directory into
/usr/local/lib/python3.9/site-packages/functions
. Then in your scripts you can just do
from functions import my_helper
etc
r

redsquare

11/24/2022, 2:39 PM
@Fred Birchall that is neat - not a python expert and i have been doing the pypi package dance to reuse functions
f

Fred Birchall

11/24/2022, 4:11 PM
My original solution was to create a
setup.py
file and
pip install -e .
but when the git sync happened, it broke the python installation (don’t know the details of how the
-e
flag works but the sync broke it every time…). But I have in a couple of instances, I’ve manually wrote code within
site-packages
to be shared across multiple processes. It’s probably not the most production ready solution, but it does work! 😂 Hope this helps wit your issue! Also keen to see if there are better solutions in the Prefect world
r

redsquare

11/24/2022, 10:56 PM
I find packaging an overhead for something as easy as sharing functions around, the .net world put me off over packaging shared code
3 Views