In v2, is anyone packaging dependencies in storage...
# ask-community
b
In v2, is anyone packaging dependencies in storage along with the flow files? Seems like this could allow for dependencies per flow in a mono-repo without needing a separate docker image per flow. Kind of how aws lambda zip files work.
r
we have a base image with common deps then install anything else on demand with extra_pip_packages (k8s)
f
Also running on k8s, we have a central
functions
library that we have stored in git and use kubernetes/git-sync to sync the library into the pods and then have a symbolic link to mirror this directory into
/usr/local/lib/python3.9/site-packages/functions
. Then in your scripts you can just do
from functions import my_helper
etc
r
@Fred Birchall that is neat - not a python expert and i have been doing the pypi package dance to reuse functions
f
My original solution was to create a
setup.py
file and
pip install -e .
but when the git sync happened, it broke the python installation (don’t know the details of how the
-e
flag works but the sync broke it every time…). But I have in a couple of instances, I’ve manually wrote code within
site-packages
to be shared across multiple processes. It’s probably not the most production ready solution, but it does work! 😂 Hope this helps wit your issue! Also keen to see if there are better solutions in the Prefect world
r
I find packaging an overhead for something as easy as sharing functions around, the .net world put me off over packaging shared code