In v2, is anyone packaging dependencies in storage along with the flow files? Seems like this could allow for dependencies per flow in a mono-repo without needing a separate docker image per flow. Kind of how aws lambda zip files work.

we have a base image with common deps then install anything else on demand with extra_pip_packages (k8s)

Also running on k8s, we have a central

functions

library that we have stored in git and use kubernetes/git-sync to sync the library into the pods and then have a symbolic link to mirror this directory into

/usr/local/lib/python3.9/site-packages/functions

. Then in your scripts you can just do

from functions import my_helper

etc

@Fred Birchall that is neat - not a python expert and i have been doing the pypi package dance to reuse functions

My original solution was to create a

setup.py

file and

pip install -e .

but when the git sync happened, it broke the python installation (don’t know the details of how the

-e

flag works but the sync broke it every time…). But I have in a couple of instances, I’ve manually wrote code within

site-packages

to be shared across multiple processes. It’s probably not the most production ready solution, but it does work! 😂 Hope this helps wit your issue! Also keen to see if there are better solutions in the Prefect world

I find packaging an overhead for something as easy as sharing functions around, the .net world put me off over packaging shared code