Is there anyway to use a custom package that is in...
# ask-community
f
Is there anyway to use a custom package that is installed in my environment to declare where the flow function is? Currently the deployment object needs a path, and does not accept the flow definition without a path.
j
If say you are running in a container? Then you can pip install the package with the flow in and then point to site packages - so the path on the deployment is something like "/usr/local/lib/python3.9/site-packages/" - that being said I think this is a hack - there doesn't seem to be first class support for storing flows within a package - I would very much like to see it! The idea of having to maintain my deps and my src separately really doesn't sit well with me...
f
@Joshua Greenhalgh Yeah this is confusing. My flow currently is run via a process worker that will insantiate some tasks run in different containers. I've tried to hardcode the path of my code in the
Deployment
object and use rsync before submitting a flow to make sure that the path contains the latest version of my code, but this is rather hacky and inefficient. Installable workflows should be a given in a python only tool such as prefect IMO.
j
I agree - I tried digging through the import logic in the src but its pretty dense lots of use of importlib and modifying paths etc - so although pointing at site-packages works for me - I cannot explain why...
f
Apparently prefect projects support pulling from a git repo but I really can't understand how to integrate them into a pipeline that triggers dynamic workflow. The documentation is rather confusing.
j
Yeah but even then its still just pulling raw src code - means you have to separately make sure the env its running in has all the correct dependencies...
👍 1
when getting src/deps can be accomplished at the same time with pip installing a module this should be possible...
a
I like the idea of being able to run a flow in a custom package as a deployment! Would one of you be willing to open an issue on the
prefect
repo so we can look into the feasibility of implementing this functionality?
f
I will, and if it were possible I'd like to contribute too.
🙌 1
gratitude thank you 1
🙌 1
So I don't know if this works for others but my issue was fixed via this pull request, couldn't request reviews from anyone in the repo don't know why @alex @Joshua Greenhalgh https://github.com/PrefectHQ/prefect/pull/9987
👏 1