Hey all we are wondering how to add a custom python package

Question

Hey all we are wondering how to add a custom python package or module as a dependency to our prefect flow We have tried different options but all fail Any suggestions slightly smiling face

fabian wolfmann · Answer

Hi i think you should make a python package in order to add dependencies

bral · Answer

Or you can put your your custom module in your python environment path site packages directory

Severin Ryberg [sevberg] · Answer

< fabian wolfmann> This was our first thought but it doesn t really work when the python package is developed alongside the Prefect flow as in in the same repository That is unless you want to push your changes to the repo every time you want to test if it will work in the flow But in that case I think the development speed would suffer considerably confused

Severin Ryberg [sevberg] · Answer

< bral> Would you mind explaining this a bit more From my understanding during the docker build Prefect will try to pull the required packages from e g pypi So the local python environment is not copied over into the docker image

fabian wolfmann · Answer

im creating a python package you can create a `setup py` with Prefect as dependency and make `pip install e ` with e flag you say its editable package and it updated every time you call your package im working like this and have no issue

Severin Ryberg [sevberg] · Answer

< fabian wolfmann> Thanks a lot for the feedback But I guess I must be still missing something obvious since our package is already installed in editable mode But when the docker file is built it gives a module not found error during the health check In comparison simply running the flow in a local executor works without an issue

fabian wolfmann · Answer

yes because docker will pull the python packages from pypi not local you should try look for how to install local packages on docker image

bral · Answer

< fabian wolfmann> so when you are writing flow you can store all your custom plugins module in directory with project After when flow deployed registered on server you need add dependency of your package to environment where agent work and if you use dask cluster to environment for every worker too if multiserver cluster In your case you can add your local pypi server in pip conf inside docker image

Robin · Answer

Thanks already for the discussion pray Good point that we have to install the package in the two parts It s still not clear to me 1 How to install the custom package in docker using `pip install e ` 2 How to install the custom package in the prefect environment a dask kubernetes cluster on AWS EKS in our case

Robin · Answer

```In your case you can add your local pypi server in pip conf inside docker image``` < bral> could you elaborate on this

Robin · Answer

Concerning 1 Do you mean something like the following discussion when refering to `pip install e ` <https stackoverflow com questions 44708481 how to install local packages using pip as part of a docker build>

bral · Answer

1 First install python and pip in your Dockerfile 2 Copy your pip conf using COPY directive <https hackernoon com custom python pypi repository 409f14975374|https hackernoon com custom python pypi repository 409f14975374>

bral · Answer

Usefull link <https www docker com blog containerized python development part 1 |https www docker com blog containerized python development part 1 >

Robin · Answer

Thanks a lot We will try it out tomorrow greetings from Europe slightly smiling face

Robin · Answer

< Severin Ryberg sevberg > implemented an elegant solution that solved the issue for us

Severin Ryberg [sevberg] · Answer

That s only half of the solution actually stuck out tongue winking eye The other half depends on this though so I ll submit it once this change is accepted I ll make a note here when its all in

Severin Ryberg [sevberg] · Answer

Hello again everyone In case you are interested two new PR s have been merged into the latest Prefect version The first mentioned above by Robin allows entire directories to be copied into the Docker image s build environment The second allows for injecting arbitrary Docker commands into the build via the new extra dockerfile commands argument Therefore to add a local python module to your Docker image now one only needs to add the module s directory to the files dictionary and then add the following argument `extra dockerfile commands= RUN pip install e path to module in image `

Robin · Answer

PS Actually I guess that ` e` does probably not make so much sense and might break some python packages e g `snowflake connector python` in some cases

Severin Ryberg [sevberg] · Answer

I suggested the e flag to avoid having a double copy of the python module in the docker image If you didn t want to use it then you should additionally use a RUN command to remove the first copy Anyway regarding the issue of installing the snowflake connector this is normally something you would install via a normal pip install command anyway without copying any files from the host before hand so I don t think this is a big concern