Ben Welsh
02/14/2022, 8:37 PMpython_dependencies
kwarg to the class to include open-source packages. But how do I get a private package from Google Artifact Registry included as well? Is there an established pattern for this?Kevin Kho
02/14/2022, 8:43 PMDockerStorage
as the interface to build the image. If it’s too limiting, you can supply your own Dockerfile or image and just handle it yourself.pip
to the Artifact Registry. Each one of the dependencies are just pip installed here by adding commands to the containerpip install some_library"
magically worked. I’ll look a bitAnna Geller
02/14/2022, 10:31 PM--index-url
should do the trick - you can add it to your Dockerfile like so:
RUN pip install --index-url <https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/> PACKAGE
But when you register your flow and build the image, your terminal must be authenticated with Artifact Registry Reader permissions.
In my last job we were also using private artifactory and you could also just add the index URL at the top of your requirements.txt
like so:
--index-url <https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple>
--extra-index-url <https://pypi.org/simple>
# PyPi dependencies
pandas==1.4.1
# your custom private packages
custom==0.1