I have a private Python package bundled up and stored in Google Artifact Registry (as opposed to an open-source package on PyPI). I'd like to include it in my flow, which uses a Docker storage instance in production. I know that I can use the

python_dependencies

kwarg to the class to include open-source packages. But how do I get a private package from Google Artifact Registry included as well? Is there an established pattern for this?

So this is happening because you are using the

DockerStorage

as the interface to build the image. If it’s too limiting, you can supply your own Dockerfile or image and just handle it yourself.

Adding

--index-url

should do the trick - you can add it to your Dockerfile like so:

RUN pip install --index-url <https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/> PACKAGE

But when you register your flow and build the image, your terminal must be authenticated with Artifact Registry Reader permissions. In my last job we were also using private artifactory and you could also just add the index URL at the top of your

requirements.txt

like so:

--index-url <https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple>
--extra-index-url <https://pypi.org/simple>

# PyPi dependencies
pandas==1.4.1

# your custom private packages
custom==0.1