Thread
#prefect-server
    b

    Ben Welsh

    7 months ago
    I have a private Python package bundled up and stored in Google Artifact Registry (as opposed to an open-source package on PyPI). I'd like to include it in my flow, which uses a Docker storage instance in production. I know that I can use the
    python_dependencies
    kwarg to the class to include open-source packages. But how do I get a private package from Google Artifact Registry included as well? Is there an established pattern for this?
    Kevin Kho

    Kevin Kho

    7 months ago
    So this is happening because you are using the
    DockerStorage
    as the interface to build the image. If it’s too limiting, you can supply your own Dockerfile or image and just handle it yourself.
    But for this, I think there is a chance we can get it to work by pointing your
    pip
    to the Artifact Registry. Each one of the dependencies are just pip installed here by adding commands to the container
    So this will work if doiing
    pip install some_library"
    magically worked. I’ll look a bit
    You can also add extra commands to the Docker build if ever we can’t pip install it
    I think you need to add your own
    pip install
    command with the extra things like this
    Anna Geller

    Anna Geller

    7 months ago
    Adding
    --index-url
    should do the trick - you can add it to your Dockerfile like so:
    RUN pip install --index-url <https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/> PACKAGE
    But when you register your flow and build the image, your terminal must be authenticated with Artifact Registry Reader permissions. In my last job we were also using private artifactory and you could also just add the index URL at the top of your
    requirements.txt
    like so:
    --index-url <https://LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple>
    --extra-index-url <https://pypi.org/simple>
    
    # PyPi dependencies
    pandas==1.4.1
    
    # your custom private packages
    custom==0.1