Hi <@ULVA73B9P>, how can I configure Prefect, that...
# ask-marvin
o
Hi @Marvin, how can I configure Prefect, that not pypi but my personal artifactory is used for pulling pypi packages? My company does not allow access to pypi directly, but i can pull pypi packages from my local artifactory.
m
thought for 217.7 seconds
Great question. Short answer: Prefect doesn’t control where pip pulls packages from. You configure the runtime environment (container/VM/pod) that runs your flows to use your Artifactory instead of PyPI. Here are reliable ways to do it in Prefect 3.x: 1) Bake deps into a Docker image (recommended) - Create a Dockerfile that points pip to your Artifactory and installs your requirements. - Use that image in your deployment. Example Dockerfile:
Copy code
FROM python:3.11-slim

# Optional: set pip to use your Artifactory globally
RUN mkdir -p /etc/pip && \
    printf "[global]\nindex-url = <https://artifactory.myco.com/api/pypi/pypi/simple>\ntrusted-host = <http://artifactory.myco.com|artifactory.myco.com>\n" > /etc/pip/pip.conf

# Or set via env vars (works at build and runtime)
ENV PIP_INDEX_URL="<https://artifactory.myco.com/api/pypi/pypi/simple>"
ENV PIP_TRUSTED_HOST="<http://artifactory.myco.com|artifactory.myco.com>"

# Install your deps
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Install your code
WORKDIR /app
COPY . /app
Then reference the image in your deployment:
Copy code
# prefect.yaml (excerpt)
deployments:
  - name: my-deployment
    entrypoint: flows/my_flow.py:flow
    image: <http://registry.myco.com/prefect/my-flow:latest|registry.myco.com/prefect/my-flow:latest>
Tip: If your Artifactory requires auth, don’t hardcode creds. Use Docker BuildKit secrets or a private base image with pip already configured. 2) Kubernetes worker with runtime installs If you’re using Prefect’s Kubernetes worker and installing packages at container startup, configure pip and pass the packages to install. The worker image supports installing extra packages via an env var; set pip to use Artifactory via env vars or pip.conf. Kubernetes Pod template snippet:
Copy code
env:
  - name: PIP_INDEX_URL
    value: <https://artifactory.myco.com/api/pypi/pypi/simple>
  - name: PIP_TRUSTED_HOST
    value: <http://artifactory.myco.com|artifactory.myco.com>
  - name: EXTRA_PIP_PACKAGES
    value: "pandas==2.2.2 requests"
Docs: Prefect 3: Kubernetes deployments Note: Runtime installs are convenient but slower and can be brittle. For repeatable runs, prefer baking deps into your image (option 1). 3) Local/VM process worker If you run flows on a machine directly (ProcessWorker), configure pip on that machine: - Set env vars before installing:
PIP_INDEX_URL
,
PIP_EXTRA_INDEX_URL
(if you mirror),
PIP_TRUSTED_HOST
- Or create `~/.pip/pip.conf`:
Copy code
[global]
index-url = <https://artifactory.myco.com/api/pypi/pypi/simple>
trusted-host = <http://artifactory.myco.com|artifactory.myco.com>
Auth notes - Use credentials via env vars or a pip.conf that references a token, or use `.netrc`/keyring. - In Kubernetes, store creds in Secrets and mount as env vars. - In Docker builds, use BuildKit secrets so creds aren’t baked into the image layers. Would you share how you’re running your flows today? - Docker image + worker - Kubernetes worker - Local/VM process worker I can provide a tailored snippet for your setup (including handling Artifactory auth).