<@ULVA73B9P> Do you have any advice for speeding u...
# ask-marvin
e
@Marvin Do you have any advice for speeding up the "Executing deployment step: pip_install_requirements" seen in the logs when running a flow?
m
thought for 125.9 seconds
Yes — that log line usually means the worker is installing your flow's Python dependencies at run time, which can add a lot of startup overhead. Best advice: try to avoid runtime installs entirely. Fastest options 1. Bake dependencies into your image - If you're using Docker / Kubernetes / ECS, install requirements in the image build instead of in a deployment pull step. - Then remove
pip_install_requirements
from
pull
. - If needed, disable auto-install behavior with:
Copy code
PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES=false
2. Use a persistent environment - If your worker runs in a long-lived environment and dependencies are already installed, don't reinstall them for every run. 3. Avoid pulling/installing on every execution - If you're pulling code from Git, consider using storage options that cache between runs, like
GitRepository(..., pull_interval=...)
, so code isn't refreshed every single time. Why it's slow
pip_install_requirements
runs during deployment/workspace preparation, so every flow run may: - read
requirements.txt
- resolve deps - download packages - install them That can be especially slow with large packages like pandas, torch, db drivers, etc. Recommended pattern Instead of this in `prefect.yaml`:
Copy code
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/my-org/my-repo.git>
  - prefect.deployments.steps.pip_install_requirements:
      requirements_file: requirements.txt
Prefer a prebuilt image where your Dockerfile does:
Copy code
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . /app
WORKDIR /app
Then your deployment just points at that image. If you must keep runtime installs A few things can help: - keep
requirements.txt
small - pin versions to reduce resolver work - use a pip cache / persistent volume if your infra supports it - avoid reinstalling unchanged local packages - use
uv
-based/project installs if that's already part of your environment Prefect docs - prefect.yaml deployment steps - store flow code / deployment storage If you want, I can help you optimize this for your specific setup — e.g. Docker worker, Kubernetes worker, process worker, or Git-based deployment.