Romain Lefebvre
09/26/2025, 1:56 PMMarvin
09/26/2025, 1:56 PMMarvin
09/26/2025, 2:03 PMpull:
- prefect.deployments.steps.pull.git_clone:
repository: <https://github.com/your-org/your-repo>
branch: main
- prefect.deployments.steps.utility.run_shell_script:
script: |
if [ -n "${EXTRA_PKG}" ]; then
echo "Installing ${EXTRA_PKG}..."
python -m pip install "${EXTRA_PKG}"
else
echo "No EXTRA_PKG specified; skipping."
fi
expand_env_vars: true
Then, provide EXTRA_PKG
via job variables on your deployment (varies by worker type). For a Process worker you can set env via job variables; with Python deploy this looks like:
flow.from_source("<https://github.com/your-org/your-repo>", entrypoint="flows/my_flow.py:flow").deploy(
name="my-deploy",
work_pool_name="my-pool",
job_variables={
"env": {"EXTRA_PKG": "polars==1.9.0"}
},
)
Docs: Customize job variables
2) Install conditionally inside the flow (based on flow parameters)
- If you must key off flow parameters, do the install inside your flow before imports that need it:
from prefect import flow
import sys, subprocess
@flow
def my_flow(extra_pkg: str | None = None):
if extra_pkg:
subprocess.check_call([sys.executable, "-m", "pip", "install", extra_pkg])
# rest of your flow
Note: On-the-fly installs can slow runs and reduce reproducibility; prefer pre-baked images or the job-variable approach when possible.
Other options
- Create separate deployments (or images) per dependency set and choose the appropriate deployment at run time.
Quick clarification to tailor the best solution:
- When you say “job parameters,” do you mean flow run parameters (the inputs to your flow) or job variables (infrastructure-level fields/env for the worker)?
- Which worker type are you using (Process, Docker, Kubernetes)? The exact job_variables shape varies by worker.
Related docs:
- Deploy via Python
- Job variables: Customize job variablesRomain Lefebvre
09/26/2025, 3:04 PMMarvin
09/26/2025, 3:04 PMMarvin
09/26/2025, 3:05 PMfrom prefect.deployments import run_deployment
run_deployment(
"my-flow/my-deploy",
parameters={"extra_pkg": "<git+ssh://github.com/org/customlib.git#egg=customlib>"},
job_variables={"env": {"EXTRA_PKG": "<git+ssh://github.com/org/customlib.git#egg=customlib>"}},
)
prefect.yaml pull steps:
pull:
- prefect.deployments.steps.pull.git_clone:
repository: <https://github.com/your-org/your-repo>
branch: main
- prefect.deployments.steps.utility.run_shell_script:
script: |
if [ -n "${EXTRA_PKG}" ]; then
echo "Installing ${EXTRA_PKG}..."
python -m pip install "${EXTRA_PKG}"
else
echo "No EXTRA_PKG specified; skipping."
fi
expand_env_vars: true
Docs: Customize job variables, Deploy via Python
- Install inside the flow based on the parameter (simple, but less reproducible)
from prefect import flow
import sys, subprocess
@flow
def my_flow(extra_pkg: str | None = None):
if extra_pkg:
subprocess.check_call([sys.executable, "-m", "pip", "install", extra_pkg])
# proceed with imports/logic that require the package
Runtime context (if needed): prefect.runtime
Notes and trade-offs:
- Security: Installing arbitrary packages from parameters can be risky — consider validating/allowlisting inputs.
- Reproducibility: On-the-fly installs can slow runs and make environments non-deterministic. If possible, prefer pinned wheels/private index and/or prebuilt images per variant.
Quick question so I can tailor the best approach:
- How are you triggering runs (UI/manual, API, or from another flow)? If you can start runs via API or a parent flow, passing both parameters
and job_variables
per run is the cleanest way to drive pull-step installs.