Does anyone know how you re supposed to use a Python package Prefect Community #ask-community

Does anyone know how you're supposed to use a Pyth...

Shareef Jalloq

09/23/2025, 4:45 PM

Does anyone know how you're supposed to use a Python package as a deployment source? I'm trying to use

GitRepository

in the Python API yet there doesn't seem to be any way to add that package to PYTHONPATH or pip install it.

Nate

09/23/2025, 5:12 PM

hi @Shareef Jalloq - if you're cloning your code at runtime then you'd run some

pull

step like

run_shell_script

to install your library at runtime via

pip

uv

Nate

09/23/2025, 5:12 PM

https://docs.prefect.io/v3/how-to-guides/deployments/prefect-yaml#utility-steps

Nate

09/23/2025, 5:13 PM

https://github.com/zzstoatzz/prefect-pack/blob/main/prefect.yaml#L72

Shareef Jalloq

10/15/2025, 2:49 PM

@Nate Thanks. I've been having a play but it looks like the utility steps are run inside the worker's venv. That's not good. We're running in an HPC environment so can't use Docker. We use Slurm to submit jobs and environment modules to load specific EDA tools for each job. What I need to do is: 1. Run the worker in its own venv that is static and not updated by flows 2. Have a pull step clone my GitRepository 3. Have a utility step create a new unique venv and pip install my GitRepository for each worker job How would I activate a venv and/or pass PATH/PYTHONPATH to the subsequent flow?

Shareef Jalloq

10/16/2025, 9:16 AM

Just for anyone else trying to do this, I managed to get it to work by overriding the job

command

job_variables

. I create a unique venv and pip install the same Git SHA as would be cloned by the

GitRepository

. A bit roundabout but better than the other options.

Copy code

command = [
        "bash -lc '",
        "VENV=$(mktemp -d)/venv;",
        "python3 -m venv $VENV;",
        "source $VENV/bin/activate;",
        "python -m pip install --upgrade pip wheel;",
        "SOURCE_REF=${FPGA_AUTOMATION_SOURCE_REFERENCE:-};",
        f'INSTALL_TARGET="fpga-automation[server] @ git+{sanitised_url}";',
        'if [ -n "$SOURCE_REF" ]; then',
        '  INSTALL_TARGET="${INSTALL_TARGET}@${SOURCE_REF}";',
        "fi;",
        'python -m pip install "$INSTALL_TARGET";',
        "FLOW_RUN_ID=${PREFECT__FLOW_RUN_ID:-${PREFECT_FLOW_RUN_ID:-}};",
        'if [ -z "$FLOW_RUN_ID" ]; then',
        '  echo "Missing Prefect flow run ID in PREFECT__FLOW_RUN_ID/PREFECT_FLOW_RUN_ID" >&2;',
        "  exit 2;",
        "fi;",
        'exec python -m prefect flow-run execute "$FLOW_RUN_ID"',
        "'",
    ]

Open in Slack

Previous Next