https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> how do i deploy a deployment on a sel...
# ask-marvin
r
@Marvin how do i deploy a deployment on a self-hosted prefect in a different VM in a process work pool. show yaml example and the cli command. for example if you are on prefect cloud, you would run prefect cli cloud login. what about self hosted?
m
✅ thought for 461.9 seconds
Great question — here’s the end-to-end for deploying to a self-hosted Prefect Server from one VM and running on another VM with a Process work pool. Assuming Prefect 3.x. High-level flow - You do NOT “log in” for self-hosted. Instead, you point your CLI and workers at your Server API by setting 
PREFECT_API_URL
. - Create a Process work pool on your self-hosted Server. - Define your deployment (YAML) so the worker on the other VM can pull your code (e.g., from Git). - Run 
prefect deploy
from your dev machine to register the deployment to your Server. - Start a worker on the other VM to poll the pool and execute runs locally. Configure your CLI to talk to self-hosted (both VMs) On each machine (your dev box and the worker VM), set a profile that points to your Server API:
Copy code
prefect profile create self-hosted
prefect profile use self-hosted
prefect config set PREFECT_API_URL=http://<your-server-host>:4200/api
- For Prefect Cloud you’d run 
prefect cloud login
. For self-hosted Server there is no login; you just set 
PREFECT_API_URL
(and no API key unless you’ve added your own gateway/auth in front of Server). Create a Process work pool (once) Run this from your dev machine after setting `PREFECT_API_URL`:
Copy code
prefect work-pool create vm-process-pool --type process
Example deployment YAML (prefect.yaml) Place this at the root of your project repository. This example pulls code from Git so the worker on the other VM can run it without pre-installing your repo.
Copy code
name: my-project
prefect-version: ">=3.0.0"

# Define one or more deployments for this project
deployments:
  - name: my-deployment
    entrypoint: flows/etl.py:etl_flow            # path in repo to your flow
    description: "ETL pipeline via process worker"
    parameters:
      run_date: "2025-01-01"
    tags: []
    work_pool:
      name: vm-process-pool
      job_variables:
        stream_output: true
        # working_dir can be set if your flow needs a specific subdir
        # working_dir: ./my-project
    pull:
      - prefect.deployments.steps.git_clone:
          repository: <https://github.com/acme/my-project.git>
          branch: main
          # access_token: ${GITHUB_TOKEN}        # if needed for private repos
      - prefect.deployments.steps.set_working_directory:
          directory: my-project                  # adjust if your repo has a subdir
    schedules: []
Deploy the deployment to your self-hosted Server From your project root (where 
prefect.yaml
lives):
Copy code
prefect deploy
- This registers the deployment with the self-hosted API defined by your active profile. Start a worker on the other VM On the VM that will execute the flow (after setting the same 
PREFECT_API_URL
profile):
Copy code
prefect worker start -p vm-process-pool
- The worker will poll the 
vm-process-pool
, clone your repo via the 
pull
steps, and run the flow locally as a process. - Make sure this VM has required system deps (e.g., git) and Python packages your flow needs. If you want the worker to install Python deps from the repo, you can add another pull step:
Copy code
- prefect.deployments.steps.pip_install_requirements:
      requirements_file: requirements.txt
One-shot CLI alternative (without YAML) If you prefer, you can register a single deployment directly from your entrypoint (YAML is generally more repeatable):
Copy code
prefect deploy ./flows/etl.py:etl_flow -n my-deployment -p vm-process-pool
Note: to run on a different VM, you’ll still need a way for the worker to get your code (e.g., Git pull steps in 
prefect.yaml
, a packaged module installed on the VM, or a shared filesystem). The YAML approach above is the most portable. Common gotchas - Ensure your self-hosted API is reachable from both machines (open ports, correct host/IP). - Use the same profile/API URL on your dev machine and worker VM. - For private repos, provide a token via environment variable and reference it in the 
git_clone
step.
If you share your repo layout (file paths, private/public), I can tailor the 
prefect.yaml
exactly to your setup.
r
@Marvin in the prefect.yaml file how do i install the repo requirements using uv without triggering error on RuntimeError: 
run_shell_script
failed with error code 1: error: externally-managed-environment
m
✅ thought for 427.2 seconds
Short answer: don’t install into the system Python from a pull step. Create a venv with uv in your pull steps, then make the flow run inside that venv by adjusting the job’s PATH/VIRTUAL_ENV. That avoids the PEP 668 “externally-managed-environment” error. Here’s a working Prefect 3.x pattern using 
uv venv
+ 
uv pip install
in `prefect.yaml`: - Clone the repo - Set the working dir to the repo - Create a venv with uv and install requirements into it (no shell “activate” needed) - Ensure the flow run uses that venv by setting PATH/VIRTUAL_ENV in job variables Example prefect.yaml
Copy code
deployments:
- name: my-deployment
  entrypoint: flows/my_flow.py:flow
  work_pool:
    name: my-process-pool
    job_variables:
      # Ensure the flow process uses the venv interpreter and site-packages
      env:
        VIRTUAL_ENV: "{{ clone.directory }}/.venv"
        PATH: "{{ clone.directory }}/.venv/bin:{{ env.PATH }}"
  pull:
    - prefect.deployments.steps.pull.git_clone:
        id: clone
        repository: <https://github.com/your-org/your-repo.git>
        branch: main
    - prefect.deployments.steps.pull.set_working_directory:
        directory: "{{ clone.directory }}"
    - prefect.deployments.steps.utility.run_shell_script:
        id: setup_venv
        script: |
          set -euxo pipefail
          # Ensure uv is available (if your workers don’t already have it)
          if ! command -v uv >/dev/null 2>&1; then
            curl -LsSf <https://astral.sh/uv/install.sh> | sh
            export PATH="$HOME/.local/bin:$PATH"
          fi
          # Create a project-local venv and install deps into it
          uv venv .venv
          uv pip install --python .venv/bin/python -r requirements.txt
Why this works - The error you saw happens when pip/uv tries to modify a system-managed Python (PEP 668). Installing into a project venv sidesteps that entirely. - Pull steps run before the flow is imported/executed. By setting 
job_variables.env.PATH
and 
VIRTUAL_ENV
, the worker’s flow process will use 
.venv/bin/python
and the venv’s site-packages. Notes and alternatives - Windows path tweak: 
.venv\Scripts\python.exe
and add 
.venv\Scripts
to 
PATH
. - If you’re using Docker/Kubernetes workers: - Best practice: bake deps into the image during build. - If you must install at runtime into the container’s system Python, uv can do it explicitly:
Copy code
UV_SYSTEM_PYTHON=1 uv pip install --system -r requirements.txt
Use this only inside containers you control; it opts into modifying system site-packages. - Prefer 
uv pip sync -r requirements.txt
if you keep a lockfile or want exact synchronization. Docs for reference - Pull steps and yaml structure: prefect.yaml guide - Pull steps API: prefect.deployments.steps.pull - Utility steps, including run_shell_script: prefect.deployments.steps.utility Quick check - Make sure your worker environment has 
uv
available (either preinstalled on hosts/images or installed in the pull step as shown). - Confirm your 
requirements.txt
lives at the repo root (or adjust the path in the script). Want me to tailor this to your worker type (process, Docker, Kubernetes) and OS? If so, which worker are you using and is 
uv
already available in that environment?
r
@Marvin Unexpected exception encountered when trying to load flow Traceback (most recent call last): File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps step_output = await run_step(step, upstream_outputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/deployments/steps/core.py", line 137, in run_step result = await from_async.call_soon_in_new_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Call.new(step_func, **inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ).aresult() ^^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult return await asyncio.wrap_future(self.future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync result = self.fn(*self.args, **self.kwargs) TypeError: run_shell_script() got an unexpected keyword argument 'prefect.deployments.steps.utility.run_shell_script' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/engine.py", line 112, in <module> flow: "Flow[..., Any]" = load_flow(flow_run) ~~~~~~~~~^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/flow_engine.py", line 150, in load_flow flow = run_coro_as_sync( load_flow_from_flow_run(flow_run, use_placeholder_flow=False) ) File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync return call.result() ~~~~~~~~~~~^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result return self.future.result(timeout=timeout) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/datalake/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run output = await run_steps( ^^^^^^^^^^^^^^^^ deployment.pull_steps, print_function=run_logger.info ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/datalake/prefect-pmcare/.venv/lib/python3.13/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps raise StepExecutionError(f"Encountered error while running {fqn}") from exc prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.run_shell_script getting this error. why
m
✅ thought for 79.3 seconds
Thanks for the traceback — this error is almost always a YAML shape issue. What’s happening - Prefect is correctly resolving the step function 
run_shell_script(...)
. - But your YAML is passing an extra keyword argument named 
prefect.deployments.steps.utility.run_shell_script
into that function. - Since 
run_shell_script
does not accept that argument, Python raises: TypeError: run_shell_script() got an unexpected keyword argument 'prefect.deployments.steps.utility.run_shell_script' This typically comes from a small indentation/structure mistake where the fully-qualified step name appears inside the arguments block instead of being the key of the step item. Correct shape for pull steps Each list item in 
pull:
must be a single-key mapping where the key is the step’s fully-qualified name and the value is that step’s arguments. Example:
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      id: clone
      repository: <https://github.com/your-org/your-repo.git>
      branch: main

  - prefect.deployments.steps.pull.set_working_directory:
      directory: "{{ clone.directory }}"

  - prefect.deployments.steps.utility.run_shell_script:
      id: setup_venv
      script: |
        set -euxo pipefail
        if ! command -v uv >/dev/null 2>&1; then
          curl -LsSf <https://astral.sh/uv/install.sh> | sh
          export PATH="$HOME/.local/bin:$PATH"
        fi
        uv venv .venv
        uv pip install --python .venv/bin/python -r requirements.txt
Common mistake that triggers your error This (note the extra nested key) will cause the exact TypeError you’re seeing:
Copy code
- prefect.deployments.steps.utility.run_shell_script:
    script: |
      ...
    prefect.deployments.steps.utility.run_shell_script: true   # <-- This line (or similar) becomes an unexpected kwarg
Or using the function name at the wrong level:
Copy code
- run_shell_script:
    prefect.deployments.steps.utility.run_shell_script: true   # <-- becomes a kwarg
    script: |
      ...
Double-check: - The step key should be exactly 
prefect.deployments.steps.utility.run_shell_script
- Inside that mapping, only valid args for the function are allowed: 
script
, 
directory
, 
env
, 
stream_output
, 
expand_env_vars
, and optionally 
id
as a reserved keyword handled by Prefect. Docs for reference - run_shell_script API (valid args): prefect.deployments.steps.utility - YAML structure guide: prefect.yaml guide If you paste your current 
pull:
section, I can pinpoint the exact line to fix.
r
@Marvin definitions: actions: git_main_pull: &git_main_pull - prefect.deployments.steps.git_clone: id: pmcare-repo repository: https://github.com/repo-url.git branch: main credentials: "{{ prefect.blocks.github-credentials.cred }}" - prefect.deployments.steps.run_shell_script: id: setup_venv script: | set -euxo pipefail # Ensure uv is available (if your workers don't already have it) if ! command -v uv >/dev/null 2>&1; then curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:$PATH" fi # Create a project-local venv and install deps into it uv venv .venv uv pip install --python .venv/bin/python -r requirements.txt directory: "{{ pmcare-repo.directory }}"
m
✅ thought for 93.4 seconds
You’re super close — the error is because the step names are wrong. In Prefect 3.x the correct fully-qualified names are: - git clone: 
prefect.deployments.steps.pull.git_clone
- run shell script: 
prefect.deployments.steps.utility.run_shell_script
Using 
prefect.deployments.steps.git_clone
or 
prefect.deployments.steps.run_shell_script
(missing the submodule) causes Prefect to mis-parse the step and pass the step name as a keyword arg to the function, which triggers the “unexpected keyword argument” TypeError you saw. Fix your YAML like this:
Copy code
definitions:
  actions:
    git_main_pull: &git_main_pull
      - prefect.deployments.steps.pull.git_clone:
          id: pmcare-repo
          repository: <https://github.com/repo-url.git>
          branch: main
          credentials: "{{ prefect.blocks.github-credentials.cred }}"
      - prefect.deployments.steps.utility.run_shell_script:
          id: setup_venv
          directory: "{{ pmcare-repo.directory }}"
          script: |
            set -euxo pipefail
            # Ensure uv is available (if your workers don't already have it)
            if ! command -v uv >/dev/null 2>&1; then
              curl -LsSf <https://astral.sh/uv/install.sh> | sh
              export PATH="$HOME/.local/bin:$PATH"
            fi
            # Create a project-local venv and install deps into it
            uv venv .venv
            uv pip install --python .venv/bin/python -r requirements.txt
Notes - Remove angle brackets from the URL in YAML (they were likely just for formatting in chat). - Valid args for run_shell_script are only: 
script
, 
directory
, 
env
, 
stream_output
, 
expand_env_vars
, plus 
id
handled by Prefect. - The 
credentials
parameter on 
git_clone
is correct for referencing a 
github-credentials
block. Docs for reference - Steps (pull): prefect.deployments.steps.pull.git_clone - Steps (utility): prefect.deployments.steps.utility.run_shell_script If this still errors, paste the new traceback and your full 
pull:
section and I’ll pinpoint it.
r
@Marvin can you fix this definitions: actions: git_main_pull: &git_main_pull - prefect.deployments.steps.git_clone: id: healthproximate-materialized repository: https://github.com/ValianceHealth/healthproximate-materialized-etl.git branch: main credentials: "{{ prefect.blocks.github-credentials.valiance-health-github-cred }}" - prefect.deployments.steps.run_shell_script: id: setup_venv directory: "{{ healthproximate-materialized.directory }}" script: bash -c "uv pip install --system ." # Deployments section - defines your flow deployments deployments: - name: caseaggregate-deployment pull: *git_main_pull tags: [] description: "Aggregate case data deployment in healthproximate" entrypoint: app/caseaggregate.py:main_flow parameters: {} work_pool: name: default job_variables: env: ENV: dev schedules: - cron: "0 0 * * *" timezone: "Asia/Bangkok" The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/prefect/engine.py", line 112, in <module> flow: "Flow[..., Any]" = load_flow(flow_run) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 150, in load_flow flow = run_coro_as_sync( ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync return call.result() ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result return self.future.result(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run output = await run_steps( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps raise StepExecutionError(f"Encountered error while running {fqn}") from exc prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.utility.run_shell_script getting this
m
✅ thought for 140.0 seconds
Thanks for sharing the snippet and the traceback. The error is happening while executing the 
run_shell_script
step. There are a few likely issues in your YAML that can cause this: - 
uv
is probably not installed in your worker image, so 
uv pip install --system .
fails - Your step id uses a hyphen (
healthproximate-materialized
) which can make Jinja templating brittle - The repository URL is wrapped in Slack-style angle brackets; remove those - You don’t need to wrap the script with `bash -c`; the step already runs in a shell and your quoting may be breaking YAML Here are two working options. Option A: Use pip (simplest, no uv dependency)
Copy code
definitions:
  actions:
    git_main_pull: &git_main_pull
      - prefect.deployments.steps.git_clone:
          id: repo
          repository: <https://github.com/ValianceHealth/healthproximate-materialized-etl.git>
          branch: main
          credentials: "{{ prefect.blocks.github-credentials.valiance-health-github-cred }}"
      - prefect.deployments.steps.run_shell_script:
          id: setup_env
          directory: "{{ repo.directory }}"
          script: |
            python -m pip install --upgrade pip
            pip install .
          stream_output: true

deployments:
  - name: caseaggregate-deployment
    pull: *git_main_pull
    tags: []
    description: "Aggregate case data deployment in healthproximate"
    entrypoint: app/caseaggregate.py:main_flow
    parameters: {}
    work_pool:
      name: default
      job_variables:
        env:
          ENV: dev
    schedules:
      - cron: "0 0 * * *"
        timezone: "Asia/Bangkok"
Option B: Keep uv, but install it first
Copy code
definitions:
  actions:
    git_main_pull: &git_main_pull
      - prefect.deployments.steps.git_clone:
          id: repo
          repository: <https://github.com/ValianceHealth/healthproximate-materialized-etl.git>
          branch: main
          credentials: "{{ prefect.blocks.github-credentials.valiance-health-github-cred }}"
      - prefect.deployments.steps.run_shell_script:
          id: install_uv
          script: "python -m pip install -U 'uv>=0.5.6'"
          stream_output: true
      - prefect.deployments.steps.run_shell_script:
          id: setup_env
          directory: "{{ repo.directory }}"
          script: "uv pip install --system ."
          stream_output: true

deployments:
  - name: caseaggregate-deployment
    pull: *git_main_pull
    tags: []
    description: "Aggregate case data deployment in healthproximate"
    entrypoint: app/caseaggregate.py:main_flow
    parameters: {}
    work_pool:
      name: default
      job_variables:
        env:
          ENV: dev
    schedules:
      - cron: "0 0 * * *"
        timezone: "Asia/Bangkok"
Notes: - I changed the 
git_clone
step id to 
repo
so you can safely reference it as 
{{ repo.directory }}
. Hyphens in ids can break Jinja’s dot-notation. - Removed the Slack 
<...>
around the repository URL. - Switched the script to either a block scalar (
|
) or plain string and removed 
bash -c
to avoid quoting pitfalls. - Set 
stream_output: true
so you’ll see the real error next time if something fails. If you want to confirm the step names and outputs: - 
git_clone
exposes 
directory
that you can reference like 
{{ repo.directory }}
. - 
run_shell_script
is correctly named and returns 
stdout
and 
stderr
. Links: - git_clone step source - run_shell_script step source - Example pattern using uv in deployments: docs example Questions to help finalize: - Which worker type/image are you using? If it doesn’t have uv preinstalled, Option A is safer.
- Does your GitHub credentials block slug 
valiance-health-github-cred
exist in your Prefect instance? If not, we can adjust that too.
21 Views
Open in Slack
PreviousNext
Powered by