https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> my flow is crashing during run_shell_...
# ask-marvin
g
@Marvin my flow is crashing during run_shell_script step and I can't make sense of it here is the logs
Copy code
Unexpected exception encountered when trying to load flow
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 137, in run_step
    result = await from_async.call_soon_in_new_thread(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/utility.py", line 193, in run_shell_script
    raise RuntimeError(
RuntimeError: `run_shell_script` failed with error code 1: Downloading pygments (1.2MiB)
Downloading uv (20.1MiB)
Downloading pydantic-core (1.9MiB)
 Downloaded pydantic-core
 Downloaded uv
 Downloaded pygments
Installed 35 packages in 111ms
Bytecode compiled 1000 files in 1.06s


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/engine.py", line 112, in <module>
    flow: "Flow[..., Any]" = load_flow(flow_run)
                             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 150, in load_flow
    flow = run_coro_as_sync(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2810, in load_flow_from_flow_run
    output = await run_steps(
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.run_shell_script
m
✅ thought for 329.3 seconds
Thanks for sharing the logs — the key line is:
Copy code
RuntimeError: `run_shell_script` failed with error code 1
That means the shell command in your deployment step exited non‑zero. The output you see (the uv/pygments/pydantic-core downloads) is just what the script printed before it failed. A few common causes and quick checks: - You’re using bash features with a sh shell. By default, 
run_shell_script
executes via the system shell (typically 
/bin/sh
on Linux, which is often 
dash
, not 
bash
). Bash-only features like 
source
, 
[[ ... ]]
, functions, or 
set -o pipefail
will fail under 
sh
. - Fix: Explicitly invoke bash in your script:
Copy code
script: |
      bash -lc '
        set -euxo pipefail
        # your commands here
      '
Or drop bash-isms if you want to stick to POSIX sh. - A later command is failing silently. The uv install logs look successful, so a command after that likely returned 1. Add tracing to see exactly what’s failing:
Copy code
script: |
    set -eux  # use -x for tracing; avoid pipefail on sh
    which bash || true
    echo "PWD=$(pwd)"
    uv --version || true
    uv pip install -r requirements.txt
    # next commands that might be failing
If you want 
pipefail
, switch to bash as shown above. - Env var expansion surprises. If your script references 
$VARS
that you provide via the step’s 
env
, the shell will expand them. If you used unusual quoting or want Prefect to pre-expand the script text, set 
expand_env_vars: true
. - Working directory issues. If you assume a certain CWD, set 
directory:
in the step to the right path (e.g., your repo root). If you can, please share: - The exact 
run_shell_script
step from your 
prefect.yaml
(the 
script
, 
directory
, 
env
, and any other options) - Where this is running (worker type/container image/OS) - Your Prefect version (assuming 3.x unless you say otherwise) Example of a robust step you can try to isolate the problem:
Copy code
steps:
  - run_shell_script:
      script: |
        bash -lc '
          set -euxo pipefail
          echo "PWD=$(pwd)"
          which python || true
          python -V || true
          uv pip install -r requirements.txt
          echo "Done installing"
          # put the next command that was failing here
        '
      directory: "{{ project.root }}"
      env:
        # any env you need
      stream_output: true
If you’re installing requirements, consider using the built-in step instead of rolling your own shell command: - API: run_shell_script and pip_install_requirements - How to structure steps in prefect.yaml Once you share the script content, I can pinpoint the exact bash/sh issue or the specific command that’s returning 1.
18 Views
Open in Slack
PreviousNext
Powered by