https://prefect.io
Join Slack
<@ULVA73B9P> my flow is crashing during run_shell_script step and I can't make sense of it here is t...
# ask-marvin
g
@Marvin my flow is crashing during run_shell_script step and I can't make sense of it here is the logs
Copy code
Unexpected exception encountered when trying to load flow
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 137, in run_step
    result = await from_async.call_soon_in_new_thread(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/utility.py", line 193, in run_shell_script
    raise RuntimeError(
RuntimeError: `run_shell_script` failed with error code 1: Downloading pygments (1.2MiB)
Downloading uv (20.1MiB)
Downloading pydantic-core (1.9MiB)
 Downloaded pydantic-core
 Downloaded uv
 Downloaded pygments
Installed 35 packages in 111ms
Bytecode compiled 1000 files in 1.06s


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/engine.py", line 112, in <module>
    flow: "Flow[..., Any]" = load_flow(flow_run)
                             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 150, in load_flow
    flow = run_coro_as_sync(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2810, in load_flow_from_flow_run
    output = await run_steps(
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.run_shell_script
m
✅ thought for 329.3 seconds
Thanks for sharing the logs — the key line is:
Copy code
RuntimeError: `run_shell_script` failed with error code 1
That means the shell command in your deployment step exited non‑zero. The output you see (the uv/pygments/pydantic-core downloads) is just what the script printed before it failed. A few common causes and quick checks: - You’re using bash features with a sh shell. By default, 
run_shell_script
executes via the system shell (typically 
/bin/sh
on Linux, which is often 
dash
, not 
bash
). Bash-only features like 
source
, 
[[ ... ]]
, functions, or 
set -o pipefail
will fail under 
sh
. - Fix: Explicitly invoke bash in your script:
Copy code
script: |
      bash -lc '
        set -euxo pipefail
        # your commands here
      '
Or drop bash-isms if you want to stick to POSIX sh. - A later command is failing silently. The uv install logs look successful, so a command after that likely returned 1. Add tracing to see exactly what’s failing:
Copy code
script: |
    set -eux  # use -x for tracing; avoid pipefail on sh
    which bash || true
    echo "PWD=$(pwd)"
    uv --version || true
    uv pip install -r requirements.txt
    # next commands that might be failing
If you want 
pipefail
, switch to bash as shown above. - Env var expansion surprises. If your script references 
$VARS
that you provide via the step’s 
env
, the shell will expand them. If you used unusual quoting or want Prefect to pre-expand the script text, set 
expand_env_vars: true
. - Working directory issues. If you assume a certain CWD, set 
directory:
in the step to the right path (e.g., your repo root). If you can, please share: - The exact 
run_shell_script
step from your 
prefect.yaml
(the 
script
, 
directory
, 
env
, and any other options) - Where this is running (worker type/container image/OS) - Your Prefect version (assuming 3.x unless you say otherwise) Example of a robust step you can try to isolate the problem:
Copy code
steps:
  - run_shell_script:
      script: |
        bash -lc '
          set -euxo pipefail
          echo "PWD=$(pwd)"
          which python || true
          python -V || true
          uv pip install -r requirements.txt
          echo "Done installing"
          # put the next command that was failing here
        '
      directory: "{{ project.root }}"
      env:
        # any env you need
      stream_output: true
If you’re installing requirements, consider using the built-in step instead of rolling your own shell command: - API: run_shell_script and pip_install_requirements - How to structure steps in prefect.yaml Once you share the script content, I can pinpoint the exact bash/sh issue or the specific command that’s returning 1.
17 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by