Hey there, I'm using the ShellOperation and I'm wondering how I can make it fail fast when adding multiple statements. Lets say the first statement fails, the second one shoudn't start.

from prefect_shell import ShellOperation

with ShellOperation(
    commands=["exit 1", "echo 'Hello, world!'"],
) as shell_operation:
    shell_process = shell_operation.trigger()
    shell_process.wait_for_completion()
    shell_output = shell_process.fetch_result()

This runs both statements and then throws the error.

hi @Timo - how about adding a

&&

between?

In [1]: from prefect_shell import ShellOperation
   ...:
   ...: shell_operation = ShellOperation(commands=["exit 1", "&&", "echo 'Hello world!'"])

In [2]: shell_operation.run()
09:34:20.328 | INFO    | prefect.ShellOperation - PID 16310 triggered with 3 commands running inside the '.' directory.

...

RuntimeError: PID 16310 failed with return code 1.

Hi @Nate, thanks for providing the solution. This works pretty well. 🎉 Even if I add these in a DbtOperation. The project and profile parameter are added to each command.

sweet! glad it worked for you 🦜

Sorry @Nate, thought it works but noticed I had deployed a hard coded versions. If I add the && in DbtOperation it adds the --profiles-dir/--project-dir to it as well which then fails because this is an invalid cmd. so

["dbt run -m model_1", "&&", "dbt run -m model_2"]

ends in

dbt run -m model_1 --project_dir /path --profiles_dir /profile 
&& --project_dir /path --profiles_dir /profile 
dbt run -m model_2 --project_dir /path --profiles_dir /profile

hmm, would it be a pain in your case to run these models as separate

ShellOperation

s?

yeah because then i can use the dbt execute task instead. Thought using DbtOperation would be a better replacement because we can run multiple cmds with the same config. If I need to add a different operation for every dbt command and then execute them one after another there is no benefit over just using tasks. (theres more overhead because i need to manage multiple operations instead lets say one per flow)

ah yeah I see what you mean would it be preferable to have a bool arg accepted by the init of

ShellOperation

that fails fast as

&&

does if properly placed between commands? if so i can open a ticket and work on that as time allows

the problem is related to the

DbtOperation

. For

ShellOperation

it fails fast when adding

&&

(as ur example showed above). But for the

DbtOperation

the

--project_dir

and

--profiles_dir

argument are added to each command. But it also adds this to the

&&

which then fails if the first dbt command runs successful.

dbt run -m model_1 --project_dir /path --profiles_dir /profile 
&& --project_dir /path --profiles_dir /profile                  <------------------- WILL FAIL HERE
dbt run -m model_2 --project_dir /path --profiles_dir /profile

So there might be a option to prevent adding these parameter to specific commands or adding a exception if the command is

&&

.

ah sorry, to clarify, i said

ShellOperation

because

DbtCoreOperation

inherits from it so i was suggesting adding some kwarg to shell operation that we could pass through to the dbt impl and allow something like

dbt_op = DbtCoreOperation(["dbt run -m model_1", "dbt run -m model_2"], stop_on_failure=True)

which would raise the failure if model 1's run fails

yeah this would be perfect

cool - added an issue for tracking

awesome - thanks 🙌

I know this is not what you were asking, but you can also run multiple models in one dbt run command

dbt run --select model_1 model_2

. If you want faster failure you can also add a

--fail-fast

flag, which will end the dbt operation if a single node fails.

ooh! using those together sounds like it might do the trick - what do you think @Timo? thanks @Sean Williams!!! 🦸

Of course for this simplified example this would work. But in my case we first run a

dbt build

and then execute a macro call by using

dbt run-operation

afterwards. You can't combine this in one cmd. Another example maybe instead of using

dbt build

you first want to call`dbt run` and then call a

dbt test

not for all but for special models. You then need to put these into two commands.

ah i see - okay!

Åxf