Dear prefect people, we have some troubles on running our dbt model with prefect. Using dbt run, we are able to run the model locally, however in prefect, we only get the following error

ERROR - prefect.DbtShellTask | Command failed with exit code 1

Any thoughts on what could be wrong or how to get further information and debug the flow are appreciated! 🙂

Hi @Robin! When you’re able to run locally and can’t run it in Cloud, it could be missing dependencies on the agent. What agent are you using? Did you try using

log_stderr=True

to get the error?

Ah for clarification: • without prefect,

dbt run

works locally • with prefect,

dbt run

works neither locally nor in cloud

Ah gotcha. Can you post how you’re using the

DbtShellTask

? Also try the

log_stderr

.

from prefect import Flow

from prefect.tasks.dbt import DbtShellTask

PROFILES = "."

with Flow("dbt_flow") as flow:

    task = DbtShellTask(
        environment="Deployment",
        profiles_dir=PROFILES,
    )(command="dbt debug")

flow.run()

where the

dbt_flow.py

is in the dbt project folder where the

dbt_project.yml

lies and

profiles.yml

is in the same folder

Am I missing it? Not quite seeing where

dbt_flow.py

is written in the Flow

Still only getting

INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 1')

I guess my suggestion would be to leave

profiles_dir

blank and use

dbt debug

to see if that’ll work. It will create a

profiles.yml

for you. We can try this and see if it works?

hmm, ok

Sorry about that. I’ll keep trying though 😢

Thanks for your help!

About that, when I use something like:

task = DbtShellTask(
        profile_name='default',
        environment="Deployment",
        profiles_dir="/tmp/",
        return_all=True,
        log_stderr=True
    )

nope, I only get that one line 🤔

Can we try corrupting the command like

dbbtt debug

? Just to see if you’ll get an error like

dbbtt not found

good point

Why don’t we try this ?

still no change 🤔

That’s really weird. I get logs with

ls

. I’m thinking the same. I’m on a mac. Are you on a linux or mac? There were changes to the ShellTask in 0.14.15 and 0.14.16 but I don’t think they should be relevant for this. Just checking, what Prefect version are you on?

ah

Ah: https://github.com/PrefectHQ/prefect/pull/4397

That indeed seems to explain it 🙂

Ok let’s hope DbtShellTask gives some informative messages now 🤞

So I need to add bash exe to path?

This fix was released in 0.14.16: https://github.com/PrefectHQ/prefect/pull/4403

got it

That’s a good suggestion. I can bring this up with the team and maybe open an issue about it

Thank you for the support!

Hi, I'll reopen this thread again 🙂 I have the same issues as Robin but add bash to path does not work so far for me and I use prefect 0.14.21 so the shell fix from april should be included imho. Thats my code

with Flow(name="dbt_flow") as f:
    task = DbtShellTask(
        log_stderr=True,
        log_stdout=True,
        return_all=True,
        stream_output=True,
        profiles_dir=profile_path

    )(command='dbt run --models myModel')

out = f.run()

That's the output

[2021-06-09 16:06:15+0200] INFO - prefect.FlowRunner | Beginning Flow run for 'dbt_flow'
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2021-06-09 16:06:15+0200] INFO - prefect.DbtShellTask | /bin/bash: C:\Users\M7856~1.\AppData\Local\Temp\prefect-5yoqhqg_: No such file or directory
[2021-06-09 16:06:15+0200] ERROR - prefect.DbtShellTask | Command failed with exit code 127
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 127')
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
[2021-06-09 16:06:15+0200] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.

What I did wrong? Thanks for help Best Michael

So you’re on Windows and adding bash to the path did not work in running the DbtTask?

maybe it will be a littlebit emberrassing for me 😄 but I just added in the enviorment variables on windows to path: C:\Windows\System32\bash.exe

I think that’s right…can you go in the command line and try using it to confirm it works?

yes I enterd bash in cmd and it starts the bash

What error do you get when using the ShellTask?

the same only with ShellTask.... but he is trying to access "c:\users\ ..." should that not be like "mnt/c/user/ ..." is that some how possible to change the path?

This is a bit beyond me. Can you open a new thread in the community channel so I can direct it to the team easily?

done