https://prefect.io logo
#prefect-community
Title
# prefect-community
p

Paul Butler

03/07/2022, 5:33 PM
Copy code
Question for the Community - I am trying to use Prefect to schedule and monitor a Dbt project/pipeline.   I follow example and pass in dbt_kwargs for connecting to snowflake.  My dbt project runs fine with dbt run or dbt compile command, but the DbtShellTask fails - yet does not provide any reason for error.  Anywhere more detailed logging is recorded??  

Sample tasks like the hello one included here, run OK.   I'm using Studio Code to debug this code, but also try to run it in Python IDLE and get same error output     I'm running this:

from prefect import task, Flow, Parameter
from prefect.tasks.shell import ShellTask
from prefect.tasks.dbt import DbtShellTask

@task(log_stdout=True)
def say_hi(name):
    print("hello {}!".format(name))

with Flow(name="dbt_flow") as f:
    name = Parameter('name')
    say_hi(name)

    task = DbtShellTask(
        profile_name='default',
        environment='dev',
        dbt_kwargs={
            'type': 'snowflake',
            'threads': 4,
            'account': 'mysnowflake.account',
            'user': '<mailto:myemal@myco.com|myemal@myco.com>',
            'authenticator': 'externalbrowser',
            'role': 'ROLENAME',
            'database': 'DBNAME',
            'warehouse': 'ENGINEERING_XS',
            'schema': 'DV_PROTO'
        },
        overwrite_profiles=False,
        profiles_dir='C:\\Users\myDBTuser\.dbt'
    )(command='dbt compile')

out = f.run(name='Paul') 

[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2022-03-07 17:18:07+0000] ERROR - prefect.DbtShellTask | Command failed with exit code 1
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 1')
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Starting task run...
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | hello Paul!
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Finished task run for task with final state: 'Success'
k

Kevin Kho

03/07/2022, 5:38 PM
Hi @Paul Butler, could we move the code to the thread to keep the main channel cleaner? For the
DbtShellTask
, you can add more logging with some combination of these :
Copy code
DbtShellTask(..., log_stderr=True, return_all=True, stream_output=True)
p

Paul Butler

03/07/2022, 5:45 PM
Thanks. It says bash not recognized. I'm on a Windows VM. Do I need to install bash? running Python 3.7
I installed Prefect via "pip install prefect"
k

Kevin Kho

03/07/2022, 5:48 PM
So
DbtShellTask
uses the
ShellTask
under the hood which needs
bash
installed. You can find more info here
And then the example in the comment below
p

Paul Butler

03/07/2022, 6:28 PM
Thank you! using the shell argument worked. I passed in full path to bash on my VM, as in: shell=r"C:\\Program Files\\Git\\bin\\bash.exe"
k

Kevin Kho

03/07/2022, 6:31 PM
Nice!
7 Views