Paul Butler

    Paul Butler

    6 months ago
    Question for the Community - I am trying to use Prefect to schedule and monitor a Dbt project/pipeline.   I follow example and pass in dbt_kwargs for connecting to snowflake.  My dbt project runs fine with dbt run or dbt compile command, but the DbtShellTask fails - yet does not provide any reason for error.  Anywhere more detailed logging is recorded??  
    
    Sample tasks like the hello one included here, run OK.   I'm using Studio Code to debug this code, but also try to run it in Python IDLE and get same error output     I'm running this:
    
    from prefect import task, Flow, Parameter
    from prefect.tasks.shell import ShellTask
    from prefect.tasks.dbt import DbtShellTask
    
    @task(log_stdout=True)
    def say_hi(name):
        print("hello {}!".format(name))
    
    with Flow(name="dbt_flow") as f:
        name = Parameter('name')
        say_hi(name)
    
        task = DbtShellTask(
            profile_name='default',
            environment='dev',
            dbt_kwargs={
                'type': 'snowflake',
                'threads': 4,
                'account': 'mysnowflake.account',
                'user': '<mailto:myemal@myco.com|myemal@myco.com>',
                'authenticator': 'externalbrowser',
                'role': 'ROLENAME',
                'database': 'DBNAME',
                'warehouse': 'ENGINEERING_XS',
                'schema': 'DV_PROTO'
            },
            overwrite_profiles=False,
            profiles_dir='C:\\Users\myDBTuser\.dbt'
        )(command='dbt compile')
    
    out = f.run(name='Paul') 
    
    [2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
    [2022-03-07 17:18:07+0000] ERROR - prefect.DbtShellTask | Command failed with exit code 1
    [2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 1')
    [2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
    [2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Starting task run...
    [2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | hello Paul!
    [2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Finished task run for task with final state: 'Success'
    Kevin Kho

    Kevin Kho

    6 months ago
    Hi @Paul Butler, could we move the code to the thread to keep the main channel cleaner? For the
    DbtShellTask
    , you can add more logging with some combination of these :
    DbtShellTask(..., log_stderr=True, return_all=True, stream_output=True)
    Paul Butler

    Paul Butler

    6 months ago
    Thanks. It says bash not recognized. I'm on a Windows VM. Do I need to install bash? running Python 3.7
    I installed Prefect via "pip install prefect"
    Kevin Kho

    Kevin Kho

    6 months ago
    So
    DbtShellTask
    uses the
    ShellTask
    under the hood which needs
    bash
    installed. You can find more info here
    And then the example in the comment below
    Paul Butler

    Paul Butler

    6 months ago
    Thank you! using the shell argument worked. I passed in full path to bash on my VM, as in: shell=r"C:\Program Files\Git\bin\bash.exe"
    Kevin Kho

    Kevin Kho

    6 months ago
    Nice!