Paul Butler
03/07/2022, 5:33 PMQuestion for the Community - I am trying to use Prefect to schedule and monitor a Dbt project/pipeline. I follow example and pass in dbt_kwargs for connecting to snowflake. My dbt project runs fine with dbt run or dbt compile command, but the DbtShellTask fails - yet does not provide any reason for error. Anywhere more detailed logging is recorded??
Sample tasks like the hello one included here, run OK. I'm using Studio Code to debug this code, but also try to run it in Python IDLE and get same error output I'm running this:
from prefect import task, Flow, Parameter
from prefect.tasks.shell import ShellTask
from prefect.tasks.dbt import DbtShellTask
@task(log_stdout=True)
def say_hi(name):
print("hello {}!".format(name))
with Flow(name="dbt_flow") as f:
name = Parameter('name')
say_hi(name)
task = DbtShellTask(
profile_name='default',
environment='dev',
dbt_kwargs={
'type': 'snowflake',
'threads': 4,
'account': 'mysnowflake.account',
'user': '<mailto:myemal@myco.com|myemal@myco.com>',
'authenticator': 'externalbrowser',
'role': 'ROLENAME',
'database': 'DBNAME',
'warehouse': 'ENGINEERING_XS',
'schema': 'DV_PROTO'
},
overwrite_profiles=False,
profiles_dir='C:\\Users\myDBTuser\.dbt'
)(command='dbt compile')
out = f.run(name='Paul')
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2022-03-07 17:18:07+0000] ERROR - prefect.DbtShellTask | Command failed with exit code 1
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 1')
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Starting task run...
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | hello Paul!
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Finished task run for task with final state: 'Success'
Kevin Kho
DbtShellTask
, you can add more logging with some combination of these :
DbtShellTask(..., log_stderr=True, return_all=True, stream_output=True)
Paul Butler
03/07/2022, 5:45 PMKevin Kho
DbtShellTask
uses the ShellTask
under the hood which needs bash
installed. You can find more info herePaul Butler
03/07/2022, 6:28 PMKevin Kho