https://prefect.io logo
r

Raghav Thirumulu

12/15/2020, 6:00 PM
hi everyone! i'm attempting to get DBT shell tasks running via prefect. attaching my code below. anyone have any ideas on what i'm getting wrong? i can confirm that the
profiles.yml
was created correctly. im able to run dbt commands with the newly created
profiles.yml
outside of the prefect package
conversely, this does run. does anyone know how to check the dbt logs for this?
h

Hui Zheng

12/15/2020, 6:16 PM
Hi Raghav, In your first example, you define the dbt task by instantiate the task class. In that way, you need to use task init(). then you do task run() in the flow.
so try this in your first example,
Copy code
marketing_task = dbtTask(
… # except command param
)
With Flow() as flow:
   marketing_task( command= "your dbt command")
r

Raghav Thirumulu

12/15/2020, 6:23 PM
hi Hui, thanks for reaching out! still no luck. i attempted
marketing_task.run('command')
within the flow as well
j

James Parham

12/15/2020, 7:30 PM
I had a similar issue, I don't recall the error message so it may be unrelated, but I ended up using the full directory path.
r

Raghav Thirumulu

12/15/2020, 7:34 PM
i don't think the issue is with the directory. i attempted to run a simple shell task (
cd ~
). i have a feeling prefect is unable to access my shell. i am working on windows so wondering if that may be a problem
j

James Parham

12/15/2020, 7:41 PM
This works for on a Mac running locally: profile_directory='/Users/****/development/dbt_projects/sf_sandbox/.dbt' with Flow(name="dbt_flow") as f: task = DbtShellTask( profile_name='sf_sandbox', environment='dev', overwrite_profiles=True, profiles_dir=profile_directory, set_profiles_envar=True, log_stderr=True, return_all=True, dbt_kwargs={ 'type': 'snowflake', 'account': '********.east-us-2.azure', # User/password auth 'user': '***********', 'password': '********', 'role': '******', 'database': '******', 'warehouse': 'COMPUTE_WH', 'schema': 'DBT' } )(command='dbt run --project-dir=/Users/********/development/dbt_projects/sf_sandbox')
r

Raghav Thirumulu

12/15/2020, 8:00 PM
@James Parham tried with the full directory. still no luck 😞
h

Hui Zheng

12/15/2020, 8:09 PM
that’s interesting. try to print the error log from DBTshellTask to see why it fails. they recently enabled that . see this issue and the its resolution.
Copy code
<https://github.com/PrefectHQ/prefect/issues/3584>
2 Views