https://prefect.io
Join Slack
Hey Everyone - For those using <DbtShellTask>, is it expected to have to run `dbt compile` before ex...
# ask-community
r
Hey Everyone - For those using DbtShellTask, is it expected to have to run 
dbt compile
before executing 
dbt run
?
a
Hi @Ryan Brennan it’s the first time I hear about dbt compile 😄 I didn’t have to use it. Here is a demo in a blog post: https://www.prefect.io/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/
r
Thanks @Anna Geller! I’ll take a look
For some more context my flow looks like this
Copy code
with Flow(FLOW_NAME) as flow:
    pull_repo = pull_from_git(repository=github_task.task_config['repository'])
    deps = dbt(
        command="dbt deps",
        upstream_tasks=[pull_repo],
    )
    run = dbt(
        command='dbt run -m +<MODEL_NAME>',
        upstream_tasks=[pull_repo, deps],
    )

if __name__ == '__main__':
    flow.run()
If I just run this I get a model specification error saying that the dbt node <MODEL_NAME> doesn’t exist
but if I change the first command to 
command="dbt deps & dbt compile"
it works
a
did you cd into dbt project directory?
on the dbt shell task:
Copy code
helper_script=f"cd {DBT_PROJECT}",
r
Yep, my task looks like this
Copy code
dbt = DbtShellTask(
    return_all=True,
    profile_name='<PROFILE_NAME>',
    environment='prod',
    set_profiles_envar=False,
    overwrite_profiles=False,
    log_stdout=True,
    helper_script='cd dbt',
    log_stderr=True,
    dbt_kwargs={<DICT OF SECRETS>
    },
… now it’s working 🤷‍♂️
a
interesting. I always use:
Copy code
overwrite_profiles=True,
this ensures that a new clean dbt profile gets created based on the kwargs set on this task. And since I saw you clone the repository anyway, it should do no harm to overwrite it. Maybe you can give it a try.
it’s a demo effect 😄 nice that it’s working now!
r
I don’t think 
overwrite_profiles=True,
will work for us because we’re an agency and work across 10+ dbt/prefect instances
so overwriting profiles.yml would remove all of the other creds too
But seems to be working now, thanks your help!
🙌 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by