Hey I struggle a little bit with DBT Do I need to add dbt co Prefect Community #ask-community

Hey ! I struggle a little bit with DBT Do I need ...

Maverick Humbert

06/24/2022, 11:55 AM

Hey ! I struggle a little bit with DBT Do I need to add dbt-core && dbt-postgres if I added prefect[dbt] ?

Kevin Kho

06/24/2022, 2:11 PM

Hey Maverick, I need to get someone with more dbt knowledge than me to respond. Could you move the code to the thread though when you get the chance?

Maverick Humbert

06/24/2022, 2:13 PM

Yes of course !

Maverick Humbert

06/24/2022, 2:13 PM

I pull my DBT project (& copy profiles.yml in ~/.dbt):

def pull_dbt_repo():

github_token = os.environ.get("GITHUB_ACCESS_TOKEN")

shutil.rmtree("dbt", ignore_errors=True)  _# Delete folder on run_

dbt_repo_name = "dbt-thesandbox"

dbt_repo = (

f"https://{github_token}:<mailto:x-oauth-basic@github.com|x-oauth-basic@github.com>/xxx/{dbt_repo_name}"

pygit2.clone_repository(dbt_repo, "dbt")

pathlib.Path(os.path.expanduser('~/.dbt/')).mkdir(parents=True, exist_ok=True)

shutil.copy('dbt/dbt_thesandbox/profiles.yml', os.path.expanduser('~/.dbt/'))

I setup the dbt task (& link profiles dir to ~/.dbt):

dbt = DbtShellTask(

profiles_dir=os.path.expanduser('~/.dbt/'),

log_stdout=True,

log_stderr=True,

overwrite_profiles=True,

profile_name='dbt_thesandbox',

helper_script="cd dbt/dbt_thesandbox",

Execute a specific dbt model:

try:

history_marts = dbt(

command='dbt run --select models/marts/tsb_history',

upstream_tasks=[transfer_final]

except Exception as e:

print(e)

Result (with prefect[dbt]):

/tmp/prefect-q5dqm2jn: line 2: dbt: command not found

FAIL signal raised: FAIL('Command failed with exit code 127')

Result (with dbt-core & dbt-postgres dependencies + prefect[dbt] ):

Could not run dbt 'Command failed with exit code 2

BUT, it's working with

dbt debug --config-dir

& fail with

dbt run --select models/marts/tsb_history

Is my helper script wrong ? Is the path wrong somewhere ? Should I delete deps dbt-core & dbt-postgres ? Thanks for your help !

Nate

06/24/2022, 2:36 PM

hey @Maverick Humbert, there isn't actually a

dbt

extra for the

prefect

pypi distribution. you should have seen something like

Copy code

❯ pip install "prefect[dbt]"
Collecting prefect[dbt]

WARNING: prefect 1.2.2 does not provide the extra 'dbt'

when trying to install it so, you'd need to install

dbt-core

and any other dbt packages yourself to your venv

Maverick Humbert

06/24/2022, 6:05 PM

Thanks ! That's what I though so I quickly added the dependencies for that. My flow is now working, but I don't understand something. looks like my

helper_script

is not linked with my

command

--select

should be wrong with that

helper_script

right ?

dbt = DbtShellTask(

profiles_dir=os.path.join(os.path.expanduser("~"), ".dbt"),

log_stdout=True,

log_stderr=True,

overwrite_profiles=False,

profile_name='dbt_thesandbox',

helper_script="cd dbt/dbt_thesandbox",

schedule = Schedule(clocks=[CronClock("30 0 * * *")])

with Flow("[LAND's: Old & Current] Create history", schedule=schedule) as flow:

pull_repo = pull_dbt_repo()

transfer_final = tx_receipt()

history_marts = dbt(

command='dbt run --select ./dbt/dbt_thesandbox/models/marts/tsb_history',

upstream_tasks=[transfer_final]

5 Views