https://prefect.io logo
Title
v

Vincenzo

03/14/2023, 4:36 PM
[THEORETICAL QUESTION] Hi everyone, new to DBT / Prefect and Data Engineering. I am currently building my first DE capstone project where I am pulling data from an API, and pushing it to BQ. Orchestration happens with Prefect. I want to use DBT to partition and prepare the data in BQ for Looker. I am having issues wrapping my head around how I can make my code reproducible for others on GitHub as I am using DBT cloud. I found repo from Anna but it does not fully answer my question on how to make the code reproducible (especially in terms of DBT credentials). What would be the best practices here? Would I need to explain them how to setup the DBT cloud as well or is there a better way?
1
s

Sean Williams

03/15/2023, 5:03 PM
Have you seen prefect-dbt? You can save your credentials to a block, so you don't have to have your credentials in your code on github.
v

Vincenzo

03/19/2023, 2:19 PM
@Sean Williams thanks for this, yes I have seen it. I assume that means that I would include the dbt-transformation step as a step within my flow / new deployment. I will have a look into this thanks.
s

Sean Williams

03/20/2023, 3:45 PM
Yep. If you're using dbt Cloud, you can call
run_dbt_cloud_job()
from prefect-dbt to trigger that job.
v

Vincenzo

03/21/2023, 9:07 AM
Thanks @Sean Williams I tried it but it does not work for me, I am trying to find the issue since yesterday without luck. https://prefect-community.slack.com/archives/C03CY667GGM/p1679389532247009