https://prefect.io logo
t

Triet Le

08/22/2023, 5:16 AM
Hi team, I’m setting flows for my dbt core on BigQuery following prefect-dbt docs. It appears to me that I need to declare a specific dataset for each
BigQueryTargetConfig
block in order to register a
DbtCliProfile
block ? Let’s say I have 30 datasets on prod, do I have to do this step 30 times? This sounds quite cumbersome for me so I’d appreciate if you could give me any pointer on this
Copy code
from prefect_gcp.credentials import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile

credentials = GcpCredentials.load("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")
target_configs = BigQueryTargetConfigs(
    schema="SCHEMA-NAME-PLACEHOLDER",  # also known as dataset
    credentials=credentials,
)
target_configs.save("TARGET-CONFIGS-BLOCK-NAME-PLACEHOLDER")

dbt_cli_profile = DbtCliProfile(
    name="PROFILE-NAME-PLACEHOLDER",
    target="TARGET-NAME-placeholder",
    target_configs=target_configs,
)
dbt_cli_profile.save("DBT-CLI-PROFILE-BLOCK-NAME-PLACEHOLDER")
I think I’ll stick with
trigger_dbt_cli_command
n

Nate

08/22/2023, 4:46 PM
hi @Triet Le - to be candid I don't have a ton of experience using dbt but I understand the block side of things 🙂 are you forced to save your
dbt_cli_profile
for each dataset? or could you construct / use it dynamically each time you need one?
s

Sean Williams

08/22/2023, 9:04 PM
Let’s say I have 30 datasets on prod, do I have to do this step 30 times?
Nope! You can just use one target e.g.
prod
for all of your models. Think of it as the default schema to you want to put your models in. You can override that default a few different ways in your dbt project.
🙏 2