Triet Le
08/22/2023, 5:16 AMBigQueryTargetConfig
block in order to register a DbtCliProfile
block ? Let’s say I have 30 datasets on prod, do I have to do this step 30 times? This sounds quite cumbersome for me so I’d appreciate if you could give me any pointer on this
from prefect_gcp.credentials import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile
credentials = GcpCredentials.load("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")
target_configs = BigQueryTargetConfigs(
schema="SCHEMA-NAME-PLACEHOLDER", # also known as dataset
credentials=credentials,
)
target_configs.save("TARGET-CONFIGS-BLOCK-NAME-PLACEHOLDER")
dbt_cli_profile = DbtCliProfile(
name="PROFILE-NAME-PLACEHOLDER",
target="TARGET-NAME-placeholder",
target_configs=target_configs,
)
dbt_cli_profile.save("DBT-CLI-PROFILE-BLOCK-NAME-PLACEHOLDER")
trigger_dbt_cli_command
Nate
08/22/2023, 4:46 PMdbt_cli_profile
for each dataset? or could you construct / use it dynamically each time you need one?Sean Williams
08/22/2023, 9:04 PMLet’s say I have 30 datasets on prod, do I have to do this step 30 times?Nope! You can just use one target e.g.
prod
for all of your models. Think of it as the default schema to you want to put your models in. You can override that default a few different ways in your dbt project.