Aaron Gonzalez
01/11/2023, 3:19 PMBigQueryTargetConfigs
prefect-dbt block .parser = ArgumentParser()
parser.add_argument('--env')
args = parser.parse_args()
ENV = args.env
credentials = GcpCredentials.load('prefect-srvc-usr')
bq_target_dict = {
'schema': f'zone_hawk_{ENV}',
'project': 'my_gcp_project',
'threads': 20,
'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict).save(f'bq-target-configs-{ENV}')
dbt_cli_dict = {
'name': 'bigbird',
'target': f'{ENV}',
'target_configs': BigQueryTargetConfigs.load(f'bq-target-configs-{ENV}')
}
dbt_cli_profile = DbtCliProfile(
**dbt_cli_dict
).save(f'dbt-cli-profile-{ENV}', overwrite=True)
dbt_cli_profile = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")
And then call the dbt_cli_profile.get_profile()
method my GcpCredentials block in the BigQueryTargetConfigs is gone.
{'config': {},
'bigbird': {'target': 'dev',
'outputs': {'dev': {'type': 'bigquery',
'schema': 'zone_hawk_dev',
'threads': 20}}}}
TargetConfigs
.......so I've just been wondering what the potential use of BigQueryTargetConfigs
is? 😕Andrew Huang
01/11/2023, 5:46 PMTargetConfigs
so it should be accepted. The gcp credentials block “disappearing” should be expected because it should convert to “method” https://github.com/PrefectHQ/prefect-dbt/blob/main/prefect_dbt/cli/configs/bigquery.py#L105-L121
however, the output config you show seems like it did not work as expected. can you provide how you instantiated GcpCredentials?Aaron Gonzalez
01/11/2023, 5:50 PM- name: Deploy Blocks to Prefect Cloud
id: build-blocks
working-directory: ${{ env.WORKINGDIR }}
run: |
cat <<EOF > gcp_cred_block.py
from prefect_gcp import GcpCredentials
service_account_info = ${{ secrets.GCP_CREDENTIALS }}
gcp_creds = GcpCredentials(service_account_info=service_account_info)
gcp_creds.save("prefect-srvc-usr", overwrite=True)
EOF
python gcp_cred_block.py
python blocks.py --env $ENVIRONMENT
Andrew Huang
01/11/2023, 5:50 PM{'config': {},
'bigbird': {'target': 'abc',
'outputs': {'abc': {'type': 'bigquery',
'schema': 'zone_hawk_abc',
'threads': 20,
'project': 'my_gcp_project',
'keyfile_json': {'type': 'service_account',
...
from prefect_gcp import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile
ENV = "abc"
credentials = await GcpCredentials.load('gcp-credentials')
bq_target_dict = {
'schema': f'zone_hawk_{ENV}',
'project': 'my_gcp_project',
'threads': 20,
'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict)
dbt_cli_dict = {
'name': 'bigbird',
'target': f'{ENV}',
'target_configs': bq_target_configs
}
dbt_cli_profile = DbtCliProfile(
**dbt_cli_dict
)
dbt_cli_profile.get_profile()
import prefect_dbt
import prefect_gcp
print(prefect_gcp.__version__, prefect_dbt.__version__)
Aaron Gonzalez
01/11/2023, 5:55 PMIn [1]: import prefect_dbt
...: import prefect_gcp
...: print(prefect_gcp.__version__, prefect_dbt.__version__)
0.2.3 0.2.7
get_profile()
output does not include those method
and keyfile_json
keys.Andrew Huang
01/11/2023, 6:03 PMGcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='myproject')
Aaron Gonzalez
01/11/2023, 6:10 PMAndrew Huang
01/11/2023, 6:11 PMBigQueryTargetConfigs(extras=None, type='bigquery', threads=20, project='my_gcp_project', credentials=GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='abc'))
Aaron Gonzalez
01/11/2023, 6:13 PMAndrew Huang
01/11/2023, 6:14 PMDbtCliProfile(name='bigbird', target='abc', target_configs=BigQueryTargetConfigs(extras=None, type='bigquery', threads=20, project='my_gcp_project', credentials=GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='abc')), global_configs=None)
Aaron Gonzalez
01/11/2023, 6:15 PMIn [18]: dbt_cli_profile2 = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")
In [19]: dbt_cli_profile2
Out[19]: DbtCliProfile(name='bigbird', target='dev', target_configs=TargetConfigs(extras=None, type='bigquery', threads=20), global_configs=None)
Andrew Huang
01/11/2023, 6:17 PMAaron Gonzalez
01/11/2023, 6:18 PMAndrew Huang
01/11/2023, 6:18 PMfrom prefect_gcp import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile
ENV = "abc"
credentials = GcpCredentials.load('gcp-credentials')
bq_target_dict = {
'schema': f'zone_hawk_{ENV}',
'project': 'my_gcp_project',
'threads': 20,
'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict)
dbt_cli_dict = {
'name': 'bigbird',
'target': f'{ENV}',
'target_configs': bq_target_configs
}
dbt_cli_profile = DbtCliProfile(
**dbt_cli_dict
).save("my-profile", overwrite=True)
dbt_cli_profile.load("my-profile")
prefect block register -m prefect_dbt
and prefect block register -m prefect_gcp
then try saving it again and reloadingAaron Gonzalez
01/11/2023, 6:18 PMAndrew Huang
01/11/2023, 6:18 PMAaron Gonzalez
01/11/2023, 6:20 PMbq_target_configs = BigQueryTargetConfigs(**bq_target_dict).save(f'bq-target-configs-{ENV}')
dbt_cli_dict = {
'name': 'bigbird',
'target': f'{ENV}',
'target_configs': BigQueryTargetConfigs.load(f'bq-target-configs-{ENV}')
}
dbt_cli_profile = DbtCliProfile(**dbt_cli_dict).save(f'dbt-cli-profile-{ENV}', overwrite=True)
dbt_cli_profile2 = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")
Andrew Huang
01/11/2023, 6:22 PMAaron Gonzalez
01/11/2023, 6:27 PMAndrew Huang
01/11/2023, 6:34 PMTargetConfigs
upon loading it uses that instead of BigQueryTargetConfigs
to load. so the workaround is that you save your BigQueryTargetConfigs to pass into dbt profiletarget_configs: Union[SnowflakeTargetConfigs, BigQueryTargetConfigs, PostgresTargetConfigs, TargetConfigs]
Mateo Merlo
03/14/2023, 10:55 PM