https://prefect.io logo
Powered by
Title
a

Aaron Gonzalez

01/11/2023, 3:19 PM
Newbie question: I've wondering how people have used the 
BigQueryTargetConfigs
prefect-dbt block .
Take the following blocks code: 
parser = ArgumentParser()
parser.add_argument('--env')
args = parser.parse_args()
ENV = args.env


credentials = GcpCredentials.load('prefect-srvc-usr')
bq_target_dict = {
    'schema': f'zone_hawk_{ENV}',
    'project': 'my_gcp_project',
    'threads': 20,
    'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict).save(f'bq-target-configs-{ENV}')

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': BigQueryTargetConfigs.load(f'bq-target-configs-{ENV}')
}
dbt_cli_profile = DbtCliProfile(
    **dbt_cli_dict
).save(f'dbt-cli-profile-{ENV}', overwrite=True)
I can see that I have a BigQueryTargetConfigs and DbtCliProfile blocks now available in my Prefect Cloud UI. ✅ But! If I follow this up with the following: 
dbt_cli_profile = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")
And then call the 
dbt_cli_profile.get_profile()
method my GcpCredentials block in the BigQueryTargetConfigs is gone. 
{'config': {},
 'bigbird': {'target': 'dev',
  'outputs': {'dev': {'type': 'bigquery',
    'schema': 'zone_hawk_dev',
    'threads': 20}}}}
I can see that the DbtCliProfile class only accepts a target_configs attr of type 
TargetConfigs
.......so I've just been wondering what the potential use of 
BigQueryTargetConfigs
is? 😕
Actually it looks like this might be related to issue 104. https://github.com/PrefectHQ/prefect-dbt/issues/104 Was able to work around it for now.
a

Andrew Huang

01/11/2023, 5:46 PM
BigQueryTargetConfigs inherits from 
TargetConfigs
so it should be accepted. The gcp credentials block “disappearing” should be expected because it should convert to “method” https://github.com/PrefectHQ/prefect-dbt/blob/main/prefect_dbt/cli/configs/bigquery.py#L105-L121 however, the output config you show seems like it did not work as expected. can you provide how you instantiated GcpCredentials?
a

Aaron Gonzalez

01/11/2023, 5:50 PM
Thanks for reaching out @Andrew Huang. My GcpCredentials block is created & overwritten by a gh action step. 
- name: Deploy Blocks to Prefect Cloud
        id: build-blocks
        working-directory: ${{ env.WORKINGDIR }}
        run: |
          cat <<EOF > gcp_cred_block.py
          from prefect_gcp import GcpCredentials

          service_account_info = ${{ secrets.GCP_CREDENTIALS }}
          gcp_creds = GcpCredentials(service_account_info=service_account_info)
          gcp_creds.save("prefect-srvc-usr", overwrite=True)
          EOF
          python gcp_cred_block.py
          python blocks.py --env $ENVIRONMENT
a

Andrew Huang

01/11/2023, 5:50 PM
this is what I get when I use a service account-info in my credentials 
{'config': {},
 'bigbird': {'target': 'abc',
  'outputs': {'abc': {'type': 'bigquery',
    'schema': 'zone_hawk_abc',
    'threads': 20,
    'project': 'my_gcp_project',
    'keyfile_json': {'type': 'service_account',
...
from prefect_gcp import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile

ENV = "abc"
credentials = await GcpCredentials.load('gcp-credentials')
bq_target_dict = {
    'schema': f'zone_hawk_{ENV}',
    'project': 'my_gcp_project',
    'threads': 20,
    'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict)

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': bq_target_configs
}
dbt_cli_profile = DbtCliProfile(
    **dbt_cli_dict
)
dbt_cli_profile.get_profile()
I am wondering if you have the latest versions? 
import prefect_dbt
import prefect_gcp
print(prefect_gcp.__version__,  prefect_dbt.__version__)
a

Aaron Gonzalez

01/11/2023, 5:55 PM
In [1]: import prefect_dbt
   ...: import prefect_gcp
   ...: print(prefect_gcp.__version__,  prefect_dbt.__version__)
0.2.3 0.2.7
I guess my question was more about why when I save and load the BigQueryTargetConfigs and DbtCliProfile blocks that 
get_profile()
output does not include those 
method
and 
keyfile_json
keys.
a

Andrew Huang

01/11/2023, 6:03 PM
yeah I’m wondering about that too. can you print credentials? do you get something like this? 
GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='myproject')
✅ 1
to clarify, does that checkmark mean you have the same?
a

Aaron Gonzalez

01/11/2023, 6:10 PM
Sorry. Yes. I get the same output as you.
a

Andrew Huang

01/11/2023, 6:11 PM
ok what if you print bq_target_configs?
does credentials still exist? 
BigQueryTargetConfigs(extras=None, type='bigquery', threads=20, project='my_gcp_project', credentials=GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='abc'))
a

Aaron Gonzalez

01/11/2023, 6:13 PM
yes. If I save, and then load the BigQueryTargetConfigs block, assign to variable bq_target_configs, and print it I do get the output that contain the GcpCredentials
a

Andrew Huang

01/11/2023, 6:14 PM
and I assume when you print dbt_cli_profile, target_configs still has credentials? 
DbtCliProfile(name='bigbird', target='abc', target_configs=BigQueryTargetConfigs(extras=None, type='bigquery', threads=20, project='my_gcp_project', credentials=GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='abc')), global_configs=None)
a

Aaron Gonzalez

01/11/2023, 6:15 PM
That is where things fall apart. 
In [18]: dbt_cli_profile2 = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")

In [19]: dbt_cli_profile2
Out[19]: DbtCliProfile(name='bigbird', target='dev', target_configs=TargetConfigs(extras=None, type='bigquery', threads=20), global_configs=None)
a

Andrew Huang

01/11/2023, 6:17 PM
when I load mine, it works
a

Aaron Gonzalez

01/11/2023, 6:18 PM
😵
a

Andrew Huang

01/11/2023, 6:18 PM
from prefect_gcp import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile

ENV = "abc"
credentials = GcpCredentials.load('gcp-credentials')
bq_target_dict = {
    'schema': f'zone_hawk_{ENV}',
    'project': 'my_gcp_project',
    'threads': 20,
    'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict)

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': bq_target_configs
}
dbt_cli_profile = DbtCliProfile(
    **dbt_cli_dict
).save("my-profile", overwrite=True)


dbt_cli_profile.load("my-profile")
perhaps you can try 
prefect block register -m prefect_dbt
and 
prefect block register -m prefect_gcp
then try saving it again and reloading
a

Aaron Gonzalez

01/11/2023, 6:18 PM
BigQueryTargetConfigs(**bq_target_dict) save and load this
a

Andrew Huang

01/11/2023, 6:18 PM
oh ok
message has been deleted
😕 still seems to exist
a

Aaron Gonzalez

01/11/2023, 6:20 PM
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict).save(f'bq-target-configs-{ENV}')

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': BigQueryTargetConfigs.load(f'bq-target-configs-{ENV}')
}
dbt_cli_profile = DbtCliProfile(**dbt_cli_dict).save(f'dbt-cli-profile-{ENV}', overwrite=True)

dbt_cli_profile2 = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")
a

Andrew Huang

01/11/2023, 6:22 PM
ooh ok I think I’m reproducing it
added an issue to track: https://github.com/PrefectHQ/prefect-dbt/issues/114
a

Aaron Gonzalez

01/11/2023, 6:27 PM
:thank-you:
a

Andrew Huang

01/11/2023, 6:34 PM
ok I think you were onto something earlier that it’s because it’s typed 
TargetConfigs
upon loading it uses that instead of 
BigQueryTargetConfigs
to load. so the workaround is that you save your BigQueryTargetConfigs to pass into dbt profile
okay I was able to fix with updating the typing: 
target_configs: Union[SnowflakeTargetConfigs, BigQueryTargetConfigs, PostgresTargetConfigs, TargetConfigs]
message has been deleted
thanks for reporting! I’ll try to get this out soon once I add tests
m

Mateo Merlo

03/14/2023, 10:55 PM
Hey, I went through the same issue and seems prefect-dbt 0.3.0 solves it.
But I'm getting a conflict of dependencies with other packages when I try to install it using pipenv
anyone has this issue?
Screenshot from 2023-03-14 19-56-31.png
28 Views
#prefect-dbtJoin Slack
Powered by