https://prefect.io logo
Join Slack
Powered by
Newbie question: I've wondering how people have us...
# prefect-dbt
a
Newbie question: I've wondering how people have used the 
BigQueryTargetConfigs
prefect-dbt block .
Take the following blocks code:
Copy code
parser = ArgumentParser()
parser.add_argument('--env')
args = parser.parse_args()
ENV = args.env


credentials = GcpCredentials.load('prefect-srvc-usr')
bq_target_dict = {
    'schema': f'zone_hawk_{ENV}',
    'project': 'my_gcp_project',
    'threads': 20,
    'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict).save(f'bq-target-configs-{ENV}')

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': BigQueryTargetConfigs.load(f'bq-target-configs-{ENV}')
}
dbt_cli_profile = DbtCliProfile(
    **dbt_cli_dict
).save(f'dbt-cli-profile-{ENV}', overwrite=True)
I can see that I have a BigQueryTargetConfigs and DbtCliProfile blocks now available in my Prefect Cloud UI. ✅ But! If I follow this up with the following:
Copy code
dbt_cli_profile = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")
And then call the 
dbt_cli_profile.get_profile()
method my GcpCredentials block in the BigQueryTargetConfigs is gone.
Copy code
{'config': {},
 'bigbird': {'target': 'dev',
  'outputs': {'dev': {'type': 'bigquery',
    'schema': 'zone_hawk_dev',
    'threads': 20}}}}
I can see that the DbtCliProfile class only accepts a target_configs attr of type 
TargetConfigs
.......so I've just been wondering what the potential use of 
BigQueryTargetConfigs
is? 😕
Actually it looks like this might be related to issue 104. https://github.com/PrefectHQ/prefect-dbt/issues/104 Was able to work around it for now.
a
BigQueryTargetConfigs inherits from 
TargetConfigs
so it should be accepted. The gcp credentials block “disappearing” should be expected because it should convert to “method” https://github.com/PrefectHQ/prefect-dbt/blob/main/prefect_dbt/cli/configs/bigquery.py#L105-L121 however, the output config you show seems like it did not work as expected. can you provide how you instantiated GcpCredentials?
a
Thanks for reaching out @Andrew Huang. My GcpCredentials block is created & overwritten by a gh action step.
Copy code
- name: Deploy Blocks to Prefect Cloud
        id: build-blocks
        working-directory: ${{ env.WORKINGDIR }}
        run: |
          cat <<EOF > gcp_cred_block.py
          from prefect_gcp import GcpCredentials

          service_account_info = ${{ secrets.GCP_CREDENTIALS }}
          gcp_creds = GcpCredentials(service_account_info=service_account_info)
          gcp_creds.save("prefect-srvc-usr", overwrite=True)
          EOF
          python gcp_cred_block.py
          python blocks.py --env $ENVIRONMENT
a
this is what I get when I use a service account-info in my credentials
Copy code
{'config': {},
 'bigbird': {'target': 'abc',
  'outputs': {'abc': {'type': 'bigquery',
    'schema': 'zone_hawk_abc',
    'threads': 20,
    'project': 'my_gcp_project',
    'keyfile_json': {'type': 'service_account',
...
Copy code
from prefect_gcp import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile

ENV = "abc"
credentials = await GcpCredentials.load('gcp-credentials')
bq_target_dict = {
    'schema': f'zone_hawk_{ENV}',
    'project': 'my_gcp_project',
    'threads': 20,
    'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict)

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': bq_target_configs
}
dbt_cli_profile = DbtCliProfile(
    **dbt_cli_dict
)
dbt_cli_profile.get_profile()
I am wondering if you have the latest versions?
Copy code
import prefect_dbt
import prefect_gcp
print(prefect_gcp.__version__,  prefect_dbt.__version__)
a
Copy code
In [1]: import prefect_dbt
   ...: import prefect_gcp
   ...: print(prefect_gcp.__version__,  prefect_dbt.__version__)
0.2.3 0.2.7
I guess my question was more about why when I save and load the BigQueryTargetConfigs and DbtCliProfile blocks that 
get_profile()
output does not include those 
method
and 
keyfile_json
keys.
a
yeah I’m wondering about that too. can you print credentials? do you get something like this?
Copy code
GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='myproject')
✅ 1
to clarify, does that checkmark mean you have the same?
a
Sorry. Yes. I get the same output as you.
a
ok what if you print bq_target_configs?
does credentials still exist?
Copy code
BigQueryTargetConfigs(extras=None, type='bigquery', threads=20, project='my_gcp_project', credentials=GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='abc'))
a
yes. If I save, and then load the BigQueryTargetConfigs block, assign to variable bq_target_configs, and print it I do get the output that contain the GcpCredentials
a
and I assume when you print dbt_cli_profile, target_configs still has credentials?
Copy code
DbtCliProfile(name='bigbird', target='abc', target_configs=BigQueryTargetConfigs(extras=None, type='bigquery', threads=20, project='my_gcp_project', credentials=GcpCredentials(service_account_file=None, service_account_info=SecretDict('{'type': '**********', 'auth_uri': '**********', 'client_id': '**********', 'token_uri': '**********', 'project_id': '**********', 'private_key': '**********', 'client_email': '**********', 'private_key_id': '**********', 'client_x509_cert_url': '**********', 'auth_provider_x509_cert_url': '**********'}'), project='abc')), global_configs=None)
a
That is where things fall apart.
Copy code
In [18]: dbt_cli_profile2 = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")

In [19]: dbt_cli_profile2
Out[19]: DbtCliProfile(name='bigbird', target='dev', target_configs=TargetConfigs(extras=None, type='bigquery', threads=20), global_configs=None)
a
when I load mine, it works
a
😵
a
Copy code
from prefect_gcp import GcpCredentials
from prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile

ENV = "abc"
credentials = GcpCredentials.load('gcp-credentials')
bq_target_dict = {
    'schema': f'zone_hawk_{ENV}',
    'project': 'my_gcp_project',
    'threads': 20,
    'credentials': credentials
}
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict)

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': bq_target_configs
}
dbt_cli_profile = DbtCliProfile(
    **dbt_cli_dict
).save("my-profile", overwrite=True)


dbt_cli_profile.load("my-profile")
perhaps you can try 
prefect block register -m prefect_dbt
and 
prefect block register -m prefect_gcp
then try saving it again and reloading
a
BigQueryTargetConfigs(**bq_target_dict) save and load this
a
oh ok
message has been deleted
😕 still seems to exist
a
Copy code
bq_target_configs = BigQueryTargetConfigs(**bq_target_dict).save(f'bq-target-configs-{ENV}')

dbt_cli_dict = {
    'name': 'bigbird',
    'target': f'{ENV}',
    'target_configs': BigQueryTargetConfigs.load(f'bq-target-configs-{ENV}')
}
dbt_cli_profile = DbtCliProfile(**dbt_cli_dict).save(f'dbt-cli-profile-{ENV}', overwrite=True)

dbt_cli_profile2 = DbtCliProfile.load(f"dbt-cli-profile-{ENV}")
a
ooh ok I think I’m reproducing it
a
🙏
a
ok I think you were onto something earlier that it’s because it’s typed 
TargetConfigs
upon loading it uses that instead of 
BigQueryTargetConfigs
to load. so the workaround is that you save your BigQueryTargetConfigs to pass into dbt profile
okay I was able to fix with updating the typing:
Copy code
target_configs: Union[SnowflakeTargetConfigs, BigQueryTargetConfigs, PostgresTargetConfigs, TargetConfigs]
message has been deleted
thanks for reporting! I’ll try to get this out soon once I add tests
m
Hey, I went through the same issue and seems prefect-dbt 0.3.0 solves it.
But I'm getting a conflict of dependencies with other packages when I try to install it using pipenv
anyone has this issue?
28 Views
Open in Slack
PreviousNext
Powered by