Hi all I m trying to run a dbt library locally using dbt cli Prefect Community #prefect-dbt

Hi all! I'm trying to run a dbt library locally us...

Farid

02/06/2023, 10:31 PM

Hi all! I'm trying to run a dbt library locally using dbt-cli in Prefect. I've noticed an error I tend to get is:

Copy code

ERROR: Runtime Error
  Credentials in profile "snowflake_dbt_transformations", target "dev" invalid: 'database' is a required propertyDefined profiles:
 - snowflake_dbt_transformations
For more information on configuring profiles, please consult the dbt docs:

<https://docs.getdbt.com/docs/configure-your-profile>

Upon investigation, it seems like the dbtCliProfile passed to the dbtTrigger task gets saved in the

~/.dbt/profiles.yml

first and then used inside the dbtCli shell task. However the problem is the method where it saves the dbtCliProfile into a yaml file does not save the

TargetConfigs

resulting in the profile file being incomplete. Has anyone had an experience with this before and know any workarounds?

Farid

02/06/2023, 10:32 PM

I have tried passing the dbtCliProfile everytime but given it saves it as a Yaml first and then uses it for the dbt tasks, it does not work either:

Copy code

dbt_command = (
            "dbt test" if dbt_select_param is None else "dbt test " + dbt_select_param
        )
        # Executing `dbt test` to test the models and validate the "expectations" we have about values in them.
        dbt_test_result = trigger_dbt_cli_command.with_options(
            name="dbt-test",
            description="Testing the dbt models and transformations.",
        )(
            command=dbt_command,
            project_dir=DBT_REPO_LOCAL_PATH,
>>          dbt_cli_profile=dbt_cli_profile_block,
>>          overwrite_profiles=True,
        )

Farid

02/06/2023, 10:35 PM

Copy code

$ cat ~/.dbt/profiles.yml 

config: {}
snowflake_dbt_transformations:
  outputs:
    dev:
      schema: DBT_DEFAULT
      threads: 1
      type: snowflake
  target: dev

Farid

02/07/2023, 12:02 AM

I realised the recently merged Fix save load roundtrip PR resolves this issue too, so I changed the requirements file of my prefect repo to use this specific commit before an official release on pypi:

Copy code

$ cat requirements.txt

s3fs
# prefect-dbt[cli]
git+<https://github.com/PrefectHQ/prefect-dbt.git@dce47233e5c53a26922aa348f1220eba4600b439#egg=prefect-dbt[cli]>
prefect-gitlab
prefect-snowflake
dbt-snowflake

15 Views

Open in Slack

Previous Next