<@ULVA73B9P> Is there a way how to pass environmen...
# ask-marvin
r
@Marvin Is there a way how to pass environment variables to dbt via PrefectDbtRunner? Specifically I need to pass DB credentials to dbt. I could write the credentials to a text file and pass it to dbt but I believe that environment variables would be a bit safer.
m
thought for 96.6 seconds
Yes — and in Prefect 3.x the recommended way is to let dbt read credentials from environment variables in
profiles.yml
, while Prefect resolves the secret values for you. Short version: - Put credentials in Prefect Secret blocks - Reference them in
profiles.yml
- Run with
PrefectDbtRunner
- Prefect will resolve those references and provide them to dbt via
env_var(...)
Docs: - Prefect dbt runner guide - profiles.yml templating - dbt env_var docs Example `profiles.yml`:
Copy code
yaml
my_project:
  target: dev
  outputs:
    dev:
      type: postgres
      host: "{{ prefect.blocks.secret.db-host }}"
      user: "{{ prefect.blocks.secret.db-user }}"
      password: "{{ prefect.blocks.secret.db-password }}"
      dbname: analytics
      schema: public
      threads: 4
Then in Prefect:
Copy code
python
from pathlib import Path
from prefect import flow
from prefect_dbt import PrefectDbtRunner, PrefectDbtSettings

@flow
def run_dbt():
    settings = PrefectDbtSettings(
        project_dir=Path("./my_dbt_project"),
        profiles_dir=Path.home() / ".dbt",
    )

    runner = PrefectDbtRunner(settings=settings)
    runner.invoke(["build"])
What happens under the hood: - Prefect reads
profiles.yml
- It resolves
{{ prefect.blocks... }}
references - It rewrites them into dbt-style
{{ env_var('...') }}
- It sets those env vars for the dbt invocation So yes: you can use environment variables safely without manually writing a credentials file yourself. If you prefer, you can also set env vars directly before invoking dbt:
Copy code
python
import os
from prefect_dbt import PrefectDbtRunner

os.environ["DBT_PASSWORD"] = "secret"

PrefectDbtRunner().invoke(["build"])
and use:
Copy code
yaml
password: "{{ env_var('DBT_PASSWORD') }}"
But for Prefect-managed runs, Secret blocks are the better pattern. A couple of notes: -
PrefectDbtRunner
is the current API in Prefect 3.x. - You can control
project_dir
and
profiles_dir
through
PrefectDbtSettings
. - You do not need to manually manage a special context manager;
invoke()
handles that internally. If you want, I can also show a complete example for Snowflake, Postgres, or BigQuery credentials.