<@ULVA73B9P> I want to trigger `dbt build` from a ...
# ask-marvin
p
@Marvin I want to trigger
dbt build
from a Prefect task. The credentials to a PostgreSQL database are stored in a dbt CLIT PostgreSQL Target Config block.
m
thought for 212.8 seconds
Great use case. In Prefect 3.x, the simplest way to run
dbt build
with credentials stored in a “Dbt CLI > PostgreSQL Target Configs” block is to load that block in your flow, wrap it in a
DbtCliProfile
, then call the pre-built
run_dbt_build
task from the prefect-dbt collection. Here’s a minimal, working example:
Copy code
# pip install "prefect-dbt[postgres]"  (make sure your worker env has this)

from prefect import flow
from prefect_dbt.cli import DbtCliProfile, PostgresTargetConfigs
from prefect_dbt.cli.commands import run_dbt_build

@flow
def dbt_build_flow(
    project_dir: str = "/path/to/your/dbt/project",  # must contain dbt_project.yml
    profile_name: str = "jaffle_shop",               # must match 'profile' in dbt_project.yml
    target_name: str = "prod"                        # the dbt target to use
):
    # 1) Load your Postgres Target Configs block (created in Prefect UI)
    pg_target = PostgresTargetConfigs.load("my-postgres-target-block")

    # 2) Create a dbt CLI profile from the target configs
    dbt_profile = DbtCliProfile(
        name=profile_name,
        target=target_name,
        target_configs=pg_target,
    )

    # 3) Run dbt build using the profile
    result = run_dbt_build(
        project_dir=project_dir,
        dbt_cli_profile=dbt_profile,
        overwrite_profiles=True,   # ensures we use the generated profiles.yml
        stream_output=True,        # stream CLI output to Prefect logs
        # extra_command_args=["--select", "tag:daily"],  # optional
    )
    return result

if __name__ == "__main__":
    dbt_build_flow()
Notes and tips: - Ensure your execution environment (worker/container) has dbt for Postgres installed. The easiest path is: -
pip install "prefect-dbt[postgres]"
(this pulls in dbt-postgres) -
profile_name
must match the
profile
value in your project’s dbt_project.yml. - You do not need a local profiles.yml. The
DbtCliProfile
will generate one at runtime.
overwrite_profiles=True
makes sure that generated file is used instead of any local profiles. - If you prefer to manage the entire dbt profile as a single block, you can create a “Dbt CLI Profile” block in Prefect that points at your Postgres Target Configs block, then just do:
Copy code
from prefect_dbt.cli.credentials import DbtCliProfile
  profile = DbtCliProfile.load("my-dbt-cli-profile")
  run_dbt_build(project_dir="...", dbt_cli_profile=profile, overwrite_profiles=True)
Docs for reference: - prefect-dbt integration overview - example: Run dbt with Prefect If you share: - the name of your Postgres Target Configs block - the
profile
in your dbt_project.yml - the
target
you want to use …I can tailor the snippet exactly to your setup.
p
@Marvin
DbtCliProfile
and
run_dbt_build
imports are not found
m
thought for 28.9 seconds
Got it — if those imports aren’t found, it’s usually one of: - The prefect-dbt collection isn’t installed in the same environment where your flow runs - The collection version is older and doesn’t include those symbols - Import paths are slightly off Quick fixes to try: 1) Install/upgrade the collection (and dbt-postgres) in the same environment as your worker/runner
Copy code
pip install -U "prefect>=3" "prefect-dbt[postgres]"
2) Use the explicit import paths (these are confirmed in the current collection):
Copy code
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.commands import run_dbt_build
from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs
3) Minimal working example using your Postgres Target Configs block
Copy code
from prefect import flow
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs
from prefect_dbt.cli.commands import run_dbt_build

@flow
def dbt_build_flow(
    project_dir: str = "/path/to/dbt/project",
    profile_name: str = "jaffle_shop",
    target_name: str = "prod"
):
    # Load the Postgres Target Configs block you created in Prefect
    pg_target = PostgresTargetConfigs.load("my-postgres-target-block")

    # Build a dbt CLI profile from that block
    dbt_profile = DbtCliProfile(
        name=profile_name,
        target=target_name,
        target_configs=pg_target,
    )

    # Run dbt build
    result = run_dbt_build(
        project_dir=project_dir,
        dbt_cli_profile=dbt_profile,
        overwrite_profiles=True,
        stream_output=True,
    )
    return result
If you still get import errors, could you share: - Your Prefect version (
python -c "import prefect; print(prefect.__version__)"
) - Your prefect-dbt version (
python -c "import prefect_dbt; print(prefect_dbt.__version__)"
) - The exact import lines and the full traceback - Where this code is running (local Python, a worker container image, etc.) Docs for reference: - prefect-dbt integration - run dbt with Prefect example
p
@Marvin
Copy code
UserWarning: prefect_dbt.cli is deprecated and will be removed in a future release. Please use prefect_dbt.core instead.
m
thought for 274.4 seconds
Thanks for flagging that — the warning is expected. The
prefect_dbt.cli
module is deprecated; the recommended path is to use
prefect_dbt.core
with
PrefectDbtRunner
. You have two paths: A) Recommended (no deprecation): Use PrefectDbtRunner + a profiles.yml - PrefectDbtRunner uses dbt’s Python runner and reads credentials from a standard
profiles.yml
. - You can create
profiles.yml
at runtime from Prefect Secret blocks (or env vars), then invoke dbt. Example (no cli imports, no deprecation):
Copy code
# pip install -U "prefect>=3" "prefect-dbt[postgres]"

from prefect import flow
from prefect.blocks.system import Secret
from prefect_dbt.core import PrefectDbtRunner
import os, tempfile

@flow
def dbt_build_flow(
    project_dir: str = "/path/to/dbt/project",
    profile_name: str = "jaffle_shop",
    target_name: str = "prod",
):
    # Load creds from Prefect Secret blocks (adjust names to yours)
    host = Secret.load("pg-host").get()
    user = Secret.load("pg-user").get()
    password = Secret.load("pg-password").get()
    port = Secret.load("pg-port").get()
    dbname = Secret.load("pg-dbname").get()
    schema = Secret.load("pg-schema").get()

    # Create a temporary profiles.yml for dbt
    profiles_dir = tempfile.mkdtemp(prefix="dbt-profiles-")
    profiles_yml = f"""
{profile_name}:
  target: {target_name}
  outputs:
    {target_name}:
      type: postgres
      host: {host}
      user: {user}
      password: {password}
      port: {port}
      dbname: {dbname}
      schema: {schema}
      threads: 4
"""
    with open(os.path.join(profiles_dir, "profiles.yml"), "w") as f:
        f.write(profiles_yml)

    # Run `dbt build` via PrefectDbtRunner
    runner = PrefectDbtRunner()
    result = runner.invoke([
        "build",
        "--project-dir", project_dir,
        "--profiles-dir", profiles_dir,
        "--profile", profile_name,
        "--target", target_name,
    ])
    return result

if __name__ == "__main__":
    dbt_build_flow()
Notes: - Ensure your worker/runtime image has dbt-postgres:
pip install "prefect-dbt[postgres]"
. - If your dbt project already includes a
profiles.yml
, you can skip generating it and just pass
--profiles-dir
(or rely on
DBT_PROFILES_DIR
). B) Keep your existing dbt CLI blocks (works today, but deprecated) - You can continue using: -
from prefect_dbt.cli.credentials import DbtCliProfile
-
from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs
-
from prefect_dbt.cli.commands import run_dbt_build
- This will emit the deprecation warning but remains functional for now. It’s fine as a bridge while you migrate. Which route would you prefer? - If you want to keep using your existing “PostgreSQL Target Configs” block as-is, I can show you how to wire it up quickly (you’ll see the deprecation warning until you migrate). - If you’d like to move to
PrefectDbtRunner
now, I can help translate your block’s values into a runtime
profiles.yml
or advise on storing them as Secrets/env vars. Docs: - prefect-dbt integration overview - Run dbt with Prefect example
p
@Marvin Can I use a dbt profile block + PrefectDbtRunner ?
m
thought for 46.6 seconds
Short answer: not directly. PrefectDbtRunner does not accept a Prefect “Dbt CLI Profile” (or “PostgreSQL Target Configs”) block object. It reads dbt configuration the same way dbt does: from a profiles.yml on disk. However, you can bridge them: - Load your existing Target Configs (or Dbt CLI Profile) block - Materialize a profiles.yml from the block’s fields - Point PrefectDbtRunner at that profiles.yml Here’s a minimal bridge using your PostgreSQL Target Configs block, then running PrefectDbtRunner. This keeps your credentials in the block, but avoids the deprecated CLI runner:
Copy code
# pip install -U "prefect>=3" "prefect-dbt[postgres]"

from prefect import flow
from prefect_dbt.core import PrefectDbtRunner

# NOTE: these classes live in the deprecated module; we're only using them
# to read your existing block and write a profiles.yml.
from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs

import os
import tempfile

@flow
def dbt_build_with_profile_block(
    target_block_name: str = "my-postgres-target-block",
    project_dir: str = "/path/to/dbt/project",
    profile_name: str = "jaffle_shop",  # must match your dbt_project.yml 'profile'
    target_name: str = "prod"
):
    pg = PostgresTargetConfigs.load(target_block_name)

    # Safely extract fields (dbt uses 'dbname' for postgres; some configs call it 'database')
    host = getattr(pg, "host", None)
    user = getattr(pg, "user", None)
    password = getattr(pg, "password", None)
    try:
        # If it's a pydantic SecretStr
        password = password.get_secret_value()
    except Exception:
        pass
    port = getattr(pg, "port", 5432)
    dbname = getattr(pg, "dbname", getattr(pg, "database", None))
    schema = getattr(pg, "schema", "public")
    threads = getattr(pg, "threads", 4)

    # Write a temporary profiles.yml for dbt
    profiles_dir = tempfile.mkdtemp(prefix="dbt-profiles-")
    profiles_yml = f"""
{profile_name}:
  target: {target_name}
  outputs:
    {target_name}:
      type: postgres
      host: {host}
      user: {user}
      password: {password}
      port: {port}
      dbname: {dbname}
      schema: {schema}
      threads: {threads}
"""
    with open(os.path.join(profiles_dir, "profiles.yml"), "w") as f:
        f.write(profiles_yml)

    # Run `dbt build` via PrefectDbtRunner (no deprecated CLI usage here)
    runner = PrefectDbtRunner()
    result = runner.invoke([
        "build",
        "--project-dir", project_dir,
        "--profiles-dir", profiles_dir,
        "--profile", profile_name,
        "--target", target_name,
    ])
    return result
Important notes: - This approach uses your existing block for credential storage, but still emits a deprecation warning at import time because the Target Configs class lives in
prefect_dbt.cli
. It’s only used to read values; the actual run uses
PrefectDbtRunner
. - If you want to avoid the deprecation entirely, migrate the credentials to Prefect Secrets/env vars and render profiles.yml without importing anything from
prefect_dbt.cli
. I can share a snippet for that too. - Ensure your runtime has dbt-postgres:
pip install "prefect-dbt[postgres]"
. If you share your block name and your dbt_project.yml
profile
and
target
, I can tailor the snippet exactly. References: - prefect-dbt integration overview - Run dbt with Prefect example