Peter Žáčik
11/10/2025, 11:46 AMdbt build from a Prefect task. The credentials to a PostgreSQL database are stored in a dbt CLIT PostgreSQL Target Config block.Marvin
11/10/2025, 11:46 AMMarvin
11/10/2025, 11:49 AMdbt build with credentials stored in a “Dbt CLI > PostgreSQL Target Configs” block is to load that block in your flow, wrap it in a DbtCliProfile, then call the pre-built run_dbt_build task from the prefect-dbt collection.
Here’s a minimal, working example:
# pip install "prefect-dbt[postgres]" (make sure your worker env has this)
from prefect import flow
from prefect_dbt.cli import DbtCliProfile, PostgresTargetConfigs
from prefect_dbt.cli.commands import run_dbt_build
@flow
def dbt_build_flow(
project_dir: str = "/path/to/your/dbt/project", # must contain dbt_project.yml
profile_name: str = "jaffle_shop", # must match 'profile' in dbt_project.yml
target_name: str = "prod" # the dbt target to use
):
# 1) Load your Postgres Target Configs block (created in Prefect UI)
pg_target = PostgresTargetConfigs.load("my-postgres-target-block")
# 2) Create a dbt CLI profile from the target configs
dbt_profile = DbtCliProfile(
name=profile_name,
target=target_name,
target_configs=pg_target,
)
# 3) Run dbt build using the profile
result = run_dbt_build(
project_dir=project_dir,
dbt_cli_profile=dbt_profile,
overwrite_profiles=True, # ensures we use the generated profiles.yml
stream_output=True, # stream CLI output to Prefect logs
# extra_command_args=["--select", "tag:daily"], # optional
)
return result
if __name__ == "__main__":
dbt_build_flow()
Notes and tips:
- Ensure your execution environment (worker/container) has dbt for Postgres installed. The easiest path is:
- pip install "prefect-dbt[postgres]" (this pulls in dbt-postgres)
- profile_name must match the profile value in your project’s dbt_project.yml.
- You do not need a local profiles.yml. The DbtCliProfile will generate one at runtime. overwrite_profiles=True makes sure that generated file is used instead of any local profiles.
- If you prefer to manage the entire dbt profile as a single block, you can create a “Dbt CLI Profile” block in Prefect that points at your Postgres Target Configs block, then just do:
from prefect_dbt.cli.credentials import DbtCliProfile
profile = DbtCliProfile.load("my-dbt-cli-profile")
run_dbt_build(project_dir="...", dbt_cli_profile=profile, overwrite_profiles=True)
Docs for reference:
- prefect-dbt integration overview
- example: Run dbt with Prefect
If you share:
- the name of your Postgres Target Configs block
- the profile in your dbt_project.yml
- the target you want to use
…I can tailor the snippet exactly to your setup.Peter Žáčik
11/10/2025, 11:52 AMDbtCliProfile and run_dbt_build imports are not foundMarvin
11/10/2025, 11:58 AMMarvin
11/10/2025, 11:59 AMpip install -U "prefect>=3" "prefect-dbt[postgres]"
2) Use the explicit import paths (these are confirmed in the current collection):
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.commands import run_dbt_build
from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs
3) Minimal working example using your Postgres Target Configs block
from prefect import flow
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs
from prefect_dbt.cli.commands import run_dbt_build
@flow
def dbt_build_flow(
project_dir: str = "/path/to/dbt/project",
profile_name: str = "jaffle_shop",
target_name: str = "prod"
):
# Load the Postgres Target Configs block you created in Prefect
pg_target = PostgresTargetConfigs.load("my-postgres-target-block")
# Build a dbt CLI profile from that block
dbt_profile = DbtCliProfile(
name=profile_name,
target=target_name,
target_configs=pg_target,
)
# Run dbt build
result = run_dbt_build(
project_dir=project_dir,
dbt_cli_profile=dbt_profile,
overwrite_profiles=True,
stream_output=True,
)
return result
If you still get import errors, could you share:
- Your Prefect version (python -c "import prefect; print(prefect.__version__)")
- Your prefect-dbt version (python -c "import prefect_dbt; print(prefect_dbt.__version__)")
- The exact import lines and the full traceback
- Where this code is running (local Python, a worker container image, etc.)
Docs for reference:
- prefect-dbt integration
- run dbt with Prefect examplePeter Žáčik
11/10/2025, 12:06 PMUserWarning: prefect_dbt.cli is deprecated and will be removed in a future release. Please use prefect_dbt.core instead.Marvin
11/10/2025, 12:06 PMMarvin
11/10/2025, 12:10 PMprefect_dbt.cli module is deprecated; the recommended path is to use prefect_dbt.core with PrefectDbtRunner.
You have two paths:
A) Recommended (no deprecation): Use PrefectDbtRunner + a profiles.yml
- PrefectDbtRunner uses dbt’s Python runner and reads credentials from a standard profiles.yml.
- You can create profiles.yml at runtime from Prefect Secret blocks (or env vars), then invoke dbt.
Example (no cli imports, no deprecation):
# pip install -U "prefect>=3" "prefect-dbt[postgres]"
from prefect import flow
from prefect.blocks.system import Secret
from prefect_dbt.core import PrefectDbtRunner
import os, tempfile
@flow
def dbt_build_flow(
project_dir: str = "/path/to/dbt/project",
profile_name: str = "jaffle_shop",
target_name: str = "prod",
):
# Load creds from Prefect Secret blocks (adjust names to yours)
host = Secret.load("pg-host").get()
user = Secret.load("pg-user").get()
password = Secret.load("pg-password").get()
port = Secret.load("pg-port").get()
dbname = Secret.load("pg-dbname").get()
schema = Secret.load("pg-schema").get()
# Create a temporary profiles.yml for dbt
profiles_dir = tempfile.mkdtemp(prefix="dbt-profiles-")
profiles_yml = f"""
{profile_name}:
target: {target_name}
outputs:
{target_name}:
type: postgres
host: {host}
user: {user}
password: {password}
port: {port}
dbname: {dbname}
schema: {schema}
threads: 4
"""
with open(os.path.join(profiles_dir, "profiles.yml"), "w") as f:
f.write(profiles_yml)
# Run `dbt build` via PrefectDbtRunner
runner = PrefectDbtRunner()
result = runner.invoke([
"build",
"--project-dir", project_dir,
"--profiles-dir", profiles_dir,
"--profile", profile_name,
"--target", target_name,
])
return result
if __name__ == "__main__":
dbt_build_flow()
Notes:
- Ensure your worker/runtime image has dbt-postgres: pip install "prefect-dbt[postgres]".
- If your dbt project already includes a profiles.yml, you can skip generating it and just pass --profiles-dir (or rely on DBT_PROFILES_DIR).
B) Keep your existing dbt CLI blocks (works today, but deprecated)
- You can continue using:
- from prefect_dbt.cli.credentials import DbtCliProfile
- from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs
- from prefect_dbt.cli.commands import run_dbt_build
- This will emit the deprecation warning but remains functional for now. It’s fine as a bridge while you migrate.
Which route would you prefer?
- If you want to keep using your existing “PostgreSQL Target Configs” block as-is, I can show you how to wire it up quickly (you’ll see the deprecation warning until you migrate).
- If you’d like to move to PrefectDbtRunner now, I can help translate your block’s values into a runtime profiles.yml or advise on storing them as Secrets/env vars.
Docs:
- prefect-dbt integration overview
- Run dbt with Prefect examplePeter Žáčik
11/10/2025, 12:22 PMMarvin
11/10/2025, 12:22 PMMarvin
11/10/2025, 12:23 PM# pip install -U "prefect>=3" "prefect-dbt[postgres]"
from prefect import flow
from prefect_dbt.core import PrefectDbtRunner
# NOTE: these classes live in the deprecated module; we're only using them
# to read your existing block and write a profiles.yml.
from prefect_dbt.cli.configs.postgres import PostgresTargetConfigs
import os
import tempfile
@flow
def dbt_build_with_profile_block(
target_block_name: str = "my-postgres-target-block",
project_dir: str = "/path/to/dbt/project",
profile_name: str = "jaffle_shop", # must match your dbt_project.yml 'profile'
target_name: str = "prod"
):
pg = PostgresTargetConfigs.load(target_block_name)
# Safely extract fields (dbt uses 'dbname' for postgres; some configs call it 'database')
host = getattr(pg, "host", None)
user = getattr(pg, "user", None)
password = getattr(pg, "password", None)
try:
# If it's a pydantic SecretStr
password = password.get_secret_value()
except Exception:
pass
port = getattr(pg, "port", 5432)
dbname = getattr(pg, "dbname", getattr(pg, "database", None))
schema = getattr(pg, "schema", "public")
threads = getattr(pg, "threads", 4)
# Write a temporary profiles.yml for dbt
profiles_dir = tempfile.mkdtemp(prefix="dbt-profiles-")
profiles_yml = f"""
{profile_name}:
target: {target_name}
outputs:
{target_name}:
type: postgres
host: {host}
user: {user}
password: {password}
port: {port}
dbname: {dbname}
schema: {schema}
threads: {threads}
"""
with open(os.path.join(profiles_dir, "profiles.yml"), "w") as f:
f.write(profiles_yml)
# Run `dbt build` via PrefectDbtRunner (no deprecated CLI usage here)
runner = PrefectDbtRunner()
result = runner.invoke([
"build",
"--project-dir", project_dir,
"--profiles-dir", profiles_dir,
"--profile", profile_name,
"--target", target_name,
])
return result
Important notes:
- This approach uses your existing block for credential storage, but still emits a deprecation warning at import time because the Target Configs class lives in prefect_dbt.cli. It’s only used to read values; the actual run uses PrefectDbtRunner.
- If you want to avoid the deprecation entirely, migrate the credentials to Prefect Secrets/env vars and render profiles.yml without importing anything from prefect_dbt.cli. I can share a snippet for that too.
- Ensure your runtime has dbt-postgres: pip install "prefect-dbt[postgres]".
If you share your block name and your dbt_project.yml profile and target, I can tailor the snippet exactly. References:
- prefect-dbt integration overview
- Run dbt with Prefect example