<@ULVA73B9P> Context: try to learn prefect-dbt。alr...
# ask-marvin
p
@Marvin Context: try to learn prefect-dbt。already set up docker-compose for server docker worker postgres pgadmin, block for credentials is set up too. Use uv pyproject.toml and baked into docker image and save in Harbor. Already write and build dbt-core . I use profiles.yml and jinja to switch prod and dev easily(credentials is in blocks). In my flow.py, what I need to import and how to write simple code to integrate dbt? What dbt flolder's I need to copy in my Dockerfile?
m
thought for 183.3 seconds
Great question — here’s the quickest, correct path to wire dbt Core into a Prefect 3.x flow and package it into your Docker image. Assumptions - You’re on Prefect 3.x with a Docker worker - You have a dbt project (dbt-core) already built - You manage prod/dev switching via
profiles.yml
+ Jinja - Credentials are stored in Prefect blocks - You’re using Postgres (adjust adapter as needed) What to import in flow.py and minimal working example - Recommended integration: PrefectDbtRunner + PrefectDbtSettings from prefect-dbt
Copy code
from prefect import flow
from prefect_dbt import PrefectDbtRunner, PrefectDbtSettings

@flow(name="dbt-build")
def dbt_build_flow(
    project_dir: str = "/opt/dbt",     # path inside your container
    profiles_dir: str = "/opt/dbt",    # where profiles.yml lives
    target: str = "dev"                # or "prod"
):
    # Configure where dbt should look
    settings = PrefectDbtSettings(
        project_dir=project_dir,
        profiles_dir=profiles_dir,
        # optional: target_path=f"{project_dir}/target",
    )

    runner = PrefectDbtRunner(settings=settings)

    # Install any packages from packages.yml (safe to skip if you don't use it)
    runner.invoke(["deps"])

    # Run your dbt workflow
    # If your profiles.yml already switches targets via Jinja, you can omit --target
    runner.invoke(["build", "--target", target])

if __name__ == "__main__":
    dbt_build_flow()
Notes - If you want to pass additional dbt args, just extend the list in
invoke
, e.g.: -
runner.invoke(["run", "--select", "tag:daily"])
-
runner.invoke(["test"])
- If you prefer env vars instead of PrefectDbtSettings, you can set
DBT_PROJECT_DIR
and
DBT_PROFILES_DIR
in the environment and do
runner = PrefectDbtRunner()
. What dbt folders/files to copy into the Docker image At minimum, include your project root and these files/folders: - Required - dbt_project.yml - profiles.yml (unless you mount it separately or generate it at runtime) - models/ - Common/optional (include if you use them) - macros/ - seeds/ - tests/ - snapshots/ - analyses/ - packages.yml - Do NOT need to copy - target/ (dbt will create it) - dbt_packages/ (will be created by
dbt deps
) Example Dockerfile snippet Adapt paths to your image layout and uv/pyproject setup.
Copy code
# Assuming your Python base image and uv are already set up
# Ensure dbt-core AND your adapter are installed (e.g., dbt-postgres)

# Install Prefect + prefect-dbt + dbt + adapter
# If using pyproject/uv, put these in your pyproject instead
RUN pip install prefect prefect-dbt dbt-core dbt-postgres

# Copy the dbt project into the image
# This directory should contain dbt_project.yml, profiles.yml, models/, etc.
COPY ./dbt_project /opt/dbt

# Optionally set where dbt looks for profile and project
ENV DBT_PROJECT_DIR=/opt/dbt
ENV DBT_PROFILES_DIR=/opt/dbt

# (Optional) If you want to prefetch packages at build time:
# RUN dbt deps --project-dir /opt/dbt --profiles-dir /opt/dbt
Adapter dependency - For Postgres: ensure
dbt-postgres
is installed in the image (or in your uv pyproject) - For other warehouses, install the appropriate adapter (e.g.,
dbt-snowflake
,
dbt-bigquery
) Using blocks/Jinja in profiles.yml - Keep your current Jinja-based switching for dev/prod; the flow example above will work as long as the
profiles.yml
is resolved at runtime in the container - If you render credentials from Prefect blocks inside
profiles.yml
, make sure your flow has access to the Prefect API (API URL and key) so the block lookups can succeed in the container Reference - prefect-dbt repository: prefect-dbt repo If you share: - the path inside the container where you plan to place the dbt project - which adapter you’re using (Postgres/Snowflake/BigQuery) I can tailor the code and Dockerfile exactly to your setup.