poheng
12/04/2025, 10:38 AMMarvin
12/04/2025, 10:38 AMMarvin
12/04/2025, 10:41 AMprofiles.yml + Jinja
- Credentials are stored in Prefect blocks
- You’re using Postgres (adjust adapter as needed)
What to import in flow.py and minimal working example
- Recommended integration: PrefectDbtRunner + PrefectDbtSettings from prefect-dbt
from prefect import flow
from prefect_dbt import PrefectDbtRunner, PrefectDbtSettings
@flow(name="dbt-build")
def dbt_build_flow(
project_dir: str = "/opt/dbt", # path inside your container
profiles_dir: str = "/opt/dbt", # where profiles.yml lives
target: str = "dev" # or "prod"
):
# Configure where dbt should look
settings = PrefectDbtSettings(
project_dir=project_dir,
profiles_dir=profiles_dir,
# optional: target_path=f"{project_dir}/target",
)
runner = PrefectDbtRunner(settings=settings)
# Install any packages from packages.yml (safe to skip if you don't use it)
runner.invoke(["deps"])
# Run your dbt workflow
# If your profiles.yml already switches targets via Jinja, you can omit --target
runner.invoke(["build", "--target", target])
if __name__ == "__main__":
dbt_build_flow()
Notes
- If you want to pass additional dbt args, just extend the list in invoke, e.g.:
- runner.invoke(["run", "--select", "tag:daily"])
- runner.invoke(["test"])
- If you prefer env vars instead of PrefectDbtSettings, you can set DBT_PROJECT_DIR and DBT_PROFILES_DIR in the environment and do runner = PrefectDbtRunner().
What dbt folders/files to copy into the Docker image
At minimum, include your project root and these files/folders:
- Required
- dbt_project.yml
- profiles.yml (unless you mount it separately or generate it at runtime)
- models/
- Common/optional (include if you use them)
- macros/
- seeds/
- tests/
- snapshots/
- analyses/
- packages.yml
- Do NOT need to copy
- target/ (dbt will create it)
- dbt_packages/ (will be created by dbt deps)
Example Dockerfile snippet
Adapt paths to your image layout and uv/pyproject setup.
# Assuming your Python base image and uv are already set up
# Ensure dbt-core AND your adapter are installed (e.g., dbt-postgres)
# Install Prefect + prefect-dbt + dbt + adapter
# If using pyproject/uv, put these in your pyproject instead
RUN pip install prefect prefect-dbt dbt-core dbt-postgres
# Copy the dbt project into the image
# This directory should contain dbt_project.yml, profiles.yml, models/, etc.
COPY ./dbt_project /opt/dbt
# Optionally set where dbt looks for profile and project
ENV DBT_PROJECT_DIR=/opt/dbt
ENV DBT_PROFILES_DIR=/opt/dbt
# (Optional) If you want to prefetch packages at build time:
# RUN dbt deps --project-dir /opt/dbt --profiles-dir /opt/dbt
Adapter dependency
- For Postgres: ensure dbt-postgres is installed in the image (or in your uv pyproject)
- For other warehouses, install the appropriate adapter (e.g., dbt-snowflake, dbt-bigquery)
Using blocks/Jinja in profiles.yml
- Keep your current Jinja-based switching for dev/prod; the flow example above will work as long as the profiles.yml is resolved at runtime in the container
- If you render credentials from Prefect blocks inside profiles.yml, make sure your flow has access to the Prefect API (API URL and key) so the block lookups can succeed in the container
Reference
- prefect-dbt repository: prefect-dbt repo
If you share:
- the path inside the container where you plan to place the dbt project
- which adapter you’re using (Postgres/Snowflake/BigQuery)
I can tailor the code and Dockerfile exactly to your setup.