Mark Cottam
03/16/2026, 2:19 PMMarvin
03/16/2026, 2:19 PMMarvin
03/16/2026, 2:22 PMprofiles.yml
- dbt Cloud (recommended if you already manage jobs in dbt Cloud): Use prefect-dbt’s dbt Cloud tasks to trigger and wait on jobs
Below are concise, production-ready patterns for both.
1) dbt Core + Snowflake (PrefectDbtRunner)
- Install
pip install -U prefect prefect-dbt dbt-core dbt-snowflake
# Optional (for Snowflake Blocks if you want them elsewhere): pip install -U prefect-snowflake
# Optional (to create blocks from the UI): prefect block register -m prefect_dbt
- Flow example (runs dbt build and surfaces node-level status in Prefect)
from prefect import flow
from prefect_dbt import PrefectDbtRunner, PrefectDbtSettings
@flow
def dbt_build():
runner = PrefectDbtRunner(
settings=PrefectDbtSettings(
project_dir="/path/to/your/dbt_project",
profiles_dir="~/.dbt" # adjust as needed
),
raise_on_failure=True
)
runner.invoke(["build"])
if __name__ == "__main__":
dbt_build()
- Snowflake credentials via dbt profiles.yml (most common)
Create `~/.dbt/profiles.yml`:
my_dbt_profile:
outputs:
prod:
type: snowflake
account: "{{ env_var('DBT_SNOWFLAKE_ACCOUNT') }}"
user: "{{ env_var('DBT_SNOWFLAKE_USER') }}"
password: "{{ env_var('DBT_SNOWFLAKE_PASSWORD') }}" # or use keypair/oauth options
database: "{{ env_var('DBT_SNOWFLAKE_DATABASE') }}"
schema: "{{ env_var('DBT_SNOWFLAKE_SCHEMA') }}"
warehouse: "{{ env_var('DBT_SNOWFLAKE_WAREHOUSE') }}"
threads: 4
target: prod
Then set the env vars on your worker/agent or in your deployment:
export DBT_SNOWFLAKE_ACCOUNT="xy12345.us-east-1.snowflake"
export DBT_SNOWFLAKE_USER="dbt_user"
export DBT_SNOWFLAKE_PASSWORD="***"
export DBT_SNOWFLAKE_DATABASE="ANALYTICS"
export DBT_SNOWFLAKE_SCHEMA="TRANSFORM"
export DBT_SNOWFLAKE_WAREHOUSE="COMPUTE_WH"
Notes
- This approach gives you dbt’s standard ergonomics and the best Prefect observability (node-level logs, failures, and retries at the Prefect task level if desired).
- If you prefer to generate a profiles.yml at runtime from Prefect blocks, you can use the CLI profile helpers in prefect-dbt (DbtCliProfile + SnowflakeTargetConfigs). That’s handy when you don’t want a static profiles.yml on the image/host.
Docs
- prefect-dbt integration
- dbt Snowflake profile reference
2) dbt Cloud jobs (trigger + wait + targeted retries)
- Install
pip install -U prefect prefect-dbt
# Optional (to create blocks from the UI): prefect block register -m prefect_dbt
- Create a dbt Cloud credentials block
from prefect_dbt.cloud import DbtCloudCredentials
DbtCloudCredentials(api_key="YOUR_SERVICE_TOKEN", account_id=123456).save("dbt-cloud-creds")
- Flow to run a job and wait for completion (with targeted retries)
from prefect import flow
from prefect_dbt.cloud import DbtCloudCredentials
from <http://prefect_dbt.cloud.jobs|prefect_dbt.cloud.jobs> import run_dbt_cloud_job
@flow
def run_cloud_job():
creds = DbtCloudCredentials.load("dbt-cloud-creds")
result = run_dbt_cloud_job(
dbt_cloud_credentials=creds,
job_id=154217, # your dbt Cloud job ID
targeted_retries=3 # retry failed nodes automatically
)
return result
if __name__ == "__main__":
run_cloud_job()
Notes
- Use this if you already manage execution environments in dbt Cloud and want Prefect to coordinate runs with upstream/downstream tasks and alerting.
Docs
- dbt Cloud with Prefect
Which should you choose?Marvin
03/16/2026, 2:22 PM