Has anyone tried out dbt-prefect in v2.. more specifically what sort of patterns would you use now?

Stephen Herron

09/27/2022, 2:44 PM

Has anyone tried out dbt-prefect in v2.. more specifically what sort of patterns would you use now?

✅ 1

Stephen Herron

09/27/2022, 2:45 PM

In prefect 1 we have a flow that first clones our repo from github, then is able to use the DbtShellTask .. with helper e.g.

cd <dbt_project_dir>

(flows are ran via ECS) Thinking in prefect 2 it might be better to package the dbt code to s3 and use a storage block.. if my flow lives in one place and my dbt project in another hoping it will just work through the magic of s3fs? Although that will mean a change to our DBT cicd (not a huge effort though)

James Sopkin

09/27/2022, 2:47 PM

There is a prefect-dbt collection. I have a flow that used an s3 bucket for storage and call the trigger dbt cloud run and wait for completion task

James Sopkin

09/27/2022, 2:48 PM

https://prefecthq.github.io/prefect-dbt/

Stephen Herron

09/27/2022, 2:48 PM

that said the dbt code just upload to a an s3 sub-path where the main flow that uses the project.. so they could just live together there Thanks @James Sopkin yes that is what I’m planning to use, just the cli version

James Sopkin

09/27/2022, 2:48 PM

ahh okay, you're welcome

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.