Hello all! dbt
I use Prefect to trigger dbt jobs. All is good but they sometime fails because of external issues (network, rate limits, etc)
how did you implement a retry in Prefect with dbt? Do you use
dbt retry
?
b
Bianca Hoch
12/03/2024, 8:25 PM
Hi Mattia! If you're using dbt cloud, I believe the way to handle retries is is by setting
targeted_retries
within `run_dbt_cloud_job`(more on that here). Otherwise, if you're using dbt core, adding retries to your flow or task decorator should do the trick.
m
Mattia Sappa
12/04/2024, 2:54 PM
Hello Bianca, thank you for your answer. I use dbt Core.
I am implementing a solution where I first run
dbt build
in a task and, if the task fails`, I run a second task with
dbt retry
.
This works, however, the flow still results as Failed since the first build task failed. Is it possible to change the state of a task programmatically?
In the doc I see I can return a "Return a manual state" but it's not very convenient for my use case
b
Bianca Hoch
12/05/2024, 12:17 AM
Just came across your convo with Nate in #CL09KU1K7, i think it's alluding to what you mentioned here
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.