https://prefect.io logo
Title
m

Mansour Zayer

08/25/2022, 3:02 PM
Hello. In prefect 1, is there a docs you can point me towards for the following problem: I have daily scheduled flows that run dbt projects for the previous day. Dbt dates are provided from the flow by Parameters, and the default is
date.today()
and
date.today() - timedelta(1 day)
. Now I want to make the flows resilient. I want the flow's date parameter to be the last success flow run, so that if one day the flow run fails, the next flow run automatically takes the last 2 days. Basically, I want the dates to be
date.today()
and
last_successful_flow_run_date
. I was thinking of storing the flow run result and its parameters in an S3 bucket, then query it every time to recover the last date that the flow ran successfully. But I'm not sure if this is the best way. I'd appreciate any help with this. Thank you
1
a

ale

08/25/2022, 3:13 PM
Hey @Mansour Zayer 👋 I believe that you can also leverage the GraphQL client to fire a query to get flow runs along with their properties (which includes the run timestamp and the exec status). Once you have the data back from GraphQL, you can process the list of flow runs to extract the last successful flow run date.
👍 1
m

Mansour Zayer

08/25/2022, 3:49 PM
So I'll create a task (
get_last_run_datetime
) inside my flow (
my_flow
) that queries
my_flow
for its runs, and returns the datetime of the last successful run. Then I'll provide that datetime as input to my dbt run task. Seems like a clean and fairly simple solution. Thanks a lot Alessandro.
a

ale

08/25/2022, 3:51 PM
Yes that's the idea!
n

Nate

08/25/2022, 4:06 PM
another potential solution (if you're using Cloud) would be to use the KV store: at the beginning of each run
whatever_info_about_last_run_you_want_to_use = get_key_value('LAST_RUN')
which assumes you're doing something like this at the end of each run:
set_key_value('LAST_RUN', whatever_info_about_this_run_you_want_to_use_next_run)
:upvote: 1
m

Mansour Zayer

08/26/2022, 5:51 PM
Thanks for the input. I think using GraphQL is a more robust solution since I don't have to manage storing the run information on my own while Prefect backend obviously does it in a much better and complete way. For now, I'll go with the GraphQL solution. Also, there's always "one more" piece of information you will need as you go, so I'll end up in a loop where I have to keep storing extra info in the KV store which might contradict each other at some point. But using GraphQL, I'll just update the query.
👍 1
n

Nate

08/26/2022, 6:05 PM
Makes sense, just threw it out there in case it was helpful 🙂
🙌 1