Mansour Zayer
08/25/2022, 3:02 PMdate.today()
and date.today() - timedelta(1 day)
. Now I want to make the flows resilient. I want the flow's date parameter to be the last success flow run, so that if one day the flow run fails, the next flow run automatically takes the last 2 days. Basically, I want the dates to be date.today()
and last_successful_flow_run_date
.
I was thinking of storing the flow run result and its parameters in an S3 bucket, then query it every time to recover the last date that the flow ran successfully. But I'm not sure if this is the best way.
I'd appreciate any help with this. Thank youale
08/25/2022, 3:13 PMMansour Zayer
08/25/2022, 3:49 PMget_last_run_datetime
) inside my flow (my_flow
) that queries my_flow
for its runs, and returns the datetime of the last successful run.
Then I'll provide that datetime as input to my dbt run task.
Seems like a clean and fairly simple solution. Thanks a lot Alessandro.ale
08/25/2022, 3:51 PMNate
08/25/2022, 4:06 PMwhatever_info_about_last_run_you_want_to_use = get_key_value('LAST_RUN')
which assumes you're doing something like this at the end of each run:
set_key_value('LAST_RUN', whatever_info_about_this_run_you_want_to_use_next_run)
Mansour Zayer
08/26/2022, 5:51 PMNate
08/26/2022, 6:05 PM