Hello. In prefect 1, is there a docs you can point me towards for the following problem: I have daily scheduled flows that run dbt projects for the previous day. Dbt dates are provided from the flow by Parameters, and the default is

date.today()

and

date.today() - timedelta(1 day)

. Now I want to make the flows resilient. I want the flow's date parameter to be the last success flow run, so that if one day the flow run fails, the next flow run automatically takes the last 2 days. Basically, I want the dates to be

date.today()

and

last_successful_flow_run_date

. I was thinking of storing the flow run result and its parameters in an S3 bucket, then query it every time to recover the last date that the flow ran successfully. But I'm not sure if this is the best way. I'd appreciate any help with this. Thank you

Hey @Mansour Zayer 👋 I believe that you can also leverage the GraphQL client to fire a query to get flow runs along with their properties (which includes the run timestamp and the exec status). Once you have the data back from GraphQL, you can process the list of flow runs to extract the last successful flow run date.

So I'll create a task (

get_last_run_datetime

) inside my flow (

my_flow

) that queries

my_flow

for its runs, and returns the datetime of the last successful run. Then I'll provide that datetime as input to my dbt run task. Seems like a clean and fairly simple solution. Thanks a lot Alessandro.

Yes that's the idea!

another potential solution (if you're using Cloud) would be to use the KV store: at the beginning of each run

whatever_info_about_last_run_you_want_to_use = get_key_value('LAST_RUN')

which assumes you're doing something like this at the end of each run:

set_key_value('LAST_RUN', whatever_info_about_this_run_you_want_to_use_next_run)

Thanks for the input. I think using GraphQL is a more robust solution since I don't have to manage storing the run information on my own while Prefect backend obviously does it in a much better and complete way. For now, I'll go with the GraphQL solution. Also, there's always "one more" piece of information you will need as you go, so I'll end up in a loop where I have to keep storing extra info in the KV store which might contradict each other at some point. But using GraphQL, I'll just update the query.

Makes sense, just threw it out there in case it was helpful 🙂