Thread
#prefect-community
    j

    John

    1 year ago
    hey folks—i’m new to Prefect and am trying to understand how one might approach configuring a flow so that it has access to its own previous execution date (similar to airflow’s
    prev_execution_date_success
    variable). in reading the docs, caching doesn’t quite seem like the right fit. persisting output makes sense, but not clear to me the right way to re-consume that output on a future flow run. so at this point my inclination is to simply read/write the last run timestamp to a local file. anyone have a recipe, docs, or otherwise on a better approach?
    Amanda Wee

    Amanda Wee

    1 year ago
    I don't think prefect persists the execution date, but I could be wrong. So my approach would be similiar to what you have in mind, except that I might use a flow state handler to write the date when the flow enters success state, and then I'd use the Result interface to write to flow storage rather than writing to a file. Then I'd have a prefect task that reads from that storage location, upon which that date can be used by other tasks.
    Jay Vercellone

    Jay Vercellone

    1 year ago
    Access to this metadata would be super useful, especially since ETL jobs usually run periodically, and scoped within a certain start and end date/time.
    j

    John

    1 year ago
    thanks @Amanda Wee! i’ll give it a try. @Jay Vercellone —agreed. my use case is a incremental load in which we only need to process records which were modified since the last successful run.