@here - Any advice on how to pass results across flow runs? I have a flow that runs on a schedule, and a task in the flow that returns a particular value (in this case, a

pandas.DataFrame

). For every (second and subsequent) run of the flow, I need to retrieve the previous dataFrame, in order to do a merge/diff operation with the one generated in this run. Is this possible? I read up all the docs on States and ResultHandlers, but I dont see anything that persists across flow runs.

Hi @Venkat the use of caching will persist across flow runs however using only Prefect Core at the moment will require those cached states to live in memory. i.e. a flow running on schedule in the same process will be able to use them Prefect Cloud resolves this constraint by providing a persistence layer for those cached states to be used across runs, anywhere.

thanks @josh - the part that I didnt quite grok on the docs realted to caching, is how do I refer to a cached result within the same task, of its own result from the previous run...i.e i see the args on the task decorator to allow caching and how to set it up, but didnt quite get how to point at the previous result the next time the same task runs within the next iteration of the flow

Great! Was this issue opened by you as well? (seems related around the same time) https://github.com/PrefectHQ/prefect/issues/2186

yes, sorry for the spam 😬

Np I’ll reference this chat in that issue 🙂

👍

@Marvin archive “Cache data across flow runs”

if you could point out how i'd do the following, that would be great 1. specify a cache key on a task decorator - would it be something like

@task(cache_for=datetime.timedelta(hours=1), cache_key='prev_result')

2. it says in the docs that the result would then live in

prefect.context

3. so if I set it up appropriately, would the subsequent run of the task do a check on this key like

if prefect.context.prev_result is not None:
    #do something with it

1. Yes that’s how it should look 2. Correct it will be in context in memory 3. The function which does this check can be found here for more info