https://prefect.io logo
Title
v

Venkat

03/23/2020, 6:19 AM
@here - Any advice on how to pass results across flow runs? I have a flow that runs on a schedule, and a task in the flow that returns a particular value (in this case, a
pandas.DataFrame
). For every (second and subsequent) run of the flow, I need to retrieve the previous dataFrame, in order to do a merge/diff operation with the one generated in this run. Is this possible? I read up all the docs on States and ResultHandlers, but I dont see anything that persists across flow runs.
j

josh

03/23/2020, 11:40 AM
Hi @Venkat the use of caching will persist across flow runs however using only Prefect Core at the moment will require those cached states to live in memory. i.e. a flow running on schedule in the same process will be able to use them Prefect Cloud resolves this constraint by providing a persistence layer for those cached states to be used across runs, anywhere.
v

Venkat

03/23/2020, 5:08 PM
thanks @josh - the part that I didnt quite grok on the docs realted to caching, is how do I refer to a cached result within the same task, of its own result from the previous run...i.e i see the args on the task decorator to allow caching and how to set it up, but didnt quite get how to point at the previous result the next time the same task runs within the next iteration of the flow
the in-mem caching works just fine for my use case
j

josh

03/23/2020, 5:12 PM
Great! Was this issue opened by you as well? (seems related around the same time) https://github.com/PrefectHQ/prefect/issues/2186
v

Venkat

03/23/2020, 5:13 PM
yes, sorry for the spam 😬
j

josh

03/23/2020, 5:13 PM
Np I’ll reference this chat in that issue 🙂
v

Venkat

03/23/2020, 5:13 PM
👍
j

josh

03/23/2020, 5:13 PM
@Marvin archive “Cache data across flow runs”
v

Venkat

03/23/2020, 5:15 PM
if you could point out how i'd do the following, that would be great 1. specify a cache key on a task decorator - would it be something like
@task(cache_for=datetime.timedelta(hours=1), cache_key='prev_result')
2. it says in the docs that the result would then live in
prefect.context
3. so if I set it up appropriately, would the subsequent run of the task do a check on this key like
if prefect.context.prev_result is not None:
    #do something with it
j

josh

03/23/2020, 5:21 PM
1. Yes that’s how it should look 2. Correct it will be in context in memory 3. The function which does this check can be found here for more info