https://prefect.io logo
Title
b

Brad

08/23/2019, 11:34 AM
(Copying this from caching thread above) Hi guys, has there been any update to the caching mechanisms in the past few weeks? I’m interested in persisting results across runs to a local (or remote) cache. I would also like to be able to update a tasks status to cached based on the input parameters. I’ve been playing around with the
state_handlers
and
trigger
hooks but I can’t seem to achieve what I want
Basically, I would like to do something along the lines of:
import datetime

from prefect import task, Flow, Parameter
from prefect.engine.cache_validators import all_parameters
from prefect.engine.result_handlers import LocalResultHandler

local_cache = LocalResultHandler(dir='~/data/prefect_cache')
long_time = datetime.timedelta(weeks=52)


@task(result_handler=local_cache, checkpoint=True, state_handlers=[checkpoint], cache_validator=all_parameters,
      cache_for=long_time, trigger=trigger)
def fetch(x):
    return x

@task(result_handler=local_cache, checkpoint=True, state_handlers=[checkpoint], cache_validator=all_parameters,
      cache_for=long_time, trigger=trigger)
def inc(y):
    return y + 1

with Flow('test') as flow:
    n = Parameter('n')
    f = fetch(n)
    i = inc(f)

if __name__ == '__main__':
    flow.run(n=1)
And where this flow is run multiple times with the same parameters, have the ability to read cached parameters, and create a
Cached
state instance where applicable so we play nicely with prefect everywhere else.
c

Chris White

08/24/2019, 2:58 AM
No updates to caching in the last few weeks - persistence (of any kind, really) is a very tricky thing to implement correctly for the kinds of workflow patterns that Prefect supports, which is why we only offer it as a part of our full Cloud platform. That being said, if you are only using the pattern above, then Prefect Core stores cached states in memory in
prefect.context.caches
--> consequently, you can save this to disk after each run and initialize it prior to each run to persist caches across runs
b

Brad

08/26/2019, 7:25 AM
Thanks Chris, how would I go about initializing the cache? I can’t seem to find where the code for the cache/context lives. Any pointers on where to look would be much appreciated