Copying this from caching thread above Hi guys has there be Prefect Community #ask-community

(Copying this from caching thread above) Hi guys, ...

Brad

08/23/2019, 11:34 AM

(Copying this from caching thread above) Hi guys, has there been any update to the caching mechanisms in the past few weeks? I’m interested in persisting results across runs to a local (or remote) cache. I would also like to be able to update a tasks status to cached based on the input parameters. I’ve been playing around with the

state_handlers

and

trigger

hooks but I can’t seem to achieve what I want

Brad

08/23/2019, 11:35 AM

Basically, I would like to do something along the lines of:

Copy code

import datetime

from prefect import task, Flow, Parameter
from prefect.engine.cache_validators import all_parameters
from prefect.engine.result_handlers import LocalResultHandler

local_cache = LocalResultHandler(dir='~/data/prefect_cache')
long_time = datetime.timedelta(weeks=52)


@task(result_handler=local_cache, checkpoint=True, state_handlers=[checkpoint], cache_validator=all_parameters,
      cache_for=long_time, trigger=trigger)
def fetch(x):
    return x

@task(result_handler=local_cache, checkpoint=True, state_handlers=[checkpoint], cache_validator=all_parameters,
      cache_for=long_time, trigger=trigger)
def inc(y):
    return y + 1

with Flow('test') as flow:
    n = Parameter('n')
    f = fetch(n)
    i = inc(f)

if __name__ == '__main__':
    flow.run(n=1)

Brad

08/23/2019, 11:37 AM

And where this flow is run multiple times with the same parameters, have the ability to read cached parameters, and create a

Cached

state instance where applicable so we play nicely with prefect everywhere else.

Chris White

08/24/2019, 2:58 AM

No updates to caching in the last few weeks - persistence (of any kind, really) is a very tricky thing to implement correctly for the kinds of workflow patterns that Prefect supports, which is why we only offer it as a part of our full Cloud platform. That being said, if you are only using the pattern above, then Prefect Core stores cached states in memory in

prefect.context.caches

--> consequently, you can save this to disk after each run and initialize it prior to each run to persist caches across runs

Brad

08/26/2019, 7:25 AM

Thanks Chris, how would I go about initializing the cache? I can’t seem to find where the code for the cache/context lives. Any pointers on where to look would be much appreciated

Open in Slack

Previous Next