https://prefect.io logo
Title
j

jorwoods

06/11/2020, 9:49 PM
Thank you very much to Laura for your help earlier! One more question regarding checkpoints and caching. Adding the
target
kwarg to the task caches the result as expected. I will try to contribute back to the documentation soon now that I have a better understanding of that vs a Result's
location
. Here is a modified version of the toy example I used before, this time I am trying to get it to forcibly ignore the cache and recompute. It is not considering the tasks' cache as still valid, despite a short
cache_for
and a validator of
never_use
. I know that
cache_for
and
cache_validator
in
Task
are deprecated. I tried specifying the validator on the
LocalResult
as you will see in the example.
from prefect import Flow, task, unmapped, Parameter
from prefect.engine.results import LocalResult
from prefect.engine.executors import DaskExecutor
from prefect.engine.cache_validators import all_parameters, never_use
import datetime

delta=datetime.timedelta(seconds=5)

lr = LocalResult(validators=[never_use])

@task(log_stdout=True, checkpoint=True, 
      target='{flow_name}-{task_name}-{parameters[x]}-{parameters[y]}.pkl',
      cache_validator=never_use, cache_for=delta)
def add(x, y):
    print(f'add ran with {x} {y}')
    try:
        return sum(x) + y
    except TypeError:
        return x + y

with Flow('iterated map', result=lr) as flow:
    y = unmapped(Parameter('y', default=8))
    x = Parameter('x', default=[1,2,3])
    mapped_result = add.map(x, y=y)
    out = add(mapped_result, y)

if __name__ == "__main__":
    flow.run(executor=DaskExecutor(),)
👀 1
l

Laura Lorenz (she/her)

06/11/2020, 10:05 PM
Hi! I think maybe you had a typo there, I think you are saying that “It is not considering the tasks’ cache as *IN*valid” despite your short cache_for key. Caching configured with `cache_for`/`cache_key` and caching with `target`s are separate right now (we do want to connect them though — the issue for that is https://github.com/PrefectHQ/prefect/issues/2619). Basically it is that we haven’t souped up targets all the way yet 🙂 . So any
cache_for
/
cache_key
and even
cache_validators
or
validators
will not affect target — just any old file existing at the
target
location will mean that the task is considered Cached, no other questions asked (currently). In addition, we check for the target if configured INSTEAD OF the cache used by cache_for/cache_key, so target always wins (again, only currently).
j

jorwoods

06/11/2020, 10:12 PM
You are correct that I had a typo! But ok, good to know it’s not me misconfiguring something. I will take a look at the code and see if there is something I can contribute back to the core!
❤️ 1
l

Laura Lorenz (she/her)

06/11/2020, 10:23 PM
Awesome! And, we have a #prefect-contributors channel you can hang in too for contributing questions 🙂
👍 1