https://prefect.io logo
Title
j

Jackson Maxfield Brown

07/14/2020, 4:46 AM
Is caching done entirely based off of file existence? Has there been thought into other methods of result caching? I know there it's possible to add a
Serializer
but has there been thought into a
Verifier
or similar? (Entirely just curious at this point)
c

Chris White

07/14/2020, 4:52 AM
Hey Jackson! Currently there are two implementations of caching in Prefect, both of which technically support forms of validation: - the original form of Prefect caching: this requires you specify both
cache_for
and optionally a
cache_validator
on your task that you want cached; the cache validator can be an arbitrary function, but typically folks use one from the Prefect library. Options include input validation, parameter validation, and a few others. This form of validation works best when running against a Prefect backend in order that the
Cached
states are persisted - Task
target
- this keyword is for file existence based caching; however, it is more flexible than it appears. You can provide a custom callable function that accepts the task inputs, and all keys of Prefect context (a typical signature for these functions is
**kwargs
) and returns a string. In this callable you could implement any number of custom validation steps; the simplest is to template the filename based on the inputs to the task (which implicitly serves as a form of validation) but you could perform much more logic in the target callable than that if you want. Ultimately though, task targets do need to return a string representing the location of a file whose existence will be checked
šŸ˜ 1
for a little more info you can check out this doc: https://docs.prefect.io/core/concepts/persistence.html#output-caching
j

Jackson Maxfield Brown

07/14/2020, 4:57 AM
I feel bad I didn't go check through the docs before I posted this question. šŸ™‡ That said, this is really really cool and thanks for the explanation.
šŸ‘ 1
We have been somewhat toying with adding specific library version caching. "This file was produced by 1.0.7 of our library, so no need to recompute" for example, so this is really helpful
c

Chris White

07/14/2020, 5:00 AM
Yea you could definitely do that by using a custom target location that included prefect Dunder version (And no worries on the docs, Iā€™m well aware there are scattered pieces of info that can be hard to discover - something we hope to fix soon!)