Is caching done entirely based off of file existen...
# prefect-community
Is caching done entirely based off of file existence? Has there been thought into other methods of result caching? I know there it's possible to add a
but has there been thought into a
or similar? (Entirely just curious at this point)
Hey Jackson! Currently there are two implementations of caching in Prefect, both of which technically support forms of validation: - the original form of Prefect caching: this requires you specify both
and optionally a
on your task that you want cached; the cache validator can be an arbitrary function, but typically folks use one from the Prefect library. Options include input validation, parameter validation, and a few others. This form of validation works best when running against a Prefect backend in order that the
states are persisted - Task
- this keyword is for file existence based caching; however, it is more flexible than it appears. You can provide a custom callable function that accepts the task inputs, and all keys of Prefect context (a typical signature for these functions is
) and returns a string. In this callable you could implement any number of custom validation steps; the simplest is to template the filename based on the inputs to the task (which implicitly serves as a form of validation) but you could perform much more logic in the target callable than that if you want. Ultimately though, task targets do need to return a string representing the location of a file whose existence will be checked
šŸ˜ 1
for a little more info you can check out this doc:
I feel bad I didn't go check through the docs before I posted this question. šŸ™‡ That said, this is really really cool and thanks for the explanation.
šŸ‘ 1
We have been somewhat toying with adding specific library version caching. "This file was produced by 1.0.7 of our library, so no need to recompute" for example, so this is really helpful
Yea you could definitely do that by using a custom target location that included prefect Dunder version (And no worries on the docs, Iā€™m well aware there are scattered pieces of info that can be hard to discover - something we hope to fix soon!)