Jackson Maxfield Brown
07/14/2020, 4:46 AMSerializer
but has there been thought into a Verifier
or similar? (Entirely just curious at this point)Chris White
cache_for
and optionally a cache_validator
on your task that you want cached; the cache validator can be an arbitrary function, but typically folks use one from the Prefect library. Options include input validation, parameter validation, and a few others. This form of validation works best when running against a Prefect backend in order that the Cached
states are persisted
- Task target
- this keyword is for file existence based caching; however, it is more flexible than it appears. You can provide a custom callable function that accepts the task inputs, and all keys of Prefect context (a typical signature for these functions is **kwargs
) and returns a string. In this callable you could implement any number of custom validation steps; the simplest is to template the filename based on the inputs to the task (which implicitly serves as a form of validation) but you could perform much more logic in the target callable than that if you want. Ultimately though, task targets do need to return a string representing the location of a file whose existence will be checkedJackson Maxfield Brown
07/14/2020, 4:57 AMChris White