oliland
09/07/2019, 2:46 PMJeremiah
09/07/2019, 2:51 PMcache_validator
that invalidated the cache whenever that version changed. Then you’d run the flow on a regular schedule and as long as the cache was valid, all tasks would simply skipraise prefect.signals.Skip()
and skip the workflowoliland
09/07/2019, 2:57 PMJeremiah
09/07/2019, 2:59 PMoliland
09/07/2019, 3:00 PMJeremiah
09/07/2019, 3:00 PMoliland
09/07/2019, 5:15 PMassets[input_checksums] = output_checksums_and_paths
and can use cached assets where available.Jeremiah
09/07/2019, 5:21 PMAsset
class that your tasks consume and produce. You could even write a Task
subclass that automatically checks any Asset
inputs and raises a signal appropriately, so you wouldn’t even need the boilerplate.oliland
09/07/2019, 5:27 PMJeremiah
09/07/2019, 5:32 PMcache_for
to do it for you. 🤔 there’s something interesting hereoliland
09/07/2019, 5:35 PMJeremiah
09/07/2019, 5:36 PMoliland
09/07/2019, 5:40 PMyou’d need to store the assets in a known place and we try to avoid forced state in PrefectI think if you're using assets you'd be effectively embracing some state. So every initialised asset would require a path, and a way of checksumming itself
Jeremiah
09/07/2019, 5:40 PMResultHandlers
that could be reused here. Basically they are ways to interface Prefect with a known stateful location. As I mentioned we try not to use forced state within the engine, but this is the exception that comes to mindoliland
09/07/2019, 5:42 PMChris White
09/07/2019, 6:10 PMall_parameters
cache validator on any relevant tasks to cache based on the value of this Flow Parameter