Hi! Is there a way to use checkpointing as a persistent cache? It doesn't seem to load the checkpointed file by default

Hi @Joe Howarth! Checkpointing is intended to be used for persisting your outputs for data lineage, etc., not for re-use in future flow runs. To re-use your outputs in future flow runs, check out Prefect’s “caching” mechanism: https://docs.prefect.io/core/concepts/persistence.html#output-caching Note: Prefect Core has no formal persistence layer, so in order to re-use your cache across multiple processes will require a custom solution, or Prefect Cloud

ahh ok, is there anyway of configuring caching to work with a custom storage system or do those hooks not exist right now? (I don't see any references to custom cache handler hooks similar to the result handler)

There are no first-class hooks for it, no

any ideas for a hacky work around?

yea for sure; you just need to save / load the

prefect.context.caches

object, which is also a dictionary of cache key -> list of valid cache states

yup, with a bit of finessing, that worked!

awesome!

is this something that, with a bit more polish, would be considered pulling in officially?

that’s a great question - I won’t say it’s out of scope; however, we have a strong bias to only expose hooks / features that align with what we believe to be the best workflow practices. So, our tried-and-true persistent caching implementation is the one we have implemented in Prefect Cloud, which allows users to version their cache, control access via auth, protect their data, run in distributed environments, etc. etc. So, any implementation of persistent caching in Core would have to be justified along those dimensions. (and a huge reason for releasing a public, free tier of Cloud is to make sure our open source users have access to these sorts of features as well)

gotcha, haven't really looked at the cloud options too much yet, will do so!

anytime!