Q: is there a way to cache output persistently?
# prefect-community
z
Q: is there a way to cache output persistently?
nicholas pointed me to caching here https://prefect-community.slack.com/archives/CL09KU1K7/p1587825849167300 (thanks!) but it seems the doc implies it’s in memory only - https://github.com/PrefectHQ/prefect/issues/2104 seems to confirm
OTOH nicholas did mention if we use server (we are using self-hosted) we can use that to persist the cache
l
Hi @Ziyao Wei! Yes, tl;dr if you are using server the cache can be retrieved from result handlers (here’s a page about configuring result handlers: https://docs.prefect.io/core/advanced_tutorials/using-result-handlers.html) That being said, there are some changes in flight on the topic, a bit of a long explanation incoming…. We are working on providing a caching mechanism that more transparently goes off of persistence called `target`s (similar to Makefile or Luigi targets). The design document is here: https://docs.prefect.io/core/PINs/PIN-16-Results-and-Targets.html and the latest branch where we are working on it is here: https://github.com/PrefectHQ/prefect/pull/2391. You may be interested in a prior discussion about cache persistence specifically here too on the same vein: https://github.com/PrefectHQ/prefect/issues/2104 (update: I see that’s the issue you referred to, so you know! 🙂 ) Sorry for all the links 🙂 but hopefully that gives some extra context!
z
@Laura Lorenz (she/her) Thanks for those links, will take a deeper dive!
👍 1