I have a question and an idea with regards to caching:
1. How can I forceful clean up cache without an expiry time?
2. If I have a task dependency: A->B->C and both A and B are cached, will there be any result transfered from A to B? In theory I would be able to only get cached returned values from B and run C.
3. It would be nice to have configurable cache locations by means of env var for each flow run. Currently when using input hash and with multiple users run the same task, prefect will try to fetch the cache from the first users, which raises permission issues depending. As a workaround I can get add the user to the hash function, but that feels complicated. Any thoughts?
For 3 I will add the storage path in the hash, coming from the taskruncontext.
09/30/2022, 2:05 PM
Hey @Ching!! Is this for 1.0 or 2.0? Either way I'll provide some documentation here:
Prefect 1.0 CachingPrefect 2.0 Caching
1. Is there a reason you would want to forcefully clear the cache instead of setting the cache expiration to a lower amount?
2. In 1.0, this sounds like input caching, which can be found here. For 2.0, The above documentation describes how results and states of tasks can be cached.
3. For 2.0, you can technically set the
variable, but I would be careful about using that as a solution. For 1.0, I would recommend looking at this documentation .