<@ULVA73B9P> I’m getting inconsistent caching beha...
# ask-marvin
j
@Marvin I’m getting inconsistent caching behaviour: a task is re-running even though
cache_policy=INPUTS
and the inputs are staying the same. The task has 3 inputs which are objects. To check, when I do
hash_objects
on each of the objects individually, I get a consistent hash result, but when I do
hash_objects
on two or more, the hash changes on each run. Any ideas why this happens? Should
hash_objects
give a consistent result for more than one object?
m
It seems you're dealing with an inconsistent hash result when using the
hash_objects
function on multiple objects in Prefect's caching mechanism. From the documentation for both Prefect 2.x and 3.x, it's clear that Prefect uses cache keys carefully to decide when tasks can reuse previous results Prefect 2.x Tasks Caching and Prefect 3.x Task Caching. The
hash_objects
function should ideally provide consistent results across multiple runs if the input objects haven't changed. However, the behavior you described—consistent hashes for individual objects but inconsistent when combined—could suggest issues like: 1. Mutable Elements: If any of your object's attributes are mutable and change between runs (even something like timestamps), this can lead to differing hash values. 2. Order Sensitivity: The hash result can be sensitive to the order of objects. Ensure that you're combining the objects in the same order every time before hashing. 3. Serialization Differences: The serialization process that
hash_objects
might use could inadvertently lead to different byte sequences, hence different hashes, if the objects are complex. To address the issue, you could: - Check if any part of the object is changing (like internal states or timestamps). - Ensure the order of objects is consistent when you pass them for hashing. - Consider providing your own function for generating cache keys if you have a specific logic that could capture all your objects' states dependably. If these don't help, you might want to wrap the objects into a consistent serialization form (like converting them to a JSON string) before hashing, ensuring a consistent input shape for
hash_objects
. If the issue persists, reporting it on Prefect GitHub could help if there is a bug or undocumented feature affecting your use case.