Hi With result persistence on is it possible to trigger a ta

Question

Hi With result persistence on is it possible to trigger a task rerun if the persisted result got deleted or corrupted I am thinking about a case where the cache key for the orion DB is available but the result can t be retrieved from local storage In such a case I would like to invalidate the cache key and trigger a new task run to obtain the missing corrupted result

Mason Menges · Answer

Hey < Tim Oliver> I don t think I ve run into anything like this personally but I would think task level retries would accomodate this though it kinda depends on what you mean by task re run Are you triggering a retry of the flow run from the UI

Tim-Oliver · Answer

I just tried it out with retries but it does not work

Tim-Oliver · Answer

My understanding of task retries is that a task run is tried again when the task fails due to some internal task error e g connection timeout to a DB

Tim-Oliver · Answer

The pattern I am trying to solve is 1 Submit flow run and wait for completion 2 Go to storage and delete one of the persisted results 3 Submit another flow run with the same parameters as in 1 Now what I would like to achieve is that the 2nd flow run only recomputes the missing result and all other task runs are skipped due to the result being cached

Tim-Oliver · Answer

However what I see with and without retry on is that the flow is executed and the result is indicated as cached in the UI but it can t be retrieved from local storage because it got deleted

Tim-Oliver · Answer

It does work when I go into the first flow run and delete the task run I guess this will remove the entry from the DB and does not even try to load the local storage result

Tim-Oliver · Answer

I think I am looking for a function which is called after the validation of the cache key to validate if the local storage is present And only if both cache key and local storage are good the cached result is used

Tim-Oliver · Answer

I am also happy to open a git issue slightly smiling face

Tim-Oliver · Answer

~I think I could get there by using a special cache key fn which first computes the cache key and then tries to access the local storage If the file is found the normal cache key is returned otherwise some other unique cache key is returned ~ Thought a bit more and concluded that this will not work that easily Because I would have to recreate the unique cache key in later runs again if the re computed result should be reused

Tim-Oliver · Answer

Now I just have to figure out how I can get the local storage results

Tim-Oliver · Answer

My current understanding of how caching works 1 cache key fn is called with TaskRunContext and input arguments a cache key is computed 2 Orion looks up if cache key exists a If cache key exists state is set to completed cached b otherwise task is marked to be executed

Tim-Oliver · Answer

Would the following work 1 cache key fn is called with TaskRunContext and input arguments a compute cache key b Check via REST API if the cache key exists c If the cache key exists retrieve the result state via REST API d Check if the local storage results is valid i If valid return cache key ii otherwise delete existing task run via REST API and then return cache key

Deceivious · Answer

Did anyone actually solve this isses

Tim-Oliver · Answer

Haven t heard back and have not tried it out But I would be interested if this works