Hi I have a quick question on cached tasks will Prefect skip

Question

Hi I have a quick question on cached tasks will Prefect skip parts of the graph that are cached and don t need to be computed so it can jump to only the tasks that do need computation

nicholas · Answer

Hi < Alex Papanicolaou> that s correct cached tasks will only be re computed if no cached results exist or they ve been invalidate for some reason

Alex Papanicolaou · Answer

Ah sorry meant something slightly different and I know the answer after diving into it What I meant was that Prefect would be able to pre check the cache existing and completely removing nodes from consideration in the flow

Alex Papanicolaou · Answer

for example if only g didn t have a cache and I could check that c d e and f had caches I could load up d and f and cut out c and e entirely a and b are parameters

Alex Papanicolaou · Answer

but I know this isn t possible

Alex Papanicolaou · Answer

theoretically it is but not practically given how results caches work

nicholas · Answer

Ahh I see what you re saying sorry for misunderstanding That s not possible right now

nicholas · Answer

If you have a compelling use case for it I m sure others do too so feel free to open up a <https github com PrefectHQ prefect discussions category choices|GitHub discussion> slightly smiling face

Alex Papanicolaou · Answer

Thanks Working on something now but it s tricky and may do enough weird things we re willing to take on that it would look out of place in the library Can certainly write it up and see if it provides any inspiration