Hi, I have a quick question on cached tasks: will Prefect skip parts of the graph that are cached and don’t need to be computed so it can jump to only the tasks that do need computation?
08/15/2020, 10:22 PM
Hi @Alex Papanicolaou - that's correct, cached tasks will only be re-computed if no cached results exist (or they've been invalidate for some reason)
08/15/2020, 10:24 PM
Ah, sorry, meant something slightly different (and I know the answer after diving into it). What I meant was that Prefect would be able to pre-check the cache existing and completely removing nodes from consideration in the flow.
for example, if only g didn’t have a cache and I could check that c, d, e, and f had caches, I could load up d and f and cut out c and e entirely (a and b are parameters)
but I know this isn’t possible.
theoretically it is, but not practically, given how results/caches work.
08/15/2020, 10:27 PM
Ahh I see what you're saying, sorry for misunderstanding. That's not possible right now.
If you have a compelling use case for it I'm sure others do too, so feel free to open up a GitHub discussion 🙂
08/15/2020, 10:31 PM
Thanks. Working on something now but it’s tricky and may do enough weird things we’re willing to take on that it would look out of place in the library. Can certainly write it up and see if it provides any inspiration.