Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
a
Alex Papanicolaou
08/15/2020, 6:02 PMHi, I have a quick question on cached tasks: will Prefect skip parts of the graph that are cached and don’t need to be computed so it can jump to only the tasks that do need computation?
n
nicholas
08/15/2020, 10:22 PMHi @Alex Papanicolaou - that's correct, cached tasks will only be re-computed if no cached results exist (or they've been invalidate for some reason)
a
Alex Papanicolaou
08/15/2020, 10:24 PMAh, sorry, meant something slightly different (and I know the answer after diving into it). What I meant was that Prefect would be able to pre-check the cache existing and completely removing nodes from consideration in the flow.
for example, if only g didn’t have a cache and I could check that c, d, e, and f had caches, I could load up d and f and cut out c and e entirely (a and b are parameters)
but I know this isn’t possible.
theoretically it is, but not practically, given how results/caches work.
n
nicholas
08/15/2020, 10:27 PMAhh I see what you're saying, sorry for misunderstanding. That's not possible right now.
If you have a compelling use case for it I'm sure others do too, so feel free to open up a GitHub discussion 🙂
a
Alex Papanicolaou
08/15/2020, 10:31 PMThanks. Working on something now but it’s tricky and may do enough weird things we’re willing to take on that it would look out of place in the library. Can certainly write it up and see if it provides any inspiration.
👍 1