Tom Shaffner
01/05/2022, 8:26 PMKevin Kho
01/05/2022, 8:32 PMParameter
as an input into the task, and then create a cache_validator
based on it. So if the input into the task changes, the cache is be invalidatedtarget
in the @task
?Tom Shaffner
01/05/2022, 8:38 PMLocalResult()
set for result
cache_for
in tasks within that flow, is all I'm usingKevin Kho
01/05/2022, 8:38 PMcache_for
to set it?Tom Shaffner
01/05/2022, 8:38 PMKevin Kho
01/05/2022, 8:40 PMTom Shaffner
01/05/2022, 8:40 PMcontext
field I use to pass in parameters?Kevin Kho
01/05/2022, 8:41 PMTom Shaffner
01/05/2022, 8:41 PMKevin Kho
01/05/2022, 8:41 PMTom Shaffner
01/05/2022, 8:41 PMall_parameters
cache validator and then pass a parameter into that, right? And any chance in that parameter will then invalidate the cache?all_parameters
, correct?Kevin Kho
01/05/2022, 8:50 PMall_inputs
and all_parameters
. If you have 10 tasks but only want to invalidate one, you need to use all_inputs
, otherwise all will be invalidated with all_parameters
right? So yes the task would need to take in an extra inputall_parameters
or partial_parameters_only
Tom Shaffner
01/05/2022, 8:58 PMParameter
function to the flow itself and add a parameter key to the register command?
flow.register(_project_name_=PROJECT_NAME,_parameters_={'cache_invalid':False},_idempotency_key_=FLOW_NAME)
So far that doesn't seem to be workingKevin Kho
01/05/2022, 9:02 PMTom Shaffner
01/05/2022, 9:24 PMAn Hoang
01/05/2022, 9:31 PMtarget
caching and cache_for
caching? Is the latter just for anything that is not file-based? This helps with a question I had earlier.
I was just going to either 1) write a wrapper that builds a flow with two different decorators, one with target
and one without, or 2) use case
conditional to choose whether to use the function decorated with target = True
or the one with target = False
. But the cache validator is very interesting and I think it can serve the same purpose.
https://prefect-community.slack.com/archives/CL09KU1K7/p1639327736415300Kevin Kho
01/05/2022, 9:35 PMtarget
is file based, like cache_for
is recorded in Prefect Cloud as a statecache_for
will work for flow.run()
. It needs a backend