https://prefect.io logo
#prefect-community
Title
# prefect-community
t

Tom Shaffner

01/05/2022, 8:26 PM
Is it possible to have a parameter the user could set in the UI to have a flow tell a task to ignore its own cache? e.g. most of the time when a task reruns inside the cached period I want it to use the cache, but every now and then I want it to rerun without it. Currently I just manually delete the cache files before kicking off a run to get this to happen; is there a better way? Ideally maybe a parameter I could pass in from the UI to tell it to ignore the cache this run?
k

Kevin Kho

01/05/2022, 8:32 PM
Kind of. You would need to use that
Parameter
as an input into the task, and then create a
cache_validator
based on it. So if the input into the task changes, the cache is be invalidated
Are you using
target
in the
@task
?
t

Tom Shaffner

01/05/2022, 8:38 PM
I am not. The flow has
LocalResult()
set for
result
That, combined with
cache_for
in tasks within that flow, is all I'm using
k

Kevin Kho

01/05/2022, 8:38 PM
Ah ok you use
cache_for
to set it?
t

Tom Shaffner

01/05/2022, 8:38 PM
Yes
Sorry, edited that last message too late, yes, I am
I take your point in the above about validators. I'm guessing basically I'd just have a parameter that, when turned on, invalidates the cache? I think that could work
k

Kevin Kho

01/05/2022, 8:40 PM
Ah yeah then I would explore using the cache_validator. Yep!
t

Tom Shaffner

01/05/2022, 8:40 PM
I'm new to parameters in that case though, in the run screen of the UI, is is the
context
field I use to pass in parameters?
k

Kevin Kho

01/05/2022, 8:41 PM
No you need to register a parameter in your flow to be able to input it from the UI
t

Tom Shaffner

01/05/2022, 8:41 PM
Ahh, so once I register it, it'll show up in the run screen
k

Kevin Kho

01/05/2022, 8:41 PM
Yes exactly
t

Tom Shaffner

01/05/2022, 8:41 PM
Cool; that should pretty much give me exactly what I wanted; thanks!!
Actually, one more question to make sure I'm understanding, I'd basically need to use the
all_parameters
cache validator and then pass a parameter into that, right? And any chance in that parameter will then invalidate the cache?
And I'm passing the parameter as an extra one for the task itself, not
all_parameters
, correct?
k

Kevin Kho

01/05/2022, 8:50 PM
I think there is
all_inputs
and
all_parameters
. If you have 10 tasks but only want to invalidate one, you need to use
all_inputs
, otherwise all will be invalidated with
all_parameters
right? So yes the task would need to take in an extra input
If you want all to be invalidated, then you can use
all_parameters
or
partial_parameters_only
t

Tom Shaffner

01/05/2022, 8:58 PM
Got it. And if I'm registering a flow, adding a parameter should make it show up in the UI too? So add the
Parameter
function to the flow itself and add a parameter key to the register command?
flow.register(_project_name_=PROJECT_NAME,_parameters_={'cache_invalid':False},_idempotency_key_=FLOW_NAME)
So far that doesn't seem to be working
Oh! I take it back, it was!
Parameter now there, and I know at least what I should do for the rest above; will go from there, thanks for the help!
k

Kevin Kho

01/05/2022, 9:02 PM
Yep!
t

Tom Shaffner

01/05/2022, 9:24 PM
That worked! Thanks!
a

An Hoang

01/05/2022, 9:31 PM
sorry to hijack this thread. What's the difference between
target
caching and
cache_for
caching? Is the latter just for anything that is not file-based? This helps with a question I had earlier. I was just going to either 1) write a wrapper that builds a flow with two different decorators, one with
target
and one without, or 2) use
case
conditional to choose whether to use the function decorated with
target = True
or the one with
target = False
. But the cache validator is very interesting and I think it can serve the same purpose. https://prefect-community.slack.com/archives/CL09KU1K7/p1639327736415300
k

Kevin Kho

01/05/2022, 9:35 PM
Yes that’s right. They are two forms of caching and
target
is file based, like
cache_for
is recorded in Prefect Cloud as a state
I dont think
cache_for
will work for
flow.run()
. It needs a backend
👍 1
42 Views