Question regarding the `Result` we longed for I see that the Prefect Community #ask-community

Question regarding the `Result` we longed for: I s...

Avi A

05/27/2020, 12:35 PM

Question regarding the

Result

we longed for: I see that the default is to cache the results per task, and it’s possible to cache per day/today/tomorrow when using the target formatting. Is there a behavior similar to

cache_validator

, allowing us to cache the results per set of attributes?

👀 1

Avi A

05/27/2020, 12:40 PM

advanced: can we cache the results based on the input to the upsteam tasks? example:

Copy code

@task
def A(p: int):
  return some_object

@task
def B(p: int):
  return some_object

with Flow() as flow:
  p = Parameter('p')
  a = A(p)
  b = B(a)

can I cache the result of

based on the value of the input parameter

(not

), without specifically mentioning it in task

? meaning having it read from the upsteam?

nicholas

05/27/2020, 12:51 PM

Hi @Avi A - we have an open ticket that'll enable the behavior you've described; I'm sure your input is welcome there if you have ideas of how you might like to see it implemented (or just an emoji reaction to let them know you're also interested in seeing this)!

Avi A

05/27/2020, 12:53 PM

thanks @nicholas! Related: if I don’t supply

target

, it seems as if the task is not cached at all. I mean, it is cached, but since the exact timestamp when the task was run (not even including its name), then this cache is practically never read 😞

Avi A

05/27/2020, 12:53 PM

I mean, it’s better to use the deprecated

cache_for

nicholas

05/27/2020, 1:10 PM

Interesting @Avi A - is there any reason you might not want to use a result target with some sort of result templating?

Avi A

05/27/2020, 1:11 PM

well, because I can’t use the task parameters in the template

Avi A

05/27/2020, 1:11 PM

I see I can use the flow parameters in the template, because they appear in the context (given they were provided to the flow, and not using the defaults)

nicholas

05/27/2020, 1:22 PM

Ah ok! I think you can use task parameters in the in the location template, since they're available in context. Let me confirm that for you quickly though

Avi A

05/27/2020, 1:24 PM

I can

Avi A

05/27/2020, 1:24 PM

but only if they were provided explicitly (i.e. not used the defaults)

Avi A

05/27/2020, 1:25 PM

also, I’d like to cache a specific task based on the parameters it recursively depends on, and not necessarily on all the params

nicholas

05/27/2020, 1:31 PM

Seems like a good use case @Avi A - I'd encourage you to add your thoughts to that issue above, they'll be valuable in deciding the direction/implementation of that ticket

👍 1

3 Views

Open in Slack

Previous Next