a

    Avi A

    2 years ago
    Question regarding the
    Result
    we longed for: I see that the default is to cache the results per task, and it’s possible to cache per day/today/tomorrow when using the target formatting. Is there a behavior similar to
    cache_validator
    , allowing us to cache the results per set of attributes?
    advanced: can we cache the results based on the input to the upsteam tasks? example:
    @task
    def A(p: int):
      return some_object
    
    @task
    def B(p: int):
      return some_object
    
    with Flow() as flow:
      p = Parameter('p')
      a = A(p)
      b = B(a)
    can I cache the result of
    B
    based on the value of the input parameter
    p
    (not
    a
    ), without specifically mentioning it in task
    B
    ? meaning having it read from the upsteam?
    nicholas

    nicholas

    2 years ago
    Hi @Avi A - we have an open ticket that'll enable the behavior you've described; I'm sure your input is welcome there if you have ideas of how you might like to see it implemented (or just an emoji reaction to let them know you're also interested in seeing this)!
    a

    Avi A

    2 years ago
    thanks @nicholas! Related: if I don’t supply
    target
    , it seems as if the task is not cached at all. I mean, it is cached, but since the exact timestamp when the task was run (not even including its name), then this cache is practically never read 😞
    I mean, it’s better to use the deprecated
    cache_for
    nicholas

    nicholas

    2 years ago
    Interesting @Avi A - is there any reason you might not want to use a result target with some sort of result templating?
    a

    Avi A

    2 years ago
    well, because I can’t use the task parameters in the template
    I see I can use the flow parameters in the template, because they appear in the context (given they were provided to the flow, and not using the defaults)
    nicholas

    nicholas

    2 years ago
    Ah ok! I think you can use task parameters in the in the location template, since they're available in context. Let me confirm that for you quickly though
    a

    Avi A

    2 years ago
    I can
    but only if they were provided explicitly (i.e. not used the defaults)
    also, I’d like to cache a specific task based on the parameters it recursively depends on, and not necessarily on all the params
    nicholas

    nicholas

    2 years ago
    Seems like a good use case @Avi A - I'd encourage you to add your thoughts to that issue above, they'll be valuable in deciding the direction/implementation of that ticket