https://prefect.io logo
Channels
Powered by
k

Krzysztof Nawara

11/21/2020, 11:28 AM
Hi! I'm interested in measuring percentage of tasks in the pipeline that were read from the cache. The only way to obtain that information that I can think of would be to have a shared counter that would be incremented at the beginning of each task. I believe that global variable won't work when running in distributed mode so I was wondering about an alternative. Is it possible to modify prefect.context from within a task? And then access such accumulator in a task that runs at the very end and prints the result/saves it somewhere?
m

Marwan Sarieddine

11/21/2020, 2:18 PM
You’ll probably want to query the graphql api instead
this should get you the cached tasks for instance
Copy code
query {
  task_run(
      where: {
          state: {_eq: "Cached"},
          flow_run_id: {_eq: "<flow_id>"},
        }
    ) {
    task{
      name
    },
  }
}
and then to get the denominator - remove the 
where
clause/filter (assuming you want to divide by all your tasks )
k

Krzysztof Nawara

11/21/2020, 7:19 PM
Hey, thanks @Marwan Sarieddine for suggestion! But from what I've seen (and documentation seems to support that Cached state indicates that tasks output has been cached - not that the task has been read from cache (although in that case the state is going to be Cached too) https://docs.prefect.io/api/latest/engine/state.html#cached
m

Marwan Sarieddine

11/21/2020, 11:01 PM
I see - yes you are right @Krzysztof Nawara
Powered by