https://prefect.io logo
#prefect-community
Title
# prefect-community
t

Toby Rahloff

06/14/2022, 1:17 PM
Anyone got an idea whether it is possible to view the average lateness of a flow in the new 2.0 Cloud UI? It was possible with the old UI (top right corner if I remember correctly)
โœ… 1
a

Anna Geller

06/14/2022, 1:18 PM
can you say a bit more about what you mean by average lateness? ๐Ÿ˜„
t

Toby Rahloff

06/14/2022, 1:18 PM
๐Ÿ˜„
a

Anna Geller

06/14/2022, 1:19 PM
Late run is simply a run in a Scheduled state which is already past the expected_start_time - that's why it's late
technically speaking, Late is just a label for a scheduled run indicating that this run should have already begun but hasn't (likely due to issues in the execution layer/agent)
t

Toby Rahloff

06/14/2022, 1:20 PM
I think "avg lateness" was the exact Prefect lingo if I remember correctly. When listing flows in the UI there were two metrics: 1. Avg Runtime - This showed how long the flow really take to run on average 2. Avg Lateness - how long it took from "pending" to "running" IIRC
OK got it. For us it was really helpful to analyze the average lateness for hundreds of flow runs. Is this feature removed in the new UI version or just someplace different?
a

Anna Geller

06/14/2022, 1:21 PM
why is it important to you to have this information? did you have many late runs in the past?
let me check
t

Toby Rahloff

06/14/2022, 1:22 PM
It helped us to fine-tune the autoscaling of our agent queues. We are evaluating Kubernetes and Ray as the underlying compute resource for the agents and with Kubernetes we saw ~4 minutes avg lateness. We now have the setup for Ray and wanted to compare the avg lateness for a real-world loadtest with hundreds of jobs
a

Anna Geller

06/14/2022, 1:23 PM
you know what, let me tag @Jenny - she can answer this better than me
t

Toby Rahloff

06/14/2022, 1:23 PM
Thanks ๐Ÿ™
a

Anna Geller

06/14/2022, 1:24 PM
regarding your setup question: Ray is for running tasks, Kubernetes is for running flows late runs would help you with Kubernetes "lateness" but not with Ray's lateness
but generally speaking, don't you think that having late runs is a problem in itself? in a typical use case, you shouldn't have any late runs if your agent is working properly 24/7 or are you using Serverless where this latency of roughly 4 min is normal?
t

Toby Rahloff

06/14/2022, 2:23 PM
We had two setups: 1. Kubernetes for running flows & Ray for running tasks 2. Ray for running flows & Ray for running tasks In scenario 1 you are absolutely right. We are currently testing scenario 2 to compare the lateness of both setups
And yup, we are kinda scratching our heads at the moment why we have late runs. "or are you using Serverless where this latency of roughly 4 min is normal?" What do you mean with "serverless"? EKS Fargate?
j

Jenny

06/14/2022, 3:10 PM
Thanks for the feedback! We don't currently have the average lateness. Iโ€™ll pass your feedback on to our product team. If itโ€™s a feature you'd like to see return, it might also be good to open an issue with more details about your use case so we can track it.
t

Toby Rahloff

06/16/2022, 4:38 PM
Thanks a lot! Sure, will do. Where should I create the issue?
5 Views