Channels
#prefect-community
Title
# prefect-community
t
Toby Rahloff
06/14/2022, 1:17 PMAnyone got an idea whether it is possible to view the average lateness of a flow in the new 2.0 Cloud UI? It was possible with the old UI (top right corner if I remember correctly)
✅ 1
a
Anna Geller
06/14/2022, 1:18 PM
can you say a bit more about what you mean by average lateness? 😄
t
Toby Rahloff
06/14/2022, 1:18 PM😄
a
Anna Geller
06/14/2022, 1:19 PM
Late run is simply a run in a Scheduled state which is already past the expected_start_time - that's why it's late
technically speaking, Late is just a label for a scheduled run indicating that this run should have already begun but hasn't (likely due to issues in the execution layer/agent)
t
Toby Rahloff
06/14/2022, 1:20 PMI think "avg lateness" was the exact Prefect lingo if I remember correctly.
When listing flows in the UI there were two metrics:
1. Avg Runtime - This showed how long the flow really take to run on average
2. Avg Lateness - how long it took from "pending" to "running" IIRC
OK got it. For us it was really helpful to analyze the average lateness for hundreds of flow runs. Is this feature removed in the new UI version or just someplace different?
a
Anna Geller
06/14/2022, 1:21 PM
why is it important to you to have this information? did you have many late runs in the past?
let me check
t
Toby Rahloff
06/14/2022, 1:22 PMIt helped us to fine-tune the autoscaling of our agent queues.
We are evaluating Kubernetes and Ray as the underlying compute resource for the agents and with Kubernetes we saw ~4 minutes avg lateness.
We now have the setup for Ray and wanted to compare the avg lateness for a real-world loadtest with hundreds of jobs
a
Anna Geller
06/14/2022, 1:23 PM
you know what, let me tag @Jenny - she can answer this better than me
t
Toby Rahloff
06/14/2022, 1:23 PMThanks 🙏
a
Anna Geller
06/14/2022, 1:24 PM
regarding your setup question: Ray is for running tasks, Kubernetes is for running flows
late runs would help you with Kubernetes "lateness" but not with Ray's lateness
but generally speaking, don't you think that having late runs is a problem in itself? in a typical use case, you shouldn't have any late runs if your agent is working properly 24/7
or are you using Serverless where this latency of roughly 4 min is normal?
t
Toby Rahloff
06/14/2022, 2:23 PMWe had two setups:
1. Kubernetes for running flows & Ray for running tasks
2. Ray for running flows & Ray for running tasks
In scenario 1 you are absolutely right. We are currently testing scenario 2 to compare the lateness of both setups
And yup, we are kinda scratching our heads at the moment why we have late runs.
"or are you using Serverless where this latency of roughly 4 min is normal?"
What do you mean with "serverless"? EKS Fargate?
j
Jenny
06/14/2022, 3:10 PMThanks for the feedback! We don't currently have the average lateness. I’ll pass your feedback on to our product team. If it’s a feature you'd like to see return, it might also be good to open an issue with more details about your use case so we can track it.
t
Toby Rahloff
06/16/2022, 4:38 PMThanks a lot! Sure, will do. Where should I create the issue?
j
Jenny
06/16/2022, 5:08 PM5 Views