Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
t
Toby Rahloff
06/14/2022, 1:17 PMAnyone got an idea whether it is possible to view the average lateness of a flow in the new 2.0 Cloud UI? It was possible with the old UI (top right corner if I remember correctly)
✅ 1
a
Anna Geller
06/14/2022, 1:18 PMcan you say a bit more about what you mean by average lateness? 😄
t
Toby Rahloff
06/14/2022, 1:18 PM😄
a
Anna Geller
06/14/2022, 1:19 PMLate run is simply a run in a Scheduled state which is already past the expected_start_time - that's why it's late
technically speaking, Late is just a label for a scheduled run indicating that this run should have already begun but hasn't (likely due to issues in the execution layer/agent)
t
Toby Rahloff
06/14/2022, 1:20 PMI think "avg lateness" was the exact Prefect lingo if I remember correctly.
When listing flows in the UI there were two metrics:
1. Avg Runtime - This showed how long the flow really take to run on average
2. Avg Lateness - how long it took from "pending" to "running" IIRC
OK got it. For us it was really helpful to analyze the average lateness for hundreds of flow runs. Is this feature removed in the new UI version or just someplace different?
a
Anna Geller
06/14/2022, 1:21 PMwhy is it important to you to have this information? did you have many late runs in the past?
let me check
t
Toby Rahloff
06/14/2022, 1:22 PMIt helped us to fine-tune the autoscaling of our agent queues.
We are evaluating Kubernetes and Ray as the underlying compute resource for the agents and with Kubernetes we saw ~4 minutes avg lateness.
We now have the setup for Ray and wanted to compare the avg lateness for a real-world loadtest with hundreds of jobs
a
Anna Geller
06/14/2022, 1:23 PMyou know what, let me tag @Jenny - she can answer this better than me
t
Toby Rahloff
06/14/2022, 1:23 PMThanks 🙏
a
Anna Geller
06/14/2022, 1:24 PMregarding your setup question: Ray is for running tasks, Kubernetes is for running flows
late runs would help you with Kubernetes "lateness" but not with Ray's lateness
but generally speaking, don't you think that having late runs is a problem in itself? in a typical use case, you shouldn't have any late runs if your agent is working properly 24/7
or are you using Serverless where this latency of roughly 4 min is normal?
t
Toby Rahloff
06/14/2022, 2:23 PMWe had two setups:
1. Kubernetes for running flows & Ray for running tasks
2. Ray for running flows & Ray for running tasks
In scenario 1 you are absolutely right. We are currently testing scenario 2 to compare the lateness of both setups
And yup, we are kinda scratching our heads at the moment why we have late runs.
"or are you using Serverless where this latency of roughly 4 min is normal?"
What do you mean with "serverless"? EKS Fargate?
j
Jenny
06/14/2022, 3:10 PMThanks for the feedback! We don't currently have the average lateness. I’ll pass your feedback on to our product team. If it’s a feature you'd like to see return, it might also be good to open an issue with more details about your use case so we can track it.
t
Toby Rahloff
06/16/2022, 4:38 PMThanks a lot! Sure, will do. Where should I create the issue?
j
Jenny
06/16/2022, 5:08 PM