Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Adam Brusselback
10/30/2021, 2:08 PMAlso, I am having some terrible UI performance now that I am testing with a more full workload, the number of projects / flows (55 projects with 27 registered flows each) I have deployed just cause terrible slowness
j
Jenny
10/30/2021, 2:15 PMThanks @Adam Brusselback - we're actively investigating ui performance and have a few PRs in progress that we hope will improve things. Can you give any more information about whether there are certain pages you are seeing issues with? Or any actions that have created issues?
Also checking - looks like you're on server? (Rather than cloud?)
a
Adam Brusselback
10/30/2021, 2:18 PMYes, that's correct.
here is the request payload for the constant slow requests:
[{"operationName":"TimelineFlowRuns","variables":{"project_id":null},"query":"query TimelineFlowRuns($project_id: uuid) {\n flow_run(\n where: {flow: {project_id: {_eq: $project_id}}, state: {_eq: \"Scheduled\"}}\n limit: 10\n order_by: {scheduled_start_time: asc_nulls_last}\n ) {\n id\n flow_id\n name\n end_time\n start_time\n scheduled_start_time\n state\n __typename\n }\n}\n"},{"operationName":"FlowRuns","variables":{"projectId":null,"heartbeat":"2021-10-29T14:00:00Z"},"query":"query FlowRuns($projectId: uuid, $heartbeat: timestamptz) {\n Success: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Success\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Running: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Running\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Pending: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Pending\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Failed: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Failed\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n ClientFailed: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"ClientFailed\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Submitted: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Submitted\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Queued: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Queued\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Resume: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Resume\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Retrying: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Retrying\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Looped: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Looped\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Cached: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Cached\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Mapped: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Mapped\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n TimedOut: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"TimedOut\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n TriggerFailed: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"TriggerFailed\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Skipped: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Skipped\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Finished: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Finished\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Cancelled: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Cancelled\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n}\n"},{"operationName":"InProgressFlowRuns","variables":{"projectId":null},"query":"query InProgressFlowRuns($projectId: uuid) {\n flow_run(\n where: {flow: {project_id: {_eq: $projectId}}, state: {_in: [\"Running\", \"Submitted\", \"Cancelling\"]}}\n order_by: {start_time: desc}\n ) {\n id\n name\n start_time\n state\n state_timestamp\n flow {\n id\n name\n flow_group_id\n __typename\n }\n updated\n __typename\n }\n}\n"},{"operationName":"Notifications","variables":{"limit":5,"orderBy":{"created":"desc"},"where":{"_or":[{"tenant_id":{"_eq":"455c8ef9-0c91-413f-b50f-2eb1364e2e12"}},{"tenant_id":{"_is_null":true}}],"read":{"_eq":false}}},"query":"query Notifications($limit: Int, $offset: Int, $orderBy: [message_order_by!], $where: message_bool_exp) {\n notifications: message(\n where: $where\n limit: $limit\n offset: $offset\n order_by: $orderBy\n ) {\n id\n content\n created\n read\n content\n tenant_id\n text\n type\n updated\n __typename\n }\n}\n"},{"operationName":"NotificationsCount","variables":{"where":{"_or":[{"tenant_id":{"_eq":"455c8ef9-0c91-413f-b50f-2eb1364e2e12"}},{"tenant_id":{"_is_null":true}}],"read":{"_eq":false}}},"query":"query NotificationsCount($where: message_bool_exp) {\n message_aggregate(where: $where) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n}\n"},{"operationName":"Agents","variables":{},"query":"query Agents {\n agent {\n id\n agent_config_id\n core_version\n created\n name\n labels\n last_queried\n type\n __typename\n }\n}\n"},{"operationName":"UpcomingFlowRuns","variables":{"projectId":null},"query":"query UpcomingFlowRuns($projectId: uuid) {\n flow_run(\n where: {flow: {project_id: {_eq: $projectId}}, state: {_eq: \"Scheduled\"}}\n order_by: [{scheduled_start_time: asc}, {flow: {name: asc}}]\n ) {\n id\n name\n state\n labels\n scheduled_start_time\n version\n flow {\n id\n name\n schedule\n environment\n is_schedule_active\n __typename\n }\n __typename\n }\n}\n"}]j
Jenny
10/30/2021, 2:23 PMOk. That's helpful. Out of interest (because I've seen something similar) were those runs previously late? And how are you running them? (i.e. what type of agent but also are they on a schedule?)
a
Adam Brusselback
10/30/2021, 2:25 PMLocal agent, yes all scheduled (cron). 23 of those are hourly schedules deployed to 55 different projects
I cannot detect any load on the (dedicated) DB server for these queries, the graphql server doesn't really seem to have any load either. It seems like it's really just the client choking
j
Jenny
10/30/2021, 2:35 PMOk. Thank you. I'll pass this onto the team to see if it helps them get any more fixes for performance.
You'll notice that we're moving to a REST API for Orion. 🙂