Also, I am having some terrible UI performance now...
# prefect-uia
Adam Brusselback
10/30/2021, 2:08 PMAlso, I am having some terrible UI performance now that I am testing with a more full workload, the number of projects / flows (55 projects with 27 registered flows each) I have deployed just cause terrible slowness
j
Jenny
10/30/2021, 2:15 PMThanks @Adam Brusselback - we're actively investigating ui performance and have a few PRs in progress that we hope will improve things. Can you give any more information about whether there are certain pages you are seeing issues with? Or any actions that have created issues?
Jenny
10/30/2021, 2:17 PMAlso checking - looks like you're on server? (Rather than cloud?)
a
Adam Brusselback
10/30/2021, 2:18 PMYes, that's correct.
Adam Brusselback
10/30/2021, 2:18 PMAdam Brusselback
10/30/2021, 2:21 PMAdam Brusselback
10/30/2021, 2:23 PMhere is the request payload for the constant slow requests:
Copy code
[{"operationName":"TimelineFlowRuns","variables":{"project_id":null},"query":"query TimelineFlowRuns($project_id: uuid) {\n flow_run(\n where: {flow: {project_id: {_eq: $project_id}}, state: {_eq: \"Scheduled\"}}\n limit: 10\n order_by: {scheduled_start_time: asc_nulls_last}\n ) {\n id\n flow_id\n name\n end_time\n start_time\n scheduled_start_time\n state\n __typename\n }\n}\n"},{"operationName":"FlowRuns","variables":{"projectId":null,"heartbeat":"2021-10-29T14:00:00Z"},"query":"query FlowRuns($projectId: uuid, $heartbeat: timestamptz) {\n Success: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Success\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Running: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Running\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Pending: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Pending\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Failed: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Failed\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n ClientFailed: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"ClientFailed\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Submitted: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Submitted\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Queued: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Queued\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Resume: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Resume\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Retrying: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Retrying\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Looped: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Looped\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Cached: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Cached\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Mapped: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Mapped\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n TimedOut: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"TimedOut\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n TriggerFailed: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"TriggerFailed\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Skipped: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Skipped\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Finished: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Finished\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n Cancelled: flow_run_aggregate(\n where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Cancelled\"}}\n ) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n}\n"},{"operationName":"InProgressFlowRuns","variables":{"projectId":null},"query":"query InProgressFlowRuns($projectId: uuid) {\n flow_run(\n where: {flow: {project_id: {_eq: $projectId}}, state: {_in: [\"Running\", \"Submitted\", \"Cancelling\"]}}\n order_by: {start_time: desc}\n ) {\n id\n name\n start_time\n state\n state_timestamp\n flow {\n id\n name\n flow_group_id\n __typename\n }\n updated\n __typename\n }\n}\n"},{"operationName":"Notifications","variables":{"limit":5,"orderBy":{"created":"desc"},"where":{"_or":[{"tenant_id":{"_eq":"455c8ef9-0c91-413f-b50f-2eb1364e2e12"}},{"tenant_id":{"_is_null":true}}],"read":{"_eq":false}}},"query":"query Notifications($limit: Int, $offset: Int, $orderBy: [message_order_by!], $where: message_bool_exp) {\n notifications: message(\n where: $where\n limit: $limit\n offset: $offset\n order_by: $orderBy\n ) {\n id\n content\n created\n read\n content\n tenant_id\n text\n type\n updated\n __typename\n }\n}\n"},{"operationName":"NotificationsCount","variables":{"where":{"_or":[{"tenant_id":{"_eq":"455c8ef9-0c91-413f-b50f-2eb1364e2e12"}},{"tenant_id":{"_is_null":true}}],"read":{"_eq":false}}},"query":"query NotificationsCount($where: message_bool_exp) {\n message_aggregate(where: $where) {\n aggregate {\n count\n __typename\n }\n __typename\n }\n}\n"},{"operationName":"Agents","variables":{},"query":"query Agents {\n agent {\n id\n agent_config_id\n core_version\n created\n name\n labels\n last_queried\n type\n __typename\n }\n}\n"},{"operationName":"UpcomingFlowRuns","variables":{"projectId":null},"query":"query UpcomingFlowRuns($projectId: uuid) {\n flow_run(\n where: {flow: {project_id: {_eq: $projectId}}, state: {_eq: \"Scheduled\"}}\n order_by: [{scheduled_start_time: asc}, {flow: {name: asc}}]\n ) {\n id\n name\n state\n labels\n scheduled_start_time\n version\n flow {\n id\n name\n schedule\n environment\n is_schedule_active\n __typename\n }\n __typename\n }\n}\n"}]j
Jenny
10/30/2021, 2:23 PMOk. That's helpful. Out of interest (because I've seen something similar) were those runs previously late? And how are you running them? (i.e. what type of agent but also are they on a schedule?)
a
Adam Brusselback
10/30/2021, 2:25 PMLocal agent, yes all scheduled (cron). 23 of those are hourly schedules deployed to 55 different projects
Adam Brusselback
10/30/2021, 2:29 PMI cannot detect any load on the (dedicated) DB server for these queries, the graphql server doesn't really seem to have any load either. It seems like it's really just the client choking
j
Jenny
10/30/2021, 2:35 PMOk. Thank you. I'll pass this onto the team to see if it helps them get any more fixes for performance.
You'll notice that we're moving to a REST API for Orion. 🙂
8 Views