Adam Brusselback

    Adam Brusselback

    10 months ago
    Also, I am having some terrible UI performance now that I am testing with a more full workload, the number of projects / flows (55 projects with 27 registered flows each) I have deployed just cause terrible slowness
    Jenny

    Jenny

    10 months ago
    Thanks @Adam Brusselback - we're actively investigating ui performance and have a few PRs in progress that we hope will improve things. Can you give any more information about whether there are certain pages you are seeing issues with? Or any actions that have created issues?
    Also checking - looks like you're on server? (Rather than cloud?)
    Adam Brusselback

    Adam Brusselback

    10 months ago
    Yes, that's correct.
    here is the request payload for the constant slow requests:
    [{"operationName":"TimelineFlowRuns","variables":{"project_id":null},"query":"query TimelineFlowRuns($project_id: uuid) {\n  flow_run(\n    where: {flow: {project_id: {_eq: $project_id}}, state: {_eq: \"Scheduled\"}}\n    limit: 10\n    order_by: {scheduled_start_time: asc_nulls_last}\n  ) {\n    id\n    flow_id\n    name\n    end_time\n    start_time\n    scheduled_start_time\n    state\n    __typename\n  }\n}\n"},{"operationName":"FlowRuns","variables":{"projectId":null,"heartbeat":"2021-10-29T14:00:00Z"},"query":"query FlowRuns($projectId: uuid, $heartbeat: timestamptz) {\n  Success: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Success\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Running: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Running\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Pending: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Pending\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Failed: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Failed\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  ClientFailed: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"ClientFailed\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Submitted: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Submitted\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Queued: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Queued\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Resume: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Resume\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Retrying: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Retrying\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Looped: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Looped\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Cached: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Cached\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Mapped: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Mapped\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  TimedOut: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"TimedOut\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  TriggerFailed: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"TriggerFailed\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Skipped: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Skipped\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Finished: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Finished\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n  Cancelled: flow_run_aggregate(\n    where: {flow: {project_id: {_eq: $projectId}}, scheduled_start_time: {_gte: $heartbeat}, state: {_eq: \"Cancelled\"}}\n  ) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n}\n"},{"operationName":"InProgressFlowRuns","variables":{"projectId":null},"query":"query InProgressFlowRuns($projectId: uuid) {\n  flow_run(\n    where: {flow: {project_id: {_eq: $projectId}}, state: {_in: [\"Running\", \"Submitted\", \"Cancelling\"]}}\n    order_by: {start_time: desc}\n  ) {\n    id\n    name\n    start_time\n    state\n    state_timestamp\n    flow {\n      id\n      name\n      flow_group_id\n      __typename\n    }\n    updated\n    __typename\n  }\n}\n"},{"operationName":"Notifications","variables":{"limit":5,"orderBy":{"created":"desc"},"where":{"_or":[{"tenant_id":{"_eq":"455c8ef9-0c91-413f-b50f-2eb1364e2e12"}},{"tenant_id":{"_is_null":true}}],"read":{"_eq":false}}},"query":"query Notifications($limit: Int, $offset: Int, $orderBy: [message_order_by!], $where: message_bool_exp) {\n  notifications: message(\n    where: $where\n    limit: $limit\n    offset: $offset\n    order_by: $orderBy\n  ) {\n    id\n    content\n    created\n    read\n    content\n    tenant_id\n    text\n    type\n    updated\n    __typename\n  }\n}\n"},{"operationName":"NotificationsCount","variables":{"where":{"_or":[{"tenant_id":{"_eq":"455c8ef9-0c91-413f-b50f-2eb1364e2e12"}},{"tenant_id":{"_is_null":true}}],"read":{"_eq":false}}},"query":"query NotificationsCount($where: message_bool_exp) {\n  message_aggregate(where: $where) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n}\n"},{"operationName":"Agents","variables":{},"query":"query Agents {\n  agent {\n    id\n    agent_config_id\n    core_version\n    created\n    name\n    labels\n    last_queried\n    type\n    __typename\n  }\n}\n"},{"operationName":"UpcomingFlowRuns","variables":{"projectId":null},"query":"query UpcomingFlowRuns($projectId: uuid) {\n  flow_run(\n    where: {flow: {project_id: {_eq: $projectId}}, state: {_eq: \"Scheduled\"}}\n    order_by: [{scheduled_start_time: asc}, {flow: {name: asc}}]\n  ) {\n    id\n    name\n    state\n    labels\n    scheduled_start_time\n    version\n    flow {\n      id\n      name\n      schedule\n      environment\n      is_schedule_active\n      __typename\n    }\n    __typename\n  }\n}\n"}]
    Jenny

    Jenny

    10 months ago
    Ok. That's helpful. Out of interest (because I've seen something similar) were those runs previously late? And how are you running them? (i.e. what type of agent but also are they on a schedule?)
    Adam Brusselback

    Adam Brusselback

    10 months ago
    Local agent, yes all scheduled (cron). 23 of those are hourly schedules deployed to 55 different projects
    I cannot detect any load on the (dedicated) DB server for these queries, the graphql server doesn't really seem to have any load either. It seems like it's really just the client choking
    Jenny

    Jenny

    10 months ago
    Ok. Thank you. I'll pass this onto the team to see if it helps them get any more fixes for performance. You'll notice that we're moving to a REST API for Orion. 🙂