I ran this query in the hopes that I could identify why my task concurrency limits were being used up when the cloud UI was reporting that no flows were running.

query get_running_flows {
  flow_run(where: {state: {_eq: "Running"}}) {
      state
      created
    	flow {
        id
      }
  }
}

I received this error

{
  "errors": [
    {
      "path": [
        "flow_run",
        0,
        "created"
      ],
      "message": "Cannot return null for non-nullable field flow_run.created.",
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR"
      }
    }
  ],
  "data": null
}

So it looks like the associated flow is no longer available. I removed the

flow

clause from the api to just get the flow run ids and tried deleting them using

mutation delete_flow_run {
  deleteFlowRun(input: {flowRunId: 
    "87b8fa4f-a206-438f-95b2-364ea4187338"}) {
    success,
    error
  }
}

but I get this response

{
  "data": {
    "deleteFlowRun": {
      "success": false,
      "error": null
    }
  }
}

My flows run either using kubernetes agents or local agents and this issue impacts both of them

Can you try something like this::

query {
  task_run (where: {state: {_eq: "Running"},
                    task: {tags: {_contains: "tag"}}}) {
    id
    flow_run{
      name
      id
    }
    task{
      name
      tags
    }
  }
}

?

Hi @Kevin Kho thank you for your response. I just wanted to follow up and see if you could help me identify what is causing this issue and the best way to resolve it. There appears to be data inconsistency on the cloud server side and this is causing issues with my production flows as they are consuming concurrency limits. Using this command

query check_flows_with_labels {
  flow_run(where: {state: {_eq: "Running"}, labels: {_contains: "limited-label"}}) {
    id
    created
    name
  }
}

I was able to identify some mysterious old flow runs that are not showing up in the UI Adding a

flow { name }

clause to the query above gives this error

{
  "errors": [
    {
      "path": [
        "flow_run",
        0,
        "id"
      ],
      "message": "Cannot return null for non-nullable field flow_run.id.",
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR"
      }
    }
  ],
  "data": null
}

Trying to cancel one of the mysterious flow runs gives this issue

mutation cancel_flow_run {
  cancel_flow_run(input: {flow_run_id: "cd1924f1-a0f3-4cc5-a35b-322d4e034e2b"}) {
    state
  }
}

{
  "errors": [
    {
      "path": [
        "cancel_flow_run"
      ],
      "message": "'NoneType' object has no attribute 'flow_group_id'",
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR"
      }
    }
  ],
  "data": {
    "cancel_flow_run": null
  }
}

Hi @Kevin Kho following up here. I understand you must be busy with the hard work you team has been doing on Prefect 2.0 but this is a big source of concern for my team and is impacting my flows in production. I have raised this before in a thread with @Anna Geller. tldr: Is that my task concurrency task limits are being used up by flows that apparently do not exist anymore. This is preventing newer flows from running.