Hey all, I need some urgent assistance dealing with corrupted flow metadata. We deleted a flow yesterday after we had some performance issues. Every since then, our prefect agent has been stalling and repeatedly producing this error:

prefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]"

As far as I understand, this is a flow_run record without a flow. I need some assistance with removing this corrupted metadata as our data pipelines are all majorly delayed.

Hey David Are you able to query for the flow runs that these errors are coming from in the interactive API? Assuming you are you can run this query to cancel the flow/tasks runs that may be causing this

mutation setFlowRunStates($input: [set_flow_run_state_input!]!) {
set_flow_run_states(input: {states: $input}) {
  states {
  id
  __typename
  }
  __typename
}
}

With this input

{
  "input": {
    "flow_run_id": "fbca6355-a892-4c1c-bee8-1e1655bb718a",
    "version": 1,
    "state": {
      "type": "Cancelled",
      "message": "marked task run as Cancelled because \"because\""
    }
  }
}

Outside of that do you happen to have the Id for the flow you deleted or some of the flow runs? Feel free to DM those to me if you don't want to share them in this thread

@Mason Menges Your query example has an input of a specific flow run id. I do not have the flow run id as it’s been deleted.

Hmm would you be able to run this query to see if you have any running flows that might be related to this

query{
 	flow_run(where: {state: {_eq:"Running"}}) {
    name
    id
    state
    }
}

@Mason Menges we have other flows running on other agents that aren’t affected so that query doesn’t return the affected data. We did some research and found that someone has reported this issue last year: https://github.com/PrefectHQ/prefect/issues/4952